« previous next »
Pages: [1]

Author Topic: How Large is the Internet?  (Read 2552 times)

S. Williamson

  • formerly Dionysusigma
  • friends
  • Senior Member
  • ***
  • Posts: 3,034
  • It's not the years, it's the mileage.
How Large is the Internet?
« on: November 18, 2009, 09:29:45 AM »
I'm talking space-wise, not actual dimensions.  =D  Seriously, though, how much drive space/ double-sided sheets of paper/ whatever would it take to store the current internet, plus how much new content (in terms of paper or drive space) is added every hour?

Reason I'm asking is I've recently finished reading Fahrenheit 451 again, and akin to people memorizing books to be re-printed later, would it be possible/ feasible to have a "back-up" of the entire internet?

Report to moderator   Logged
Quote
"The chances of finding out what's really going on are so remote, the only thing to do is hang the sense of it and keep yourself occupied. I'd far rather be happy than right any day."
"And are you?"
"No, that's where it all falls apart I'm afraid. Pity, it sounds like quite a nice lifestyle otherwise."
-Douglas Adams

GigaBuist

  • friends
  • Senior Member
  • ***
  • Posts: 4,345
    • http://www.justinbuist.org/blog/
Re: How Large is the Internet?
« Reply #1 on: November 18, 2009, 09:36:26 AM »
Quote
Seriously, though, how much drive space/ double-sided sheets of paper/ whatever would it take to store the current internet, plus how much new content (in terms of paper or drive space) is added every hour?

Archive.org's Wayback Machine says in their FAQ they have 2 petabytes of data on disk and add another 20 terabytes every month.

That's a lot of floppy disks.
Report to moderator   Logged

AJ Dual

  • friends
  • Senior Member
  • ***
  • Posts: 16,162
  • Shoe Ballistics Inc.
Re: How Large is the Internet?
« Reply #2 on: November 18, 2009, 09:47:04 AM »
They don't discuss it much, but Google, Microsoft, and other heavy users of server and disk space now use commoditized shipping container modules.

They're assembled and SEALED at the factory, and have external data and power hookups. The shipping container arrives at the datacenter, is hooked in, monitored remotely/autonomously, and then when failures of drives or server card blades reach a certain percentage, the shipping container is then undocked and sent back to be refurbished.

http://news.cnet.com/8301-13860_3-10020902-56.html

Report to moderator   Logged
I promise not to duck.

MicroBalrog

  • friend
  • Senior Member
  • ***
  • Posts: 14,505
Re: How Large is the Internet?
« Reply #3 on: November 18, 2009, 10:02:51 AM »
Rapidshare alone has a capacity of 10 petabytes.
Report to moderator   Logged
Destroy The Enemy in Hand-to-Hand Combat.

"...tradition and custom becomes intertwined and are a strong coercion which directs the society upon fixed lines, and strangles liberty. " ~ William Graham Sumner

BrokenPaw

  • friends
  • Senior Member
  • ***
  • Posts: 1,674
  • Sedit qvi timvit ne non svccederet.
    • ShadowGrove Interpath Ministry
Re: How Large is the Internet?
« Reply #4 on: November 18, 2009, 12:26:28 PM »
It can't be that big.  It all fits into my mom's Dell.  She told me so.   ;/
Report to moderator   Logged
Seek out wisdom in books, rare manuscripts, and cryptic poems if you will, but seek it also in simple stones and fragile herbs and in the cries of wild birds. Listen to the song of the wind and the roar of water if you would discover magic, for it is here that the old secrets are still preserved.

zahc

  • friend
  • Senior Member
  • ***
  • Posts: 5,803
Re: How Large is the Internet?
« Reply #5 on: November 18, 2009, 12:30:36 PM »
I tried wget http://www.*, but it didn't work for some reason.
Report to moderator   Logged
Maybe a rare occurence, but then you only have to get murdered once to ruin your whole day.
--Tallpine

Marnoot

  • friend
  • Senior Member
  • ***
  • Posts: 2,965
Re: How Large is the Internet?
« Reply #6 on: November 18, 2009, 12:52:17 PM »
Quote
Eric Schmidt, the CEO of Google, the world’s largest index of the Internet, estimated the size at roughly 5 million terabytes of data. That’s over 5 billion gigabytes of data, or 5 trillion megabytes. Schmidt further noted that in its seven years of operations, Google has indexed roughly 200 terabytes of that, or .004% of the total size.

Seen that quote before. Don't have any primary sources, thus no link. Google the quote though and you'll get plenty of links.
Report to moderator   Logged

Zardozimo Oprah Bannedalas

  • Webley Juggler
  • friend
  • Senior Member
  • ***
  • Posts: 4,415
  • All I got is a fistful of shekels
Re: How Large is the Internet?
« Reply #7 on: November 18, 2009, 01:04:36 PM »
Quote
Schmidt further noted that in its seven years of operations, Google has indexed roughly 200 terabytes of that, or .004% of the total size.
Cool.  Does the dark side of the internet have more porn? Or more malware? Or is it just infested with furries?
It would be interesting to calculate the amount of non-duplicate information on the internet. I get the notion that it's a lot smaller than you would believe.
Report to moderator   Logged

AJ Dual

  • friends
  • Senior Member
  • ***
  • Posts: 16,162
  • Shoe Ballistics Inc.
Re: How Large is the Internet?
« Reply #8 on: November 18, 2009, 01:08:35 PM »
Yes, a lot of it probably is porn. However there's tons of academic and technical info too, and all of it which is that "other" 99.996% of the content of the Internet that's considered the "Deep Web". Information that's accessable, or public, but is not indexable due to the way it's linked, or is indexed only superficialy at a high level.

The analogy on your own desktop computer is the difference in time and effort between searching by file names vs. searching the actual contents of the files.

The group that can figure out a method to efficiently search and crawl the "Deep Web" will be able to write their own ticket. And assuming they aren't Google, they will be the "next Google".
Report to moderator   Logged
I promise not to duck.

Headless Thompson Gunner

  • friend
  • Senior Member
  • ***
  • Posts: 8,517
Re: How Large is the Internet?
« Reply #9 on: November 18, 2009, 01:12:35 PM »
Use a bit of lossy compression, filter out all the crap, and you'd probably find the internet is a bit smaller...
Report to moderator   Logged

lee n. field

  • friend
  • Senior Member
  • ***
  • Posts: 13,622
  • tinpot megalomaniac, Paulbot, hardware goon
Re: How Large is the Internet?
« Reply #10 on: November 18, 2009, 03:50:41 PM »
Quote
   
How Large is the Internet?

This big.
Report to moderator   Logged
In thy presence is fulness of joy.
At thy right hand pleasures for evermore.

guns and more

  • New Member
  • Posts: 38
Re: How Large is the Internet?
« Reply #11 on: November 18, 2009, 08:05:18 PM »
If you don't count porn or junk ads, it's actually not that big.
Report to moderator   Logged

KPT

  • New Member
  • Posts: 66
Re: How Large is the Internet?
« Reply #12 on: November 19, 2009, 12:21:45 AM »
Quote from: AJ Dual on November 18, 2009, 09:47:04 AM
They don't discuss it much, but Google, Microsoft, and other heavy users of server and disk space now use commoditized shipping container modules.

They're assembled and SEALED at the factory, and have external data and power hookups. The shipping container arrives at the datacenter, is hooked in, monitored remotely/autonomously, and then when failures of drives or server card blades reach a certain percentage, the shipping container is then undocked and sent back to be refurbished.

http://news.cnet.com/8301-13860_3-10020902-56.html



Google released a somewhat detailed report of their server design.

I remember reading a more detailed article but can't find it.

http://news.cnet.com/8301-1001_3-10209580-92.html

I noticed in the first pic they are using Deathstars. Wonder how that is working for them.
Report to moderator   Logged

S. Williamson

  • formerly Dionysusigma
  • friends
  • Senior Member
  • ***
  • Posts: 3,034
  • It's not the years, it's the mileage.
Re: How Large is the Internet?
« Reply #13 on: November 19, 2009, 01:34:49 AM »
I've heard that their thermal exhaust ports, if left unprotected, can present a problem if attacked.





 =D ;/ =D
« Last Edit: November 19, 2009, 01:39:37 AM by Dionysusigma »
Report to moderator   Logged
Quote
"The chances of finding out what's really going on are so remote, the only thing to do is hang the sense of it and keep yourself occupied. I'd far rather be happy than right any day."
"And are you?"
"No, that's where it all falls apart I'm afraid. Pity, it sounds like quite a nice lifestyle otherwise."
-Douglas Adams

CNYCacher

  • friend
  • Senior Member
  • ***
  • Posts: 4,438
Re: How Large is the Internet?
« Reply #14 on: November 19, 2009, 08:05:32 AM »
Quote
Google has indexed roughly 200 terabytes of that, or .004% of the total size.

Bear in mind that .004% of the DATA on the internet represents X% of the text-based information on the internet, where X is a number far greater than .004
Report to moderator   Logged
On two occasions, I have been asked [by members of Parliament], "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question.
Charles Babbage

Perd Hapley

  • Superstar of the Internet
  • friend
  • Senior Member
  • ***
  • Posts: 61,500
  • My prepositions are on/in
Re: How Large is the Internet?
« Reply #15 on: November 19, 2009, 05:04:20 PM »
Quote from: guns and more on November 18, 2009, 08:05:18 PM
If you don't count porn or junk ads, it's actually not that big.

And if you don't count all the forum spam by those pathetic losers who rack up enormous post counts... 
Report to moderator   Logged
"Doggies are angel babies!" -- my wife

Jocassee

  • Buster Scruggs Respecter
  • friends
  • Senior Member
  • ***
  • Posts: 4,591
  • "First time?"
Re: How Large is the Internet?
« Reply #16 on: November 19, 2009, 05:19:44 PM »
Quote from: KPT on November 19, 2009, 12:21:45 AM
Google released a somewhat detailed report of their server design.

I remember reading a more detailed article but can't find it.

http://news.cnet.com/8301-1001_3-10209580-92.html

I noticed in the first pic they are using Deathstars. Wonder how that is working for them.

If you mean the IBM/Hitachi models...not that great.
Report to moderator   Logged
I shall not die alone, alone, but kin to all the powers,
As merry as the ancient sun and fighting like the flowers.
Pages: [1]
« previous next »