How do social media companies like twitter or youtube not run out of space for posts?
How do social media companies like twitter or youtube not run out of space for posts?
I know memory is fairly cheap but e.g. there are millions of new videos on youtube everyday, each probably few hundred MBs to few GBs. It all has to take enormous amount of space. Not to mention backups.
You're viewing a single thread.
-
It’s the same story with AWS as well. They use vast amounts of storage and leverage different tiers of storage to get the service they want. It’s funny but they have insane amounts of SD cards ( cheapest storage available at the size) and use that for some storage and just replicate things everywhere for durability. Imagine how small 256GB SD cards are and they you have hardware to plug-in 200 of them practically stacked on top of each other. The storage doesn’t need to be the best, it just needs to be managed appropriately and actively to ensure that data is always replicated as devices fail. That’s just the cooler tier stuff. It gets complex as the data warms.
-
SD cards? I'm very skeptical. Do you have a source?
-
Yeah this seems false. SD cards are unreliable, hard to keep track of, and don’t actually store that much data for the price. I do think they use tapes though to store long term, low traffic data.
-
Used to work there but it’s been a few years so maybe things changed but that was how we originally got super cheap and durable s3.
-
We use LTO tapes in Hollywood to back up raw footage; it wouldn't surprise me if AWS uses tapes for glacier.
I got a tour of Iron Mountain once (where we sent tapes for long term archival). They had a giant room with racks and racks of LTOs, and a robot on rails that would make copies of each tape at regular intervals to keep the data from corrupting. It looked kinda like the archive room in Rogue One. Wouldn't surprise me if Iron Mountain was an inspiration for the design. Super interesting.
-
That sounds really cool! I wonder what sort of redundancy they have in case a tape gets damaged or corrupted.
-
-
-
-
Ha, I had no idea data centers use SD cards! It makes sense in hindsight, but it's still funny to think about
-