'Writing is on the wall for spinning rust': IBM joins Pure Storage in claiming disk drives will go the way of the dodo in enterprises

No shit. All they have to do is finally grow the balls to build SSD's in the same form factor as the 3.5" drives everyone in enterprise is already using, and stuff those to the gills with flash chips.

"But that will cannibalize our artificially price inflated/capacity restricted M.2 sales if consumers get their hands on them!!!"

Yep, it sure will. I'll take ten, please.

Something like that could easily fill the oodles of existing bays that are currently filled with mechanical drives, both in the home user/small scale enthusiast side and existing rackmount stuff. But that'd be too easy.

Hate to break it to you, but the 3.5" form factor would absolutely not be cheaper than an equivalent bunch of E1.S or M.2 drives. The price is not inflated due to the form factor, it's driven primarily by the cost of the NAND chips, and you'd just need more of them to take advantage of bigger area. To take advantage of the thickness of the form factor, it would need to be a multi-board solution. Also, there'd be a thermal problem, since thermal characteristics of a 3.5" application are not designed with the thermal load of that much SSD.

Add to that that 3.5" are currently maybe 24gb SAS connectors at best, which means that such a hypothetical product would be severely crippled by the interconnect. Throughput wise, talking about over 30 fold slower in theory than an equivalent volume of E1.S drives. Which is bad enough, but SAS has a single relatively shallow queue while an NVME target has thousands of deep queues befitting NAND randam access behavior. So a product has to redesign to vaguely handle that sort of product, and if you do that, you might as well do EDSFF. No one would buy something more expensive than the equivalent capacity in E1.S drives that performs only as well as the SAS connector allows,

The EDSFF defined 4 general form factors, the E1.S which is roughly M.2 sized, and then E1.L, which is over a foot long and would be the absolute most data per cubic volume. And E3.S and E3.L, which wants to be more 2.5"-like. As far as I've seen, the market only really wants E1.S despite the bigger form factors, so I tihnk the market has shown that 3.5" wouldn't have takers.
there are plenty of enterprise u.2 and m.2 22110 on ebay
You apparently have no clue what you're talking about.
I hope youre not putting m.2 drives in a server if you plan on reading the data from them at some point. Those are for consumers and there's an entirely different formfactor for enterprise storage using nvme drives.
- Enterprise systems do have m.2, though admittedly its only really used as pretty disposable boot volumes.
  
  Though they aren't used as data volumes so much, it's not due to unreliability, it's due to hot swap and power levels.
- Tell me, what would be the issue of reading data from an m.2 drive in a server?

So can someone make 3.5" SSDs then????

They can be made any size. Most SATA SSD are just a plastic housing around a board with some chips on it. The right question is when will we have a storage technology with the durability and reliability of spinning magnetized hard drive platters. The nand flash chips used in most SSD and m.2 are much more reliable than they were initially. But for long-term retention Etc. Are still off quite a good bit from traditional hard drives. Hard drives can sit for about 10 years generally before bit rot becomes a major concern. Nand flash is only a year or two iirc.
- Longer if it has some kind of small power. I think I read that somewhere.
Why? We can cram 61TB into a slightly overgrown 2.5” and like half a PB per rack unit.
- Because we don't have to pack it in too much. It'd be higher capacities for cheaper for consumers
  
  Also cooling
Given that there are already 32TB 2.5” SSDs, what does a 3.5” buy you that you couldn’t get with an adapter?
- Native slotting into server drive cages. No concerns about alignment with the front or back.
- They should be cheaper since theres a bunch more space to work with. You don't have to make the storage chips as small.
- A big heat sink like they used to put on WD Raptor drives.
- Build quality
- A better price as low density chips are cheaper.
  And you can fit in more of those in a bigger space = Cheaper.
Relevant video about the problems with high capacity ssds.

https://www.youtube.com/watch?v=Y2i8wZCXDF4
- I'm not particularly interested to watch a 40 minute video, so I skinned the transcript a bit.
  
  As my other comments show, I know there are reasons why 3.5 inch doesn't make sense in SSD context, but I didn't see anything in a skim of the transcript that seems relevant to that question. They are mostly talking about storage density rather than why not package bigger (and that industry is packaging bigger, but not anything resembling 3.5", because it doesn't make sense).
- Fourty minutes? Yeah, no. How about an equivalent text that can be parsed in five?
I want them like my 8" floppies!

I'll shed no tears, even as a NAS owner, once we get equivalent capacity SSD without ruining the bank :P

Considering the high prices for high density SSD chips...
Why are there no 3.5" SSDs with low density chips?
- Not enough of a market
  
  The industry answer is if you want that much volume of storage, get like 6 edsff or m.2 drives.
  
  3.5 inch is a useful format for platters, but not particularly needed to hold nand chips. Meanwhile instead of having to gate all those chips behind a singular connector, you can have 6 connectors to drive performance. Again, less important for a platter based strategy which is unlikely to saturate even a single 12 gb link in most realistic access patterns, but ssds can keep up with 128gb with utterly random io.
  
  Tiny drives means more flexibility. That storage product can go into nas, servers, desktops, the thinnest laptops and embedded applications, maybe wirh tweaked packaging and cooling solutions. A product designed for hosting that many ssd boards behind a single connector is not going to be trivial to modify for any other use case, bottleneck performance by having a single interface, and pretty guaranteed to cost more to manufacturer than selling the components as 6 drives.

Haven't they said that about magnetic tape as well?

Some 30 years ago?

Isn't magnetic tape still around? Isn't even IBM one of the major vendors?

Anyone who has said that doesn't know what they're talking about. Magnetic tape is unparalleled for long-term/archival storage.

This is completely different. For active storage, solid-state has been much better than spinning rust for a long time, it's just been drastically more expensive. What's being argued here is that it's not performant and while it might be more expensive initially, it's less expensive to run and maintain.
- Tape will survive, SSDs will survive. Spinning rust will die

As soon as I can get a 20TB SSD for ~$325-350, let me know. Until then, lol.

Meanwhile Western Digital moves away from SSD production and back to HDDs for massive storage of AI and data lakes and such: https://www.techspot.com/news/107039-western-digital-exits-ssd-market-shifts-focus-hard.html

Yea but isn't that more because SanDisk is going to fully focus on that? Or what am I missing?
- That SanDisk is it's own company now.
  But I don't k ow if they are still a subsidiary or completely spun of WD.

Hdds were a fad, I'm waiting for the return of tape drives. 500TB on a $20 cartridge and I can live with the 2 minute seek time.

Tape drives are still definitely a thing.
- If you exclude the introductory price of the drive and needing specialized software to read/write to it it's very affordable €/TB
It's not a real hard disk unless you can get it to walk across the server room anyway.
Tapes are still sold in pretty high densities, don't have to wait!

Spinning rust is a funny way of describing HDDs, but I immediately get it

My sample size of myself has had 1 drive fail in decades. It was a solid state drive. Thankfully it failed in a strangely intermittent way and I was able to recover the data. But still, it surprised me as one would assume solid state would be more reliable. That spinning rust has proven to be very reliable. But regardless I'm sure SSD will be/are better in every way.

I believe you see the main issue with your experiences - the sample size. With small enough sample you can experience almost anything. Wisdom is knowing what you can and what you cant extrapolate to the entire population
- I have one HDD that survived 20+ years, and an aliexpress SSD that died in 6 months. Therefore all SSDs are garbage!!!
  
  That’s also the only SSD I’ve ever had fail on me and I’ve had them since 2011. In that same time I’ve had probably 4 HDDs fail on me. Even then I know to use data from companies like backblaze that have infinitely more drives than I have.

Wouldn't a HDD based system be like 1/10th the price? I don't know if HDDs are going away any time soon.

Spinning platter capacity can't keep up with SSDs. HDDs are just starting to break the 30TB mark and SSDs are shipping 50+. The cost delta per TB is closing fast. You can also have always on compression and dedupe in most cases with flash, so you get better utilization.
- You can also have always on compression and dedupe in most cases with flash
  
  As you can with spinning disks. Nothing about flash makes this a special feature.
The disk cost is about a 3 fold difference, rather than order of magnitude now.

These disks didn't make up as much of the costs of these solutions as you'd think, so a disk based solution with similar capacity might be more like 40% cheaper rather than 90% cheaper.

The market for pure capacity play storage is well served by spinning platters, for now. But there's little reason to iterate on your storage subsystem design, the same design you had in 2018 can keep up with modern platters. Compared to SSD where form factor has evolved and the interface indicates revision for every pcie generation.
For servers physical space is also a huge concern. 2.5” drives cap out at like 6tb I think, while you can easily find an 8tb 2.5” SSD anywhere. We have 16tb drives in one of our servers at work and they weren’t even that expensive. (Relatively)
- You can put multiple 8 gig m.2 ssds into 2.5" slot.
It's losing cost advantages as time goes. Long term storage is still on tape (and that's actively developed too!), and flash is getting cheaper, and spinning disks have inherent bandwidth and latency limits. It's probably not going away entirely, but it's main usecases are being squeezed on both ends

Probably at some point as prices per TB continue to come down. I don't know anyone buying a laptop with a HDD these days. Can't imagine being likely to buy one for a desktop ever again either. Still got a couple of old ones active (one is 11 years old) but I do plan to replace them with SSDs at some point.

But you don't need 32TB storage per drive in your laptop.
- I don't know that sounds like a reasonable size for the new GTA.
- Then how will I fit my porn folder on it?

Just like magnetic tape! Oh wait..

I had a terrible experience through all my life with HDDs. Slow af, sector loss, corruption, OS corruption... I am traumatized. I got 8TB NvMe for less than $500... Since then I have not a single trouble (well except I n electric failure, BTRFS CoW tends to act weird and sometimes doesnt boot, you need manual intervention)

I agree that single HDDs are terrible, but once you raid multiple of them together, it becomes much better. And now with zfs even better still
Sounds like you may not be making enough sacrifices to The Omnisiah

/s

My datacenter is 80% nvme at this point. Just naturally. It's crazy.

Nvme is terrible value for storage density. There is no reason to use it except when you need the speed and low latency.
- There's a cost associated with making that determination and managing the storage tiering. When the NVME is only 3x more expensive per amount of data compared to HDD at scale, and "enough" storage for OS volume at the chepaest end where you can either have a good enough HDD or a good enough SDD at the same price, then OS volume just makes sense to be SSD.
  
  In terms of "but 3x is pretty big gap", that's true and does drive storage subsystems, but as the saying has long been, disks are cheap, storage is expensive. So managing HDD/SDD is generally more expensive than the disk cost difference anyway.
  
  BTW, NVME vs. non-NVME isn't the thing, it's NAND v. platter. You could have an NVME interfaced platters and it would be about the same as SAS interfaced platters or even SATA interfaced. NVME carried a price premium for a while mainly because of marketing stuff rather than technical costs. Nowadays NVME isn't too expensive. One could make an argument that number of PCIe lanes from the system seems expensive, but PCIe switches aren't really more expensive than SAS controllers, and CPUs have just so many innate PCIe lanes now.

I'm about to build a home server with a lot of storage (relatively, around 6 or 8 times 12 TB as a ballpark), and I didn't even consider anything other than spinning drives so far.

Because spinning disks are a bit cheaper than SSD?
- Especially for large sizes. Also speed isn't really much of an issue on a domestic network.

Just replace then all with flash, along with bluray (or other optical storage) for archival.

Optical media is not good for archival unless you're buying discs specifically manufactured for archival purposes.

I generally agree, it won't take long for SSDs to be cheap enough to justify the expense. HDD is in a way similar to CD/DVD, it had it's time, it even lasted much longer than expected, but eventually technology became cheaper and the slightly cheaper price didn't make sense any more.

SSD wins on all account for live systems, and long term cold storage goes to tapes. Not a lot of reasons to keep them around.

As a person hosting my own data storage, tape is completely out of reach. The equipment to read archival tapes would cost more than my entire system. It's also got extremely high latency compared to spinning disks, which I can still use as live storage.

Unless you're a huge company, spinning disks will be the way to go for bulk storage for quite a while.
- Well, tape is still relevant for the 3-2-1 backup rule and I worked in a pretty big hosting company where you would get out 400 tb of backup data each weekend. it's the only media allowing to have a real secured fully offline copy that won't depend on another online hosting service
- Define "quite a while"
  
  Sure, enterprise is likely to make the switch first, but it's also likely to kick start the price reduction to consumers. So I actually don't think it's that far away. I would guess we are like 5 years away from SSDs being the significant majority of consumer storage technology by volume.
  
  Even now, as a self hoster it's pretty reasonable to have SSDs if you are talking about single digit TB. Sure SSDs are about 2x the price, but we are talking about a difference of like 60 USD if you only need 2 TB.

"in enterprises" oh lol

I doubt it. SSDs are subject to quantuum tunneling. This means if you don't power up an SSD once in 2-5 years, your data is gone. HDDs have no such qualms. So long as they still spin, there's your data and when they no longer do, you still have the heads inside.

So you have a use case that SSDs will never replace, cold data storage. I use them for my cold offsite back ups.

Sorry dude, but bit rot is a very real thing on HDDs. They're magnetic media, which degrades over time. If you leave a disk cold for 2-5 years, there's a very good chance you'll get some bad sectors. SSDs aren't immune from bit rot, but that's not through quantum tunneling - not any more than your CPU is affected by it at least.
- I did not meant to come across as saying that HDDs don't suffer bit rot. However, there are specific long term storage HDDs that are built specifically to be powered up sporadically and resist external magnetic influences on the track. In a proper storage environment they will last over 5 years without being powered up and still retain all information. I know it because i use them in this exact scenario for over 2 decades. Conversely there are no such long term storage SSDs.
  
  SSDs store information through trapped charges which most certainly lose charge through quantuum tunneling as well as generalized charge leakage. As insulation loses effectiveness, the potential barrier for the charge allows for what is normally a manageable effect, much like in the CPU like you said, to become out of the scope of error correction techniques. This is a physical limitation that cannot be overcome.
You're wrong. HDD need about as much frequently powering up as SSD, because the magnetization gets weaker.
- Here's a copy paste from superuser that will hopefully show you that what you said is incorrect in a way i find expresses my thoughts exactly
  
  Magnetic Field Breakdown
  
  Most sources state that permanent magnets lose their magnetic field strength at a rate of 1% per year. Assuming this is valid, after ~69 years, we can assume that half of the sectors in a hard drive would be corrupted (since they all lost half of their strength by this time). Obviously, this is quite a long time, but this risk is easily mitigated - simply re-write the data to the drive. How frequently you need to do this depends on the following two issues (I also go over this in my conclusion).
  
  https://superuser.com/questions/284427/how-much-time-until-an-unused-hard-drive-loses-its-data
- Note that for HDDs, it doesn't matter if they're powered or not. The platter is not "energized" or refreshed during operation like an SSD is. Your best bet is to have some kind of parity to identify and repair those bad bits.
Nothing in this article is talking about cold storage. And if we are talking about cold storage, as others gave pointed out, HHDs are also not a great solution. LTO (magnetic tape) is the industry standard for a good reason!
- Tape storage is the gold standard but it's just not realistically applicable to low scale operations or personal data storage usage. Proper long term storage HDDs do exist and are perfectly adequate to the job as i specified above and i can attest this from personal experience.