1mo ago

Tell me why I shouldn't use btrfs

About a year ago I switched to ZFS for Proxmox so that I wouldn't be running technology preview.

Btrfs gave me no issues for years and I even replaced a dying disk with no issues. I use raid 1 for my Proxmox machines. Anyway I moved to ZFS and it has been a less that ideal experience. The separate kernel modules mean that I can't downgrade the kernel plus the performance on my hardware is abysmal. I get only like 50-100mb/s vs the several hundred I would get with btrfs.

Any reason I shouldn't go back to btrfs? There seems to be a community fear of btrfs eating data or having unexplainable errors. That is sad to hear as btrfs has had lots of time to mature in the last 8 years. I would never have considered it 5-6 years ago but now it seems like a solid choice.

Anyone else pondering or using btrfs? It seems like a solid choice.

114 comments

Don't use btrfs if you need RAID 5 or 6.
The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.
https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices
- Or run the raid 5 or 6 separately, with hardware raid or mdadm
  Even for simple mirroring there's an argument to be made for running it separately from btrfs using mdadm. You do lose the benefit of btrfs being able to automatically pick the valid copy on localised corruption, but the admin tools are easier to use and more proven in a case of full disk failure, and if you run an encrypted block device you need to encrypt half as much stuff.
- I have no problem running it with raid 5/6. The important thing is to have a UPS.
  
  I've been running a btrfs storage array with data on raid5 and metadata I believe raid1 for the last 5 or so years and have yet to have a problem because of it. I did unfortunately learn not to fully trust the windows btrfs driver but was fortunately able to restore from backups and redownloading.
  I wouldn't hesitate to set it up again for myself or anybody else, and adding a UPS would be icing on the cake. (I added UPS to my setup this last summer)
- I've got raid 6 at the base level and LVM for partitioning and ext4 filesystem for a k8s setup. Based on this, btrfs doesn't provide me with any advantages that I don't already have at a lower level.
  Additionaly, for my system, btrfs uses more bits per file or something such that I was running out of disk space vs ext4. Yeah, I can go buy more disks, but I like to think that I'm running at peak efficiency, using all the bits, with no waste.
  
  btrfs doesn’t provide me with any advantages that I don’t already have at a lower level.
  Well yeah, because it's supposed to replace those lower levels.
  Also, BTRFS does provide advantages over ext4, such as snapshots, which I think are fantastic since I can recover if things go sideways. I don't know what your use-case is, so I don't know if the features BTRFS provides would be valuable to you.

A bit of topic; am I the only one that pronounces it "butterface"?
- Not anymore.
  
  You son of a removed, I'm in.
- Ah feck. Not any more.
- Isn't it meant to be like "better FS"? So you're not too far off.
  
  i call it "butter FS"
  
  I was meant to be Better FS, but it corrupted it to btrfs without noticing.
- Related, and I cannot help but read "bcachefs" as "removed café"
- I call it butter fuss. Yours is better.
- Similarly, I read bcachefs as BCA Chefs 😅

No reason not to. Old reputations die hard, but it's been many many years since I've had an issue.
I like also that btrfs is a lot more flexible than ZFS which is pretty strict about the size and number of disks, whereas you can upgrade a btrfs array ad hoc.
I'll add to avoid RAID5/6 as that is still not considered safe, but you mentioned RAID1 which has no issues.
- I've been vaguely planning on using btrfs in raid5 for my next storage upgrade. Is it really so bad?
  
  Check status here. It looks like it may be a little better than the past, but I'm not sure I'd trust it.
  An alternative approach I use is mergerfs + snapraid + snapraid-btrfs. This isn't the best idea for a system drive, but if it's something like a NAS it works well and snapraid-btrfs doesn't have the write hole issues that normal snapraid does since it operates on r/o snapshots instead of raw data.
  
  It's affected by the write-hole phenomenon. In BTRFS case that can mean that perfectly good old data might corrupt without any notice.

You shouldn't have abysmal performance with ZFS. Something must be up.
- What's up is ZFS. It is solid but the architecture is very dated at this point.
  There are about a hundred different settings I could try to change but at some point it is easier to go btrfs where it works out of the box.
  
  Since most people with decently simple setups don't have the described problem likely somethings up with your setup.
  Yes ifta old and yes it's complicated but it doesn't have to be to get a decent performance.
  
  What seems dated in its architecture? Last time I looked at it, it struck me as pretty modern compared to what's in use today.
  
  You have angered the zfs gods!
  
  You've been downvoted, but I've seen a fair share of ZFS implementations confirm your assessment.
  E.g. "Don't use ZFS if you care about performance, especially on SSD" is a fairly common refrain in response to anyone asking about how to get the best performance out of their solution.

Did you set the correct block size for your disk? Especially modern SSDs like to pretend they have 512B sectors for some compatibility reason, while the hardware can only do 4k sectors. Make sure to set ashift=12.
Proxmox also uses a very small volblocksize by default. This mostly applies to RAIDz, but try using a higher value like 64k. (Default on Proxmox is 8k or 16k on newer versions)
https://discourse.practicalzfs.com/t/psa-raidz2-proxmox-efficiency-performance/1694
- I'm thinking of bumping mine up to 128k since I do mostly photography and videography, but I've heard that 1M can increase write speeds but decrease read speeds?
  I'll have a RAIDZ1 and a RAIDZ2 pool for hot storage and warm storage.

btrfs has been the default file system for Fedora Workstation since Fedora 33 so not much reason to not use it.

Btrfs came default with my new Synology, where I have it in Synology's raid config (similar to raid 1 I think) and I haven't had any problems.
I don't recommend the btrfs drivers for windows 10. I had a drive using this and it would often become unreachable under load, but this is more a Windows problem than a problem with btrfs

I've been using single-disk btrfs for my rootfs on every system for almost a decade. Great for snapshots while still being an in-tree driver. I also like being able to use subvolumes to treat / and /home (maybe others) similar to separate filesystems without actually being different partitions.
I had used it for my NAS array too, with btrfs raid1 (on top of luks), but migrated that over to ZFS a couple years ago because I wanted to get more usable storage space for the same money. btrfs raid5 is widely reported to be flawed and seemed to be in purgatory of never being fixed, so I moved to raidz1 instead.
One thing I miss is heterogenous arrays: with btrfs I can gradually upgrade my storage one disk at a time (without rewriting the filesystem) and it uses all of my space. For example, two 12TB drives, two 8TB drives, and one 4TB drive adds up to 44TB and raid1 cuts that in half to 22TB effective space. ZFS doesn't do that. Before I could migrate to ZFS I had to commit to buying a bunch of new drives (5x12TB not counting the backup array) so that every drive is the same size and I felt confident it would be enough space to last me a long time since growing it after the fact is a burden.
- With version 2.3 (currently in RC), ZFS will at least support RAIDZ expansion. That should already help a lot for a NAS usecase.
- Btrfs Raid 10 reportedly is stable

The question is how do you get a bad performance with ZFS?
I just tried to read a large file and it gave me uncached 280 MB/s from two mirrored HDDs.
The fourth run (obviously cached) gave me over 3.8 GB/s.
- I have never heard of anyone getting those speeds without dedicated high end hardware
  Also the write will always be your bottleneck.
  
  I have similar speeds on a truenas that I installed on a simple i3 8100
  
  This is an old PC (Intel i7 3770K) with 2 HDDs (16 TB) attached to onboard SATA3 controller, 16 GB RAM and 1 SSD (120 GB). Nothing special. And it's quite busy because it's my home server with a VM and containers.
  
  I'm seeing very similar speeds on my two-HDD RAID1. The computer has an AMD 8500G CPU but the load from ZFS is minimal. Reading / writing a 50GB /dev/urandom file (larger than the cache) gives me:
  169 MB/s write
  254 MB/s read
  What's your setup?

One time I had a power outage and one of the btrfs hds (not in a raid) couldn't be read anymore after reboot. Even with help from the (official) btrfs mailinglist It was impossible to repair the file system. After a lot of low level tinkering I was able to retrieve the files, but the file system itself was absolutely broken, no repair process was possible. I since switched to zfs, the emergency options are much more capable.
- Was that less than 2 years ago? Were you using kernel 5.15 or newer?
  
  Yes that was may/june 23 and I was on a 6.x kernel

Didn't have any btrfs problems yet, infact cow saved me a few times on my desktop.
- Can you elaborate for the curious among us?
  
  btrfs + timeshift saved me multiple times, when updates broke random stuff.

One day I had a power outage and I wasn't able to mount the btrfs system disk anymore. I could mount it in another Linux but I wasn't able to boot from it anymore. I was very pissed, lost a whole day of work
- ACID go brrr
- When did this happen?
  
  I think 5 years ago, on Ubuntu

My setup is different to yours but not totally different. I run ESXi 8, and I started to use BTRFS on some of my VM's.
I had a power failure, that was longer than the UPS could handle. Most of the system shutdown safely, a few VM's did not. All of the EXT4 VM's were easily recovered (including another one that was XFS). TWO of the BTRFS systems crashed into a non recoverable state.
Nothing I could do to fix them, they were just toast. I had no choice but to recover using backups. This made me highly aware that BTRFS is still not a reliable FS.
I am migrating everything from BTRFS to something more stable and reliable like EXT4. It's simply not worth the headache.
- When did this happen?
  
  It was only a few weeks ago (maybe 4). Systems are all kept up to date with ansible. Most are Debian but there are few Ubuntu. The two that failed were both Debian.
  Granted both that failed have high [virtual] disk usage compared to the other VM's. I cannot remember the failure now, but lots of searching confirmed that it was likely unrecoverable (they could boot, but only into read only). None of the btrfs-check "dangerous" commands could recover it, spitting out tons of errors about mismatching somethings (again, forgotten the error).
- I had almost exactly the same thing happen.

If it didn't give you problems, go for it. I've run it for years and never had issues either.

What kind of disks, and how is your ZFS set up? Something seems amis here.

Not proxmox-specific, but I've been using btrfs on my servers and laptops for the past 6 years with zero issues. The only times it's bugged out is due to bad hardware, and having the filesystem shouting at me to make me aware of that was fantastic.
The only place I don't use zfs is for my nas data drives (since I want raidz2, and btrfs raid5 is hella shady) but the nas rootfs is btrfs.

I run it now because I wanted to try it. I haven't had any issues. A friend recommended it as a stable option.

Using it here. Love the flexibility and features.

Meh. I run proxmox and other boot drives on ext4, data drives on xfs. I don't have any need for additional features in btrfs. Shrinking would be nice, so maybe someday I'll use ext4 for data too.
I started with zfs instead of RAID, but I found I spent way too much time trying to manage RAM and tuning it, whereas I could just configure RAID 10 once and be done with it. The performance differences are insignificant, since most of the work it does happens in the background.
You can benchmark them if you care about performance. You can find plenty of discussion by googling "ext vs xfs vs btrfs" or whichever ones you're considering. They haven't changed that much in the past few years.
- but I found I spent way too much time trying to manage RAM and tuning it,
  I spent none, and it works fine. what was your issue?
  
  I have four 6tb data drives and 32gb of RAM. When I set them up with zfs, it claimed quite a few gb of RAM for its cache. I tried allocating some of the other NVMe drive as cache, and tried to reduce RAM usage to reasonable levels, but like I said, I found that I was spending a lot of time fiddling instead of just configuring RAID and have it running just fine in much less time.
- Proxmox only supports btrfs or ZFS for raid
  Or at least that's what I thought
  
  ext4 and others too.

Do you rely on snapshotting and journaling? If so backup your snapshots.
- Why?
  I already take backups but I'm curious if you have had any serious issues
  
  Are you backing up files from the FS or sre you backing up the snapshots? I had a corrupted journal from a power outage that borked my install. Could not get to the snapshots on boot. Booted into a live disk and recovered the snapshot that way. Would've taken hours to restore from a standard backup, however it was minutes restoring the snapshot.
  If you're not backing up BTRFS snapshots and just backing up files you're better off just using ext4.
  https://github.com/digint/btrbk

Used it in development environment, well I didn't need the snapshot feature and it didn't have a straightforward swap setup, it lead to performance issues because of frequent writes to swap.
Not a big issue but annoyed me a bit.

Btrfs only has issues with raid 5. Works well for raid 1 and 0. No reason to change if it works for you
- I think it has more issues than just with raid 5 &6!
- It is stable with raid 0,1 and 10.
  Raid 5 and 6 are dangerous

For my jbod array, I use ext4 on gpt partitions. Fast efficient mature.
For anything else I use ext4 on lvm thinpools.
- That doesn't do error detection and correction nor does it have proper snapshots.
  
  There is error detection, crc checks and lvm does snapshots and offline deduplication
  However I run sha256 checks offline and PAR files for forward error correction

I am using btrfs on raid1 for a few years now and no major issue.
It's a bit annoying that a system with a degraded raid doesn't boot up without manual intervention though.
Also, not sure why but I recently broke a system installation on btrfs by taking out the drive and accessing it (and writing to it) from another PC via an USB adapter. But I guess that is not a common scenario.
- The whole point of RAID redundancy is uptime. The fact that btrfs doesn't boot with a degraded disk is utterly ridiculous and speaks volumes of the developers.

Raid 5/6, only bcachefs will solve it
- Btrfs Raid 5 and raid 6 are unstable and dangerous
  Bcachefs is cool but it is way to new and isn't even part of the kernel as of yet.
  
  https://en.wikipedia.org/wiki/Bcachefs it was added as of Linux 6.7
  Edit: and I've said raid 5/6 as what troubles btrfs have so you proved my point while trying to explain to me that I'm not right

btrfs raid subsystem hasn't been fixed and is still buggy, and does weird shit on scrubs. But fill your boots, it's your data.

114 comments