Tuesday, September 25, 2012

BTRFS vs ZFSonLinux: How do they compare?

  • Integration with Linux
    • ZFS: Not integrated. Has its own configuration database (not /etc/fstab), has its own boot order for mounting filesystems (not definable by you), cannot be told to bring a filesystem up after iSCSI comes up or down before iSCSI goes down.
    • BTRFS: It's just another Linux filesystem as far as the system is concerned. You bring a pool up by mounting it (preferably by label) in /etc/fstab and can define the mount order so it comes up after iSCSI.
  • Snapshots
    • ZFS: Full snapshot creation and removal capabilities, well exploited by the FreeBSD port 'zfs-periodic'. Snapshots appear in a special "dot" directory rather than cluttering up the main filesystem. This script is relatively easy to port to Linux.
    • BTRFS: Snapshots are created as "clones" of subvolumes, and destroyed as if they were subvolumes. They can be created either read-write or read-only.
  • RAID: Both of these use filesystem-level RAID where filesystem objects are stored redundantly, either as entire clones (RAID1) or, in the case of ZFS, via RAID
    • ZFS: Raid1 (mirroring) and RaidZ (similar to RAID5, except that it never does partial-stripe writes because it does variable stripe size -- the size of an object is the size of a stripe). Note that due to ZFS's COW implementation, an update to a RAID stripe cannot be corrupted by a power loss halfway through the write (see: RAID5 write hole)-- the old copy of the data (prior to the start of the write) is instead accessed when power comes back on.
    • BTRFS: Raid1 (mirroring). BTRFS currently has nothing like RaidZ. Note that putting a BTRFS filesystem on top of a software mdadm RAID5 will not give you the same reliability and performance as RaidZ, since you will still have the random write hit of partial-stripe writes and will still have the RAID5 write hole where if a stripe update fails due to power loss halfway thru the stripe write, the entire stripe is corrupted.
  • Portability
    • ZFS: A ZFS filesystem can be read / written on: Linux (via either ZFS/Fuse or ZFSonLinux), FreeBSD, OpenIndiana, and MacOS (via Zevo). Requires extra 3rd party software to be installed on Linux and MacOS, comes standard with FreeBSD and OpenIndiana.
    • Linux: Any recent Linux distribution (one with a 3.x vintage kernel) has BTRFS built in. Your BTRFS pools will be immediately available when you upgrade to a newer kernel or a newer Linux distribution, with no need to install any additional software. However, BTRFS doesn't run on any other OS.
  • Stability
    • On Linux, both BTRFS and ZFS are listed as "experimental". ZFSonLinux uses SEL (the Solaris Emulation Layer) as a "shim" between ZFS proper and Linux. Unfortunately this is sort of like nailing jello to a tree, while the underlying Linux block layer API hasn't changed in years, locking inside that block layer API has been in constant turmoil ever since the 2.6.30 timeframe as the last vestiges of the Big Kernel Lock were ferreted out and sent to the great bit bucket in the sky. The end result is that code that *used* to work may or may not cause deadlocks or strange races that cause an oops with current Linux kernels -- *UNLESS* it was developed as part of that current Linux kernel, as BTRFS is, in which case the person who changes the locks is responsible to make sure that all other kernel modules that are part of the next kernel release change their locks to match.
    • Summary: On Linux, this is a tie. BTRFS is under rapid development. ZFS is attempting to nail jello to a tree from outside the Linux kernel. Use for production data of either system on Linux is not recommended. If you want a production server running a production-quality modern snapshotting filesystem, use ZFS on FreeBSD.
Final summary:

If you must use Linux, and you must have a modern snapshotting filesystem, and you can live with a RAID1 limitation on data redundancy, I would strongly recommend going with BTRFS. The reason for this is that BTRFS is only going to get better on Linux, while ZFS is always going to be fighting the nail-jello-to-a-tree issue where Linux keeps changing underneath it and breaking things in weird ways. Unless ZFS is included as part of the Linux kernel -- and Oracle's lawyers will never allow changing the license to GPL in order to allow that -- there simply is no way ZFS will ever achieve stability except with specific kernel versions shipped with specific distributions. And even there I'm dubious.

If you need the stability of ZFS, I strongly recommend using FreeBSD and not using Linux. I have personal experience dealing with the issues that come with supporting an emulation layer on top of the Linux block layer, including dealing with some deadlocks and races caused by locking changes inside recent kernels that caused a six-week delay in the release of an important product, and I honestly cannot say that any current ZFSonLinux implementation will continue to work with the next kernel revision. I can reliably say that BTRFS will work with the next kernel revision. While production servers don't change kernel revisions often, only once every three or four years, if the next version of the server OS doesn't happen to be one that is well supported by ZFSonLinux's then-current SEL implementation, you have problems.

So: Linux -- BTRFS. If you need the functionality of ZFS -- FreeBSD. Enough said on that.

Sunday, September 23, 2012

Crack-smoking Linux zealots

It seems that every year, like the swallows flocking back to Capistrano, you get yet another burst of Linux zealots claiming that this is the year for the Linux desktop. As you know, I've said good things about Gnome 3 and its usability by mere mortals. Linux geeks whine about how they can't customize it blah de blah blah, but mere mortals don't customize their window manager -- other than setting the desktop background to a photo of their kids or grandkids, all they care about is whatever applications they're wanting to run. And from that perspective, Gnome 3 works just fine. But: An operating system is more than a window manager. And that is where Linux still is Utter Fail on the desktop.

Okay, let me tell you a story. I installed Fedora 17 on a 64GB SSD to manage my RAID arrays that serve data to my home network. For grins and giggles I installed Thunderbird and told it about a couple of my email accounts. I clicked through on a link in one of those email accounts that went to a page that had a video embedded. Well, that *would* have a video embedded, on Windows or MacOS. On Linux? Just a blank box and a message about a plugin that simply doesn't exist for Linux that needs to be installed, probably a Quicktime plugin. Now I'm sure you're saying, "but that's not Linux's fault, that's Apple's fault for patenting the algorithm used for that plugin!" But see, the deal is, end users don't care whose fault it is. All they know is that they can't see that embedded video with Linux, and they can see it with Windows or MacOS. BTW, same applies to playing MP3 files -- they simply won't play. Again, a patent issue, but Joe Sixpack doesn't care why his music files won't play on his Linux system -- all he cares about is that they don't play on his Linux system. End of game.

Now let's look at another issue, one where the excuse of patents does not hold. The SSD came out of an old Asus Aspire One netbook, which I'd put it in as part of an attempt to build a mobile GPS system, one of a couple of failed attempts that eventually ended up successful with the iPad, thus rendering the need for an SSD in the Aspire One obsolete. So I grabbed an old 160GB hard drive to put into the Aspire One to take the place of the SSD, and imaged it using the rescue disks that I'd purchased to put an image onto the SSD in the first place. Except it didn't boot. Okay, no big deal, I'll just haul a Windows XP rescue disk over there ... err... I don't happen to have one burned? No biggie, I'll just burn one. And since I have my beautiful Linux desktop sitting here, I'll just use that to do so.

So I pop the blank CD/RW disk into the Asus SATA CD/DVD writer, and attempt to burn it using the stock Brasero disk burner... and it's utter fail. Won't do it. In fact, locks up the system for 30 seconds because it locks up the SATA bus while not doing it. What the BLEEP?! So I go check Google, and it looks like Brasero has been broken on Fedora FOR YEARS, it'll write DVD's (sometimes) but CD's? No way! So I try a couple of other programs, and have similar results. Then I go down to the underlying 'wodim' program and again the same results. At that point I'm, like, "okay, maybe it's broken CD recorder hardware". So I pull out an external USB CD recorder drive -- one that I recently used on another system that doesn't have an internal DVD-RW drive -- and attempt to use that. Same result.

So maybe it's a bad disk? So I sling the disk into my MacBook Pro, erase it, and record it. It Just Works. As would have been the case, I'm 99% certain, if I had been trying to do this under Windows 7 -- it would have Just Worked.

And lest you think this is just my physical chunk of hardware and the particular version of Linux here at home, I have similar (lack of) success recording CD-R's at work on my Centos 6.3 desktop system. There, again, I have to record CD's on my MacBook Pro because Linux simply refuses to do it. Which infuriates me, because I could write CD's for *years* under Linux, but it's broken now, and nobody seems to care (at least, it's been broken in every release of Fedora since Fedora 10, which was, what, four years ago? Five years ago?), so ...

So let's recap: I can't view videos, I can't play music, and I can't burn CD-R's. I *can* do all of those things on Windows or MacOS. What Linux does well is serving data to networks. It does that very well, my Linux box serves as the Time Machine backup for my MacBook Pro as well as being the central data repository for the various Windows laptops I have hanging around. But the desktop? When it can't do simple desktop tasks that Windows and MacOS have done for literally years? Crack. That's my only explanation for why anybody would ever make a ludicrous statement like "this is the year of desktop Linux!" given the current state of Linux.


Wednesday, September 12, 2012

This is the droid you're looking for

I have a new toy now. I ditched my aging iPhone 4 upon completion of its contract (and ported its number into Google Voice), and now have a brand new Samsung Galaxy S3.

So far it's mostly all good. Battery life is bad, but we already knew that. I tried several different home screen programs but I'm sticking with TouchWiz for now because the updated one for the Galaxy S3 works as well as anything else I tried, even the backported Jellybean launcher. It has lousy reception inside company HQ but so did the iPhone 4, just an AT&T thing I guess (my Verizon iPad has great reception inside company HQ). I'm still looking for a clean solution for automatically syncing my photos into iPhoto, but iSyncr is doing a reasonably good job of getting them onto my Macbook so I'm not too displeased.

Thus far I've found substitutes for everything I did on my iPhone except one: There is no good offline GPS program like the Magellan program that I used on the iPhone. Supposedly TomTom is going to be remedying that soon. We'll see.

So anyhow, I have found one bug in the Galaxy S3's ICS Android version: It does not handle exFAT very well. I found this out the hard way when my 64GB microSD card quit working and reported, "Damaged SD Card". Indeed, checking the Internet, it appears that random exFAT corruption is an epidemic on the Galaxy S3. This afflicts any microSD over 32GB, since Microsoft officially says FAT32 won't go over 32GB. This is, of course, a lie -- FAT32 is quite capable of handling terabyte-sized filesystems -- but because Microsoft enforces this limit in all their filesystem tools, nobody knew it was a lie until they actually looked at FAT32 and realized hey, this will work with bigger filesystems! (Though there is still that nasty 4GB limit on file size to contend with).

So how did I resolve this problem? First, I put the flash into a Windows 7 laptop and ran chkdisk on it. This found and fixed some problems. But when I put it back into the Galaxy S3 it *still* said "Damaged SD Card" despite the fact that Windows 7 said it was clean. So I resolved to reformat as FAT32. I copied the data off, and then had to go find a tool that would actually format a 64GB microSD card as FAT32 since the Windows disk manager won't do so: EaseUS Partition Master.

At that point it was just a matter of copying the data back on, which was very... very... slow since Windows operates SD cards in sync mode. As in, an hour slow. I know where the async flag lives in Windows and could have flipped it, but it was trash night so I did chores around the house instead. At the end of the process I inserted the microSD into the Sammy and... no more "Damaged SD Card".

Executive summary: If you buy a microSD card with greater than 32GB capacity, it is likely that it is formatted with Microsoft's proprietary exFAT filesystem and will not work well Android unless you reformat it, even if it appears to work correctly at first. exFAT support is not supported well because it is patented by Microsoft and thus does not have the magic of dozens of eyes of Open Source developers noticing and fixing bugs in it. So reformat it using the EaseUS tool above (NOT the internal Samsung formatter, it'll put the buggy exFAT filesystem back onto it) *before* you put stuff on it. Otherwise you'll be going through this whole time-consuming dance yourself sooner or later. Fun, it was not.