Eating the seed corn: Red Hat Software, CentOS, and the end of an era

Or: Why we are migrating to Ubuntu Linux.

So, why did Linux win the server wars? Why is virtually every cloud server running Linux now, rather than some variety of Windows Server? Why are probably the majority of corporate web servers running Linux now, and a large variety of other internal application servers are running Linux?

Basically, Linux won the server wars because Linux won the poor college student market. Poor college students couldn't afford a copy of Windows Server to run on their own computer. They could afford a copy of Linux to run on their own computer. And twenty years later, they are the decision makers in IT departments who decide what technology to use.

So anyhow: After being one of those poor college students some years ago I'm now one of those decision makers. For example, I spec'ed and got quotes for $16,000 worth of server equipment last week as well as $1500 worth of software, and over the next few years will be doing the same for a six-figure amount of equipment and software. Granted, this isn't huge by IT terms, but multiply me by thousands, and it adds up quickly.

And one of the things I spec'ed was the embedded OS in our appliance. In 2010 I chose to use CentOS Linux for the embedded distribution in our appliance for a number of reasons:

1. Licensing costs. The licensing terms for Red Hat Enterprise Linux really are oriented around servers. Their per-server costs would make the appliance unprofitable. Furthermore, we're using an extremely stripped down Linux distribution for our appliance. There really is no value we would get from Red Hat support. So we went with a community distribution (CentOS) rather than a commercial distribution(RHEL or SUSE) to reflect the value that we get from the support organization around the distribution.
2. Stream of security patches. We don't have a team to watch CVE's and figure out whether they apply to our stripped-down Linux distribution. Instead, we watch the stream of RPM's coming out of the distribution and apply the ones applicable to our distribution on a regular basis.
3. Length of support. Ten years is a long time. It means we don't have to re-base our software that runs on top of the embedded OS to a new Linux distribution very often. This means less work and more play. Playing with new technologies to add to our product is a lot more fun than the grunt work of porting our software to a new Linux distribution.
4. Continuity with earlier versions of Red Hat Enterprise Linux. If you knew how to set up advanced networking on EL5, you know how to set it up on EL7 (with some caveats such as having to disable NetworkManager first). This made the job of re-basing our software considerably easier than with distributions that moved things around in major ways with each release.
5. A commitment in writing and in all major press outlets by Red Hat Software, when they acquired the CentOS project in 2014, to continue the mission of the CentOS project unchanged, which led to transitioning from Centos 6 to Centos 7 when it arrive and stabilized.

Fundamentally, Red Hat Software has broken all of the above with recent decisions such as the decision to discontinue CentOS 8 as of the end of 2021.

1. CentOS Stream is not a community distribution. It is a beta test stream for RHEL. As such, it is not an appropriate base for an embedded appliance distribution. So we will have to move to another community distribution, and Ubuntu is the community distribution that has the highest mindshare right now.
2. One reason why CentOS Stream is not a community distribution is that CentOS Stream will not be a reliable source of security patches. Yes, I understand that Facebook has their own internal distribution based on CentOS Stream. We are not Facebook. We do not have infinite money like Facebook. We do not have a team to watch the list of RPM's and apply only the finalized tested versions to our appliance distribution. Yet another reason why CentOS Stream is not appropriate for us.
3. One thing that kept us on CentOS rather than on Ubuntu LTS was that CentOS releases were supported for twice as long as Ubuntu LTS releases. Since that's no longer true, there's no reason to stay with a RHEL-derived distribution.
4. Red Hat Software broke significant continuity with earlier versions of RHEL with Enterprise Linux 8. For example, they removed all Java applet containers from the distribution other than their own JBOSS applet container. You use Tomcat? Tough. You'll have to install it from scratch and keep it updated yourself when security patches happen. And those who use the BTRFS filesystem are completely left out.
5. And finally, Red Hat Software has proven that they will break their commitments. So.

And Red Hat's explanation? "It was putting price pressure on Red Hat’s ability to capture some of the value that we create."

Uhm, no. Because the users of CentOS, by and large, are not going to buy Red Hat Enterprise Linux. Buying RHEL does not comport with their business model if they're a business, or with their budget if they're a poor college student. They are going to move to another community distribution, either another rebuild of RHEL such as Oracle Linux or Rocky Linux, or to a different distribution altogether such as Debian or Ubuntu.

So why are we choosing Ubuntu LTS rather than some other distribution?

1. Oracle Linux: I've evaluated Oracle Linux. They've made sufficient changes to make it difficult to transition from CentOS, plus have broken a few things along the way. And they don't (can't) fix the fact that Enterprise Linux 8 broke significant continuity with earlier versions of RHEL. They can't do that and maintain compatibility with RHEL 8.
2. Then there's the general problem with all of the RHEL rebuilds: mindshare and market share is declining drastically. CentOS is really bad as a desktop operating system, so the students we get in as interns, if they know Linux at all, usually know Ubuntu Linux or one of its derivatives. Combined with its derivatives, that makes Ubuntu the most popular desktop Linux right now. For desktop Linux use, WSL has made Ubuntu the most common Linux on the desktop in our office despite the fact that we work with CentOS or CentOS-derived distributions as our target.
3. Debian Linux is quite stable, but if security fixes require effort to backport to the previous stable release, they usually aren't backported. This makes it hard for us to keep our product secure without transitioning to a newer distribution.
4. Fedora -- it moves too fast. Decent desktop OS, but not a good fit for an embedded appliance.
5. The ubiquity of Ubuntu. It is available as a supported option, and fully cloud-aware, on all major cloud platforms -- Google, Microsoft, and Amazon. To create a virtual machine with many of the other Linux distributions you must first locate a third-party image, then install that possibly untrusted third-party image. And often the third-party image is not cloud aware -- e.g. it doesn't have cloud-init installed or the native cloud tools installed, making it hard to bring up new virtual machines in an automated fashion. Ubuntu is a Tier 1 supported distribution everywhere that matters.

Thus Ubuntu.

In general, the mindshare issue -- where Red Hat's mindshare amongst potential employees is declining rapidly while Ubuntu's mindshare is becoming near ubiquitous -- makes it almost a no-brainer for us. Instead of attempting to port our software stack from Centos 7 to a RHEL 8 clone as Centos 7 approaches end-of-life, we will instead port it to Ubuntu 20.04 LTS. We use a configuration management tool to actually create our appliance (we plop the deliverables onto the CM server and it configures a copy of the base OS with our changes), and the CM tool we use hides most distribution details from us, so the job of porting is mostly going to be one of adjusting package names and occasional file locations in the manifest used to create the appliance. Furthermore, we avoid the Tomcat issue. Ubuntu ships with Tomcat, while RHEL 8 does not. Granted, that's a fairly minor issue (we already have things in our manifest that manually install various software) but why make work for ourselves?

And our student interns? The ones that aren't already Ubuntu users will become Ubuntu users. And ten years from now, they will be specifying Ubuntu Linux rather than Red Hat Enterprise Linux for their corporation's deployments. Because that's how it works. If you seed your field, crops will rise. If you instead eat your seed corn, as Red Hat's current IBM pointy headed management is doing ... well, you eat well for this quarter. But good luck reaping a crop at the end of the growing season.

- ELG

Red Hat Software has phone company syndrome

Specifically: They don't care, because they don't have to.

Google "jboss sucks." You get about 946,000 results. Google "tomcat sucks". You get about 260,000 results.

So which one did Red Hat Software omit from Red Hat Enterprise Linux 7, after it had been in the distribution for close to two decades?

Hint: It's the one that sucks less.

Yeah, Red Hat kept the one that sucks more (jboss) and dumped the one that sucks less (tomcat). Because they want to push their bloated buggy application server that nobody really likes (JBoss) and if that requires breaking everybody's migration path from RHEL7 to RHEL8, well.They're Red Hat Software / IBM. They don't care. They don't have to.

Of course, we all know what happened the last time that IBM tried to shove something down people's throats, when they introduced the IBM PS/2 and OS/2. It turns out that people do *not* willingly abandon their old stuff just because the incumbent near-monopoly decides to push their own proprietary junk. Competitors soon outcompeted IBM, and now they no longer make personal computers. Because it turns out that customers will go to the competition if you do customer-hostile things.

Like Ubuntu.

Which is likely going to be what I standardize on going forward, because really, if Red Hat is going to break the world every time they do a new release, why bother?

- ELG

The quest for an integrated storage stack

In prior posts I've mentioned the multitude of problems with the standard Linux storage stack. It's inflexible -- once you've set up a stack (usually LV->VG->PV->MD->BLOCK) and opened a filesystem on it, you cannot modify it to, e.g., add a replication layer to the stac. It lacks the ability to do geographic replication in any reasonable fashion. The RAID layer in particular lacks the ability to write to (and replay) a battery-backed RAM cache to deal with the RAID 5 write hole (which, despite its name, also applies to other RAID levels and results in silently corrupted data). Throw iSCSI into this equation to provide block devices to virtual machines and, potentially, to do replication to block devices on other physical machines, and things get even more complex.

One method that has been proposed to deal with these issues is to simply not use a storage stack at all. Thus we have ZFS and BTRFS, which attempt to move the RAID layer and logical volume layers into the filesystem. This certainly solves the problem of corrupted data, but at a significant penalty in terms of performance, especially on magnetic media where the filesystem swiftly becomes fragmented. As a result running virtual machines using "block devices" that are actually files on a BTRFS filesystem results in extremely poor "disk" performance on the virtual machines. A file on a log-based subsystem is simply a poor substitute for an extent on a block device. Furthermore, use of these filesystems for databases has proven to be woefully slow compared to using a normal filesystem like XFS on top of a RAID-10 layer.

The other method that has been to abandon the Linux storage stack except as a provider of individual block devices and instead layer a distributed system like Ceph on top of it. My tests with Ceph have not been particularly promising. Performance of Ceph block devices at an individual virtual machine level were abysmal. There appears to be three reasons for this: 1) Overly pessimistic assumptions about writes on the part of Ceph, 2) The inherent latencies involved in a distributed storage stack, and 3) the fact that Ceph reads/writes via XFS filesystems layered on top of block devices, rather than to extents on raw block devices. For the latter, in my experience you will see *at least* a 10% degradation in virtual machine block device performance if the block device is implemented as a file on top of XFS rather than directly to a LVM extent.

In both cases, I wonder if we are throwing out the cart because the horse has asthma. I've worked as a software engineer for two of the pioneers of Linux-based storage -- Agami Systems, which did a NAS device with an integrated storage system, and Intransa Inc., which did scalable iSCSI storage systems with an integrated block storage subsystem. Both suffered the usual fate of pioneers -- i.e., face down dead with arrows in the back, though it took longer with Intransa than with Agami. Both wrote storage stacks for Linux which solved most of the problems of the current Linux storage stack, though each solved a different subset of those problems. There are still a significant number of businesses which do not need the expense and complexity of a full OpenStack data center in order to solve their problems, but which do need things like, e.g., logged geographic replication to replicate their data to an offsite location, something which Intransa solved ten years ago (but which, alas, died with Intransa), or real-time snapshots of virtual machine block devices at the host device level, or ...

In short: Despite the creation of distributed systems like CEPH and integrated storage management filesystems like BTRFS, there is a significant need for an integrated storage stack for Linux -- one that allows flexibility in configuring both block devices and network filesystems, which allows for easy scalability and management, which has modern features such as logged geographic replication, battery backed RAM cache support (or at least fast SSD log device support at the MD layer), and allows dynamic insertion of components into the software stack much as you could create a replication layer in the Intransa StorStac and have it sync then replicate to a remote device without ever unmounting any filesystem or making the iSCSI target inaccessible. There is simply a large number of businesses which just don't need the expense and complexity of a full OpenStack data center, which indeed don't need more than a pair of iSCSI / NAS storage appliances (a pair in order to handle replication and snapshotting), and the current Linux storage stack lacks fundamental functionality that was implemented over a decade ago but never integrated into Linux itself. It may not be possible to bring all the concepts that Agami and Intransa created into Linux (though I'll point out that all of Intransa's patents are now owned by a patent entity that allows free use for Open Source software), but we should attempt to bring as many of them as possible into the standard Linux storage stack -- because the cloud is the cloud, but most smaller businesses have no need for the cloud, they just need reliable local storage for their local physical and virtual machines.

-ELG

The Linux storage stack: is it ready for prime time yet?

I've been playing with LIO quite a bit since rolling it into production for Viakoo's infrastructure (and at home for my personal experiments). It works quite a bit differently from the way that Intransa's Storstac worked. Storstac created a target for each volume being exported, while with LIO you have a single target that exports a LUN for each volume being exported. The underlying Linux kernel functionality is there to create a target per volume, but the configuration infrastructure is oriented around the LUN per volume paradigm.
Not a big deal, you might say. But it does make a difference when connecting with the Windows initiator. With the Windows initiator, the target per volume paradigm allows you to see what volume a particular LUN is connected to (assuming you give your targets descriptive names, which StorStac does). This in turn allows you to easily coordinate management of a specific target. For example to resize it you can offline it in Windows, stop exporting it on the storage service, rescan in Windows, expand the volume on your storage server, re-export it on your storage server, then online it in Windows and expand your filesystem to fill the newly opened up space. Still, this is not a big deal. LIO does perform quite well and does have the underlying capabilities that can serve the enterprise. So what's missing to keep the Linux storage stack from being prime time? Well, here's what I see:

1. Ability to set up replication without taking down your filesystems / iSCSI exports. Intransa StorStac had replication built in, you simply set up a volume the same size on the remote target, told the source machine to replicate the volume to the remote target, and it started replicating. Right now replication is handled by DRBD in the Linux storage stack. DRBD works very well for its problem set -- local area high availability replication -- but to set up a replication after the fact on a LVM volume simply isn't possible. You have to create a drbd volume on top of a LVM volume, then copy your data into the new drbd volume. One way around this would be to automatically create a drbd volume on top of each LVM volume in your storage manager, but that adds overhead (and clutters your device table) and presents problems for udev at device assembly time. And still does not solve the problem of:
2. Geographic replication: StorStac at one time had the ability to do logged replication across a WAN. That is, assuming that your average WAN bandwidth is high enough to handle the number of writes done during the course of a workday, a log volume will collect the writes and ship them across the WAN in the correct order to be applied at the remote end. If you must do a geographic failover due to, say, California falling into the sea, you lose at most whatever log entries have not yet been applied at the remote end. Most filesystems will handle that in a recoverable manner as long as the writes are being applied in the correct order (which they are). DRBD *sort of* has the ability to do geographic replication via an external program, "drbd-proxy", that functions in much the same way as StorStac replication (that is, it keeps a log of writes in a disk volume and replays them to the remote server), but it's not at all integrated into the solution and is excruciatingly difficult to set up (which is true of drbd in general).
3. Note that LVM also has replication (of a sort) built in, via its mirror capability. You can create a replication storage pool on the remote server as a LVM volume, export it via LIO, import it via open-iscsi, create a physical volume on it, then create mirror volumes specifying this second physical volume as the place you want to put the mirror. LVM also does write logging so can handle the geographic situation. The problem comes with recovery, since what you have on the remote end is a logical volume that has a physical volume inside it that has one or more logical volumes inside it. The circumlocutions needed to actually mount and use those logical volumes inside that physical volume inside that logical volume are non-trivial, it may in fact be necessary to mount the logical volume as a loopback device then do pvscan/lvscan on the loopback device to get at those volumes. It is decidedly *not* as easy as with StorStac, where a target is a target, whether it's the target for a replication or for a client computer.

So clearly replication in the Linux storage stack is a mess, nowhere near the level of ease of use or functionality as the antiquated ten-year-old Intransa StorStac storage stack. The question is, how do we fix it? I'll think about that for a while, but meanwhile there's another issue: Linux doesn't know about SES. This is a Big Deal for big servers. SES is the SCSI Enclosure Services protocol that is implemented by most SAS fanout chips and allows control of, amongst other things, the blinky lights that can be used to identify a drive (okay, so mdmonitor told you that /dev/sdax died, where the heck is that physically located?!) . There are basically two variants extant nowadays, SAS and SAS2, that are very slightly different (alas, I had to modify StorStac to talk to the LSI SAS2X24 expander chip which very slightly changed a mode page that we depended upon to find the slot addresses). Linux itself has no notion that things like SAS disk enclosures even exist, much less any idea how to blink lights in them.

And finally, there is the RAID5/RAID6 write hole issue. Right now the only reliable way to have RAID5/RAID6 on Linux is with a hardware RAID controller that has a battery-backed stripe cache. Unfortunately once you do this, you can no longer monitor drives via smartd to catch failures before they happened (yes, I do this, and yes, it works -- I caught several drives in my infrastructure that were doing bad things before they actually failed and replaced them before I had to deal with a disaster recovery situation), you can no longer take advantage of your server's gigabytes of memory to keep a large stripe cache so that you don't have to keep thrashing the disks to load stripes in the case of random writes (if the stripe is already in cache, you just update the cache and write the dirty blocks back to the drives, rather than have to reload the entire stripe) and you can also no longer take advantage of the much faster RAID stripe computations allowed by modern server hardware (it's amazing how much faster you can do RAID stripe calculations with a 2.4Ghz Xeon than you can with an old embedded MIPS processor running at much slower speeds). In addition it is often very difficult to manage these hardware RAID controllers from within Linux. For these reasons (and other historical issues not of interest at the moment) StorStac always used software RAID. Historically, StorStac used battery-backed RAM logs for its software RAID to cache outstanding writes and recover from outages, but such battery-backed RAM log devices don't exist for modern commodity hardware such as the 12-disk Supermicro server that's sitting next to my desk. It doesn't matter anyhow, because even if it did exist, there's no provision in the current Linux RAID stack to use it.

So what's the meaning of all this? Well, the replication issue is... troubling. I will discuss that more in the future. On the other hand, things like Ceph are handling it at the filesystem level now, so perhaps block level replication via iSCSI or other block-level protocols isn't as important as it used to be. For the rest, it appears that the only thing lacking is a management framework and a utility to handle SES expander chips. The RAID[56] write hole is troublesome, but in reality data loss from that is quite rare, so I won't call it a showstopper. It appears that we can get 90% of what the Intransa StorStac storage stack used to do by using current Linux kernel functionality and a management framework on top of that, and the parts that are missing are parts that few people care about.

What does that mean for the future? Well, your guess is as good as mine. But to answer the question about the Linux storage stack: Yes, it IS ready for prime time -- with important caveats, and only if a decent management infrastructure is written to control it (because the current md/lvm tools are a complete and utter fail as anything other than tools to be used by higher-level management tools). The most important caveat being, of course, that no enterprise Linux distribution has been released yet with LIO (I am using Fedora 18 currently, which is most decidedly *not* what I want to use long-term for obvious reasons). Assuming that Red Hat 7 / Centos 7 will be based on Fedora 18, though, it appears that the Linux storage stack is the closest to being ready for prime time as it's ever been, and proprietary storage stacks are going to end up migrating to the current Linux functionality or else fall victim to being too expensive and fragile to compete.

-ELG

The killer app for virtualization

The killer application for virtualization is... running legacy operating systems.

This isn't a new thought on my part. When I was designing the Intransa StorStac 7.20 storage appliance platform I deliberately put virtualization drivers into it so that we could run Intransa StorStac as a virtual appliance on some future hardware platform not supported by the 2.6.32 kernel. And yes, that works (no joke, I tried it out of course, the only thing that didn't work was sensors but if Viakoo ever wants to deliver a virtualized IntransaBrand appliance I know how to fix the sensors). My thought was future-proofing -- I could tell from the layoffs and from the unsold equipment piled up everywhere that Intransa was not long for the world, so I decided to leave whoever bought the carcass a platform that had some legs on it. So it has drivers for the network chips in the X9 series SuperMicro motherboards (Sandy/Ivy Bridge) as well as the virtualization drivers. So there's now a pretty reasonable migration path to keep StorStac running into the next decade... first migrate it to Sandy/Ivy Bridge physical hardware, then once that's EOL'ed migrate it to running on top of a virtual platform on top of Haswell or its successors.

But what brought it to mind today was ZFS. I need some of the features of the LIO iSCSI stack and some of the newer features of libvirtd for some things I am doing, so have ended up needing to run a recent Fedora on my big home server (which is now up to 48 gigabytes of memory and 14 terabytes of storage). The problem is that two of those storage drives are offsite backups from work (ZFS replication, duh) and I need to use ZFS to apply the ZFS diffsets that I haul home from work. That was not a problem for Linux kernels up to 3.9, but now Fedora 18/19 have rolled out 3.10, and ZFSonLinux won't compile against the 3.10 kernel. I found that out the hard way when the new kernel came in, and DKMS spit up all over the floor because of ZFS.

The solution? Virtualization to the rescue! I rolled up a Centos 6.4 virtual machine, pushed all the ZFS drives into it, gave it a fair chunk of memory, and voila. One legacy platform that can sit there happily for the next few years doing its thing, while the Fedora underneath it changes with the seasons.

Of course that is nothing new. A lot of the infrastructure that I migrated from Intransa's equipment onto Viakoo's equipment was virtualized servers dating in some cases all the way back to physical servers that Intransa bought in 2003 when they got their huge infusion of VC money. Still, it's just a practical reminder of the killer app for virtualization -- the fact that it allows your OS and software to survive despite underlying drivers and architectures changing with the seasons. Now making your computer work faster can be done without changing anything at all about it -- just buy a couple of new virtualization servers with the very latest fastest hardware and then migrate your virtual machines to them. Quick, easy, and terrifies OS vendors (especially Microsoft) like crazy because now you no longer need to buy a new OS to run on the new hardware, you can just keep using your old reliable OS forever.

-ELG

Why no cloud?

So I promised I'd explain why I was setting up normal Linux-based storage and normal KVM/ESXi compute servers for our new small business's network rather than an OpenStack private cloud, so I'll do so.

1. One risky technology per deployment. It's about risk management -- the ability to manage risks in a reasonable manner. If you have multiple risky technologies, the interactions between risks rise exponentially and cause risks to be unmanageable. Normal Linux-based storage is a mature technology with over a decade of active deployment in production environments with the exception of the LIO iSCSI target. I concluded that the LIO iSCSI target was a necessity in our environment because the TGTD target provided with stable Linux distributions has multiple serious deficiencies (see earlier postings) that render it nothing more than a toy, and our legacy infrastructure was based around iSCSI talking to that pile of ancient Intransa blue boxes. So I've reached my limit on new technologies. Meanwhile OpenStack is multiple immature technologies under active development. Add that to LIO and the existing VMware ESX/ESXi servers' need for block storage and I'd require multiple storage networks to mitigate the risks. Which brings up...
2. Power and space budget. My power and space budget allows for one storage network with a total of 8U of space and 1000 watts of power consumption. I don't have power and space for two storage networks, one for OpenStack and one for ESX/ESXi.
3. Performance. The majority of what my network provides to end users is file storage via NFS and CIFS. In an OpenStack deployment file servers run as virtual machines talking to back end storage via iSCSI. This scales very well in large installations, but I don't have the power and space budget for a large installation so that's irrelevant. Running the NAS storage stack directly on the storage boxes results in much better responsiveness and real-world performance than running the NAS storage stack on a virtual machine talking to the storage boxes via iSCSI, even if the theoretical performance should be the same. The biggest issue is that this limits the size and performance of any particular data store to one storage box, but the reality is that this isn't a particularly big limitation for our environment, since we have far more iops and storage on a single storage box than any single data store in our environment will use for quite some time. (My rule of thumb is that no ext4 data store will ever be over 1Tb and no xfs data store will ever be over 2Tb, due to various limitations of those filesystems in a NAS environment... any other filesystem runs into issue #1, one risky technology per deployment, and I already hit that with LIO)
4. Deep understanding of the underlying technologies. The Linux storage stack has been mature for many years now, with the exception of LIO. I know its source code at the kernel level fairly well. If there is an issue, I know how to resolve it, even to the point of poking bytes into headers on disk drive blocks to make things work. Recovery from failure thus is low risk (see #1). OpenStack is a new and immature technology. If there is an issue, we could be down for days while I chase around in the source code trying to figure out what went wrong and how to fix it.

Note that this is *not* a slam on OpenStack as a technology, or saying that you should not use one of the OpenStack cloud providers such as RackSpace or HP. They have massive redundancies in their OpenStack deployment and people on staff who have the expertise to manage it, and do not have to deal with legacy infrastructure requirements such as our ESXi servers with their associated Windows payloads. Plus they are based around a totally different workload. Our in-house workload is primarily a NAS workload for workstations, and our compute workload is primarily a small number of virtualized test servers or build servers for our software in a variety of environments as well as a handful of infrastructure servers to e.g. handle DNS. What OpenStack mostly gives you is the ability to manage massive numbers of storage servers and massive numbers of compute servers and massive numbers of virtual machines on those compute servers, none of which is our local workload.

The workload that RackSpace etc. are supporting is mostly about Big Data and Big Compute in the cloud or about web server farms in the cloud. All of that has far larger space and power requirements than our little two-rack data center can ever provide, and the reality is that we simply use their infrastructure when we have those requirements rather than attempt to replicate their infrastructure in-house. It simply isn't reasonable for a small business to try to replicate RackSpace or Amazon AWS in-house. We don't have the space and power for the massive amount of infrastructure they use to achieve redundancy and reliability, we don't have the requirement for our local workload, and we don't have the in-house expertise. In the end, it's a case of using the appropriate technology for the appropriate task -- and for what I'm attempting to achieve for the local infrastructure of a small business, using NAS-based Linux storage was more appropriate than attempting to shoe-horn our workload into an infrastructure that would give us no more capability for our needs but would cost us in terms of power, space, performance, and maintainability.

-ELG

Storage migrations

I spent much of today setting up a pair of Linux servers to migrate data off of a 2005-vintage Intransa storage array. The Intransa storage array still works fine, but clearly the end is in sight -- I have a limited supply of spares and three drives died within the past two months alone. So I set up a 10Gb fiber connection between the iSCSI switch for the Intransa array and the iSCSI switch for my new(old) commodity Linux servers (a pair of previous-generation Supermicro 12-disk servers with a 12-disk JBOD apiece), exported iSCSI volumes via lio, told Windows to mirror its various volumes to the new volumes, and let'er rip. Note that I did traditional RAID here because I don't have the cycles or the CPU's to implement an internal cloud for a small company, and that these storage servers are also providing regular file shares via NFS and Samba (CIFS). I deliberately kept things as simple as possible in order to make it more easily manageable. In the process some clear issues with the current Linux storage stack became apparently. Thumbnail summary: The Linux storage stack is to professional storage stacks such as the old Intransa stack (or modern-day HDC or HP stacks) as Soviet toilet paper was to Charmin. Soviet toilet paper could serve as sandpaper -- it was rough, annoying, and it did the job but you certainly didn't like it. Same deal with the Linux storage stack, with the additional caveat that there are some things that antique Intransa gear would do that are pretty much impossible with the current Linux storage stack.

Rather than go off onto a long rant, here's some things to think about:

1. The Intransa unit integrated the process of creating a volume, assigning it to a volume group (RAID array) that fit your desired policy creating one if necessary (this Intransa installation has six storage cabinets each with 16 drives, so individually adding 96 drives to RAID arrays then managing which ones your volumes got placed upon would have been nightmarish) and then exporting the resulting volume as an iSCSI volume. All of this is a multi-step manual process on Linux.
2. You can create replications to replicate a volume to another Intransa realm (either local or geographically remote) at any point in time after a volume has been created, without taking the volume offline. On Linux, you have to take the volume offline, unexport it from iSCSI and/or NFS, layer drbd on top of it, then tell everybody (iscsi, NFS, fstab) to access the volume at its drbd device name now rather than at the old LVM volume name. Hint: Taking volumes offline to do something this simple is *not* acceptable in a production environment!
3. Scaling out storage by adding storage cabinets is non-trivial. I had to bring up my storage cabinets one at a time so I could RAID6 or RAID10 the cabinets (depending upon whether I was doing a scale or performance share) without spanning cabinets with my RAID groups, because spanning cabinets with SAS is a Bad Idea for a number of reasons. Policy-based storage management -- it's a great idea.
4. Better hope that your Linux server doesn't kernel panic, because there's no battery-backed RAM cache to keep unwritten data logged. It still mystifies me that nobody has implemented this idea for the Linux software RAID layer. Well, except for Agami back in 2004, and Intransa back in 2004, neither of which are around anymore and where the hardware that implemented this idea is no longer available even if they were. And Agami did it at the disk controller level, actually, while Intransa did it by the simple expedient of entirely bypassing the Linux block layer. These first-generation disk cabinets have each 4-disk shelf IP-connected to a gigabit switch pair that then uplinks via a 10Gb link to the controllers, iSCSI requests flow in via 10Gb from the iSCSI switch pair to the controllers, are processed internally to turn them into volume and RAID requests which then get turned into disk shelf read/write requests that flow out the network on the other end of the stack, and nowhere in any of this does the Linux block layer come into play. That's why it was so easy to add the battery-backed cache -- no Linux block layer to get in the way.

The last of which brings to the forefront the role of the Linux block layer. The Linux block layer is this primitive thing that was created back in the IDE disk days and hasn't advanced much since. There have been attempts via write barriers and other mechanisms to make it work in a more reliable way that doesn't lose filesystems so often, and those efforts have worked to a certain extent, but the reality is that you have lvm and dm and drbd and the various raid layers and iscsi and then filesystems all layered on top like a cake and making sure that data that comes in at the top of the cake makes it to the disk drives at the bottom without a confectionery disaster inbetween... well, it's not simple. Just ask the BTRFS team. In private. Out of earshot of young children. Because some things are just too horrible for young ears to hear.

And I said I wasn't going to go off on a long rant. Oh well. So anyhow, next thing I'll do is talk about why I went with traditional RAID cabinets rather than creating a storage cloud, the latter of which would have taken care of some of my storage management issues by making it, well, cloudy. But that is a discussion for another day.

-ELG

Crack-smoking Linux zealots

It seems that every year, like the swallows flocking back to Capistrano, you get yet another burst of Linux zealots claiming that this is the year for the Linux desktop. As you know, I've said good things about Gnome 3 and its usability by mere mortals. Linux geeks whine about how they can't customize it blah de blah blah, but mere mortals don't customize their window manager -- other than setting the desktop background to a photo of their kids or grandkids, all they care about is whatever applications they're wanting to run. And from that perspective, Gnome 3 works just fine. But: An operating system is more than a window manager. And that is where Linux still is Utter Fail on the desktop.

Okay, let me tell you a story. I installed Fedora 17 on a 64GB SSD to manage my RAID arrays that serve data to my home network. For grins and giggles I installed Thunderbird and told it about a couple of my email accounts. I clicked through on a link in one of those email accounts that went to a page that had a video embedded. Well, that *would* have a video embedded, on Windows or MacOS. On Linux? Just a blank box and a message about a plugin that simply doesn't exist for Linux that needs to be installed, probably a Quicktime plugin. Now I'm sure you're saying, "but that's not Linux's fault, that's Apple's fault for patenting the algorithm used for that plugin!" But see, the deal is, end users don't care whose fault it is. All they know is that they can't see that embedded video with Linux, and they can see it with Windows or MacOS. BTW, same applies to playing MP3 files -- they simply won't play. Again, a patent issue, but Joe Sixpack doesn't care why his music files won't play on his Linux system -- all he cares about is that they don't play on his Linux system. End of game.

Now let's look at another issue, one where the excuse of patents does not hold. The SSD came out of an old Asus Aspire One netbook, which I'd put it in as part of an attempt to build a mobile GPS system, one of a couple of failed attempts that eventually ended up successful with the iPad, thus rendering the need for an SSD in the Aspire One obsolete. So I grabbed an old 160GB hard drive to put into the Aspire One to take the place of the SSD, and imaged it using the rescue disks that I'd purchased to put an image onto the SSD in the first place. Except it didn't boot. Okay, no big deal, I'll just haul a Windows XP rescue disk over there ... err... I don't happen to have one burned? No biggie, I'll just burn one. And since I have my beautiful Linux desktop sitting here, I'll just use that to do so.

So I pop the blank CD/RW disk into the Asus SATA CD/DVD writer, and attempt to burn it using the stock Brasero disk burner... and it's utter fail. Won't do it. In fact, locks up the system for 30 seconds because it locks up the SATA bus while not doing it. What the BLEEP?! So I go check Google, and it looks like Brasero has been broken on Fedora FOR YEARS, it'll write DVD's (sometimes) but CD's? No way! So I try a couple of other programs, and have similar results. Then I go down to the underlying 'wodim' program and again the same results. At that point I'm, like, "okay, maybe it's broken CD recorder hardware". So I pull out an external USB CD recorder drive -- one that I recently used on another system that doesn't have an internal DVD-RW drive -- and attempt to use that. Same result.

So maybe it's a bad disk? So I sling the disk into my MacBook Pro, erase it, and record it. It Just Works. As would have been the case, I'm 99% certain, if I had been trying to do this under Windows 7 -- it would have Just Worked.

And lest you think this is just my physical chunk of hardware and the particular version of Linux here at home, I have similar (lack of) success recording CD-R's at work on my Centos 6.3 desktop system. There, again, I have to record CD's on my MacBook Pro because Linux simply refuses to do it. Which infuriates me, because I could write CD's for *years* under Linux, but it's broken now, and nobody seems to care (at least, it's been broken in every release of Fedora since Fedora 10, which was, what, four years ago? Five years ago?), so ...

So let's recap: I can't view videos, I can't play music, and I can't burn CD-R's. I *can* do all of those things on Windows or MacOS. What Linux does well is serving data to networks. It does that very well, my Linux box serves as the Time Machine backup for my MacBook Pro as well as being the central data repository for the various Windows laptops I have hanging around. But the desktop? When it can't do simple desktop tasks that Windows and MacOS have done for literally years? Crack. That's my only explanation for why anybody would ever make a ludicrous statement like "this is the year of desktop Linux!" given the current state of Linux.

-ELG

Linux block layer, BTRFS, and ZFS On Linux

Long time no blog. Lately I've been stuck way down in the 2.6.32 kernel block device midlayer, both initiating I/O to block devices via the submit_bio interface, and also setting up a midlayer driver.

What I'm finding is that things are a bit of a mess in the 2.6.32 kernel when it comes to device pulls and device removals. When I chug down into the scsi midlayer I see that it's supposed to be completing all bios with -EIO, but there's still situations where when I yank a drive out of the chassis, I don't get all of my endios back with errors because of races in the kernel between device removal and device teardown. The net result is I have no idea what actually made it to disk or not. Note that you will NOT see this racy behavior on a normal system where the completion (almost) always wins the race, I was generating thousands of I/O's per second to 48 disks with CPU usage pretty much maxed out as part of load testing to see how things worked at the limits.

Now, that's no problem for my particular application, which is somewhat RAID-like, or for the Linux MD layer, or for single standalone block device filesystems for that matter. What's on the disk is on the disk, and when the standalone filesystem is remounted it'll know its state at that point by looking at the log. For the RAID type stacking drivers, when it comes back the RAID layer will note that its RAID superblock is out of data and rebuild the disk via mirror or ECC recovery, a somewhat slow process but the end result is a disk in known state. So when I get the disk removal event I mark the device as failed, quit accepting I/O for it, and mark all the work pending for that device that hasn't already endio'ed as errored, and if an endio sneaks in afterwards and tries to mark that work item again as errored, no big deal (although I put a check in the endio so that it simply noops if the work already was completed by the disk removal event). This means I have to keep a work pool around, but that's a good idea anyhow since otherwise I'd be thrashing kalloc/kfree, and if the drive comes back I'll re-use that pool again.

So traditional RAID and standalone devices don't have a problem here. Where a problem exists is with filesystems like btrfs and zfs that do replication on a per-file-block level rather than on a per-block-device-block level. If they can't log whether a block made it or not because they never got an endio, they can get confused. btrfs appears to err on the side of caution (i.e., assumes it didn't get replicated, and replicate it elsewhere if possible) but when the missing volume comes back and has that additional replica on it, strange things may happen. ZFSonLinux is even worse, since its SEL (Solaris Emulation Layer) appears to assume that bios always complete, and deadlocks waiting for bio's to complete rather than properly handle disk remove events. (Note that I haven't really gone digging into SEL, I'm basing this solely on observed behavior).

The good news: The popularity of btrfs among kernel developers appears to be motivating the Linux kernel team to fix this situation in newer kernels. I was chugging through the block subsystem in 3.5 to see if there was something that could be backported to make 2.6.32 behave a bit better here, and noticed some significant new functionality to make the block subsystem more flexible and robust. I didn't backport any of it because it was easier to just modify my kernel module to behave well with the default 2.6.32 behavior (I'm talking *extensive* changes in the block layer in recent kernels), but it appears that the end result is that btrfs on the 3.5 kernel should be *significantly* more reliable than the backported version of btrfs that Red Hat has put into their 2.6.32 kernel on RHEL 6.3.

So that's my recommendation: If you want to run btrfs right now, go with the latest kernel, whether it's on RHEL 6.3 or Fedora 17 or whatever. And you know the reason for my recommendation now. Red Hat has *not* backported any of those block layer changes back to RHEL 6.3, so what you have with btrfs on their stock 2.6.32 kernel is the equivalent of having a knight in shining armor that's missing the breast-plate. Sort of renders the whole exercise useless, in the end.

-ELG

End of the FreeBSD ZFS experiment

So my write performance with ZFS on FreeBSD was abysmal. The final straw was when I was copying from one pool to another and it was running at roughly the same speed as a floppy disk. I made numerous attempts to tune both ZFS and the iSCSI initiator, and nothing that I tried made any real long-term difference. Things would speed up after I tweaked stuff, then slowly settle back down to a tedious crawl.

Out of frustration I created a 64-bit Centos 6.2 image with the exact same stats as the FreeBSD image and installed the native Linux port of ZFS. This requires a stub that goes into the kernel to meet licensing requirements, then compiles against that stub code. I then shut down the FreeBSD virtual machine, installed the iSCSI initiator on the Linux machine and scanned both of my iSCSI storage arrays to tell it about the initiator, then went to the storage arrays and unassigned the volumes from the FreeBSD machine and assigned them to the Linux machine instead. Then I scanned and logged them in on the Linux machine, and did the following command at a root login:

zpool import -f -F -m -a

They imported cleanly and everything came up.

So the next thing I did was set off my copy going. I am ZFS-mirroring between the two iSCSI storage arrays and I only have a single gigabit Ethernet port from my ESXi box to the storage arrays, so this put a maximum throughput of roughly 100 megabytes per seconds for both read and write. ZFS did this throughput to the storage arrays handily.

So clearly the problem is not ZFS. And FreeBSD has been shown to have good ZFS performance with DASD (direct-attached storage devices). So the fundamental problem appears to be the FreeBSD iSCSI initiator. I don't care enough to diagnose why it's so terrible when used with ZFS despite the fact that I hit all the tuning flags to turn up the queue depth etc., but the end result is that ZFS combined with iSCSI on FreeBSD is a no-go.

On Linux, BTW, it Just Worked once I built the zfs RPM's and installed them. I'm performing at the full speed of my network. And remember, that's the ultimate role of computer technology -- to Just Work, leaving the hard stuff of deciding what's going to go onto that server to the humans. My goal was to move bytes from point A to point B as fast as my ESXi system could do so, given the fact that I need to set up Etherchannel on my antique SMC switch to do ESXi trunking to get data from point A to point B any faster than gigabit Ethernet will take me. (I don't know if the antique will even do it, this is a production ESXi box so I have to set an outage time at an oddball time before I can move the iSCSI network Ethernet wires to the supposedly Etherchannel-trunked ports and flip the ESXi vswitch bit to ip-ip hash to split the traffic between the two trunked ports). So while it sucks that I have to manually build and install ZFS on Linux, the final result appears to work far better than it really should, considering the very beta-quality appearance of that ZFS On Linux site and the rapid updates they're making to the software.

-ELG

ZFS -- the killer app for FreeBSD?

Okay, so here's my issue. I have two iSCSI appliances. Nevermind why I have two iSCSI appliances, they were what was available, so that is what I'm using. So I want my backups mirrored between them. Furthermore, I want my backups to be versioned so I can access yesterday's backup or last month's backup without a problem. Furthermore, I want my backups to look like complete snapshots of the system that was backed up as of that specific point in time. Furthermore, because my data set is highly compressible in part and highly de-duplicable in part, I want it compressed as necessary and dedupe'ed as necessary.

So, how can I do this with Linux? Well... there is a LUFS version of ZFS that will sort of, maybe, do it in a way that regularly loses data and performs terribly. There is BTRFS which is basically a Linuxy re-implementation of ZFS that does compression but not deduplication, and which is still very much a beta-quality thing at present, they didn't even have a fsck program for it until this spring. And ... that's it. In short, I can only do it slowly and buggy.

So at present I have a FreeBSD virtual machine in my infrastructure happily digesting backups and bumping the snapshot counter along. And ZFS is a first-class citizen in FreeBSD land, not a castaway in LUFS-land like on Linux. I'd love to use BTRFS for this. But BTRFS today is at about the same stage as ZFS on Solaris in 2005, when it was an experimental feature in OpenSolaris, or ZFS on FreeBSD in 2008 when the first buggy port was released. ZFS on FreeBSD is stable and rock solid today, and BTRFS, realistically, isn't going to be stable and rock solid for another three or four years at least.

So if you haven't investigated ZFS on FreeBSD to manage large data sets in a versioned, compressed, and deduplicated fashion, perhaps you should. It solves this problem *today*, not a half decade from now. And a bird in hand is worth a dozen in four years.

- ELG

Gnome 3, Mac OS, and the Geeks (Part 2)

So in part two of this comparison, we're going to talk about workspaces.

Both Gnome 3 and Mac OS Lion have the concept of a linear ribbon of "workspaces" or virtual desktops. Gnome 3 lays out its workspaces vertically, while Mac OS Lion lays out its workspaces horizontally. So how do they compare on some routine workspace operations? Let's see...

Switching to Workspaces:

Gnome 3: There are two basic ways to do this: 1) Press the Windows key or swoosh mouse to left top of screen in order to activate the Activities screen. Move the mouse to the right side of the screen to make the workspace list pop out and select the workspace you wish to be in. OR: 2) Press CTRL-ALT-down to go to the next workspace down, or CTRL-ALT-up to go to the next workspace up. Mac: 1. Press F3 button on a recent Mac to go into Mission Control. You can also set a multi-touch gesture to do this (mine is three fingers up on the trackpad). Your workspaces will be listed horizontally at the top of the screen, move your mouse to and click on the one you want to go to. 2. Assign a multi-touch gesture to next-workspace and previous-workspace. Mine is three fingers left/right on the trackpad. 3. Assign a key sequence to next-workspace and previous-workspace. Mine is CTRL-left and CTRL-right. Because I have an Apple laptop, I can use multi-touch gestures to move left and right and to activate the Workspace switcher. This gives me one more option on the Mac. But reality is that navigating to a workspace is ridiculously easy on both systems, either from keyboard or via mouse/trackpad.

Creating Workspaces:

Gnome 3: There is always one "blank" workspace at the end of the list of workspaces. If you move to that blank workspace and open a window there, a new blank workspace is created after it automatically, without you having to do anything.

Mac OS Lion: Press the F3 function key or use the multi-touch gesture to get to Mission Control. Move your mouse pointer to the right top of the screen. A new shadowed-out workspace will pop out of the ether. Click on it. You will now have a blank workspace to work in.

The Gnome approach can be done without touching the mouse, and actually requires no intervention on your part to create the new workspace -- it simply gets created when you need it. It Just Works, which is what a computer is supposed to be -- something that Just Works, without you having to do fiddly things to make it work. It will be interesting to see whether Apple or Microsoft copy this feature in their next release.

Deleting workspaces

Gnome 3: When you close the last window on a workspace, the workspace is deleted and you are then placed on the Activities screen, from whence you can select another workspace to work in using either CTRL-ALT-up/down or the workspace pop-out at the right of the screen. This prevents workspace clutter where you have lots of those automatically-created workspaces hanging around. (Note: I am aware of the "persistent workspace" plugin, I am comparing stock configurations).

Lion: Activate Mission Control. Move to the top left corner of the little workspace icon you want to zap. A little X will appear. Click on that X. The workspace will go bye-bye, and any windows on it will be moved to the first workspace.

Again the Gnome 3 approach to this appears to me to be much simpler than the Lion approach. There's no fine motor skill needed to move the mouse pointer to the exact point where the X will appear, you simply close your windows and poof, you're done. Nothing to remember, nothing to discover, it Just Works.

Moving Windows to a Workspace

For both Lion and Gnome 3 you simply move to the workspace containing the window you want to move, trigger the Activities or Mission Control screen, grab the window you want by clicking on it, and drop it on the icon of the workspace you want to move it to.

-------------------

So that's workspaces. As you can see, Gnome 3 is quite competitive with the state of the art that is Mac OS Lion. So that brings up the next question: Why do so many of the hard-core Linux geeks hate Gnome 3? I'll discuss that in the next installation of this series.

-ELG

Gnome 3, Mac OS Lion, and the geeks (Part 1)

I've been using Gnome 3 on Fedora 15/16 and Mac OS Lion on a new Macbook Pro for several months now. What I see are two platforms that have done a significant re-thinking of the user interface to deal with some unpleasant facts:

1. The old paradigm of using menus to select programs has reached its expire-by date because the menus have attained a depth that nuclear submarines would love to attain
2. The population of most major technological countries is aging, and our old eyes simply cannot read the print on those tiny little pull-down menus anymore,
3. Fine motor control of old folks is pretty bad. We can manage to swoosh the cursor to the corner of the screen, or hit a big icon, but fiddly little menu items are hard for us to nudge a mouse into the right box to navigate a pull-down menu, and finally,
4. Pull-down menus simply aren't compatible with small touch-screens, because fingers are too fat to select tiny little things and you can't see them on a small touch screen like on a tablet anyhow.

In the process of dealing with these unpleasant facts, Mac OS Lion and Gnome Shell have also done a major re-thinking of how you do many common tasks in order to reduce the number of mouse movements / key strokes needed to do them. So let's look at some common tasks...

Program selection, Gnome 3:

Method a: Swoosh the mouse to the top left of the screen (one movement). Swoosh the mouse to the 'Applications' tab (one movement). Click left button. Select program from list of icons (possibly using the scroll wheel to scroll up and down the list).

Method b: Press the Windows key. Type the first couple of characters of the program you want to run. Use the arrow up-down to move the highlight to the icon of the program you want to run. Press ENTER.

Program selection, Mac OS Lion

Method A: Move the mouse to the icon of a rocketship on the toolbar. Click. Move mouse pointer to icon of program to run, possibly using left-right wobble wheel on your mouse or left-right two finger swipes on trackpad to go to next page of programs. Click on program.

Method B: Set a hot corner in preferences (I set bottom left), swipe mouse to there, move mouse to program to run, click.

Method C: Set a hot key in preferences such as control-opt-l, use left-right arrow keys to navigate pages. Unfortunately Mac OS has no way that I can find of using the keyboard to actually run one of those programs, you must actually navigate your mouse pointer to it and click it.

Verdict: If using a keyboard, Gnome 3 is very easy to navigate and uses a minimum of keystrokes to locate and run a program, requiring no mouse input at all. If using a mouse, the fact that you can get to the Applications icons immediately in MacOS, vs. having to click on the Applications tab after swooping to the corner, makes MacOS require one less mouse movement and one less mouse click. Score: TIE.

Select A Window, Gnome 3

Method 1: Ye olde alt-TAB, with a kick. For applications with multiple windows, you are shown a down-arrow and a down-cursor will then open window previews. Select the window you want.

Method 2: Hit the Windows key or swoop mouse to left top of screen. The windows will then swoosh out into a thumbnail pane view. Click on the window you're interested in.

Method 3: Hit the Windows key or swoop mouse to top left of screen. Select the dock icon representing the program you want to switch to. Either click it to go to the topmost window, or right-click it to select which window you want.

Select A Window, Mac OS Lion:

Method 1: Command-tab to get a list of running programs. While still holding down Command-tab, then use the arrow keys to move left-right to program whose window you want to see. Release command-tab while that program is highlighted. Note that unlike with Gnome 3, you do *not* get to choose which exact window of the program is going to be switched to -- you get whatever MacOS feels like giving you.

Method 2: Set a mouse button or gesture (I use the Page Forward button on my Logitech mouse or a triple-finger-up gesture on the trackpad). Invoke said mouse button or gesture. The windows will zoom out to pane/thumbnail view. Navigate mouse to the window you want and click on it.

Method 3: Move mouse to bottom of screen, click the dock icon of the program you want to switch to. Right-click will allow you to choose which of the windows to switch to.

Verdict: Gnome 3 wins on command-tab, its command-tab function is full-featured and works well. It ties on mouse button or gesture. It loses on dock, but only barely because its dock is not always visible and you have to move your mouse to the top right of the screen to show it, but that's only one mouse movement extra so not a huge loss. So Gnome 3 by a nose.

-----------------------------------------------------------------------

So what have we learned thus far? Well, 1) For two specific tasks, both Gnome 3 and Mac OS Lion have done a lot of work on reducing the amount of mouse movements and/or keystrokes needed to do these tasks, and almost completely eliminated any necessity for fine motor movements or reading of tiny print, and 2) Gnome 3 is quite competitive with Mac OS Lion in doing these tasks. In the next part of this series I will compare some other common operations, and finally I will summarize the results and examine one of the more bizarre things that has happened since the release of Gnome 3 -- the rabid condemnation of it by early Linux developers (including Linus Torvalds) who appear to despise it, and what that means for Linux.

Re-inventing the Linux desktop

My opinion of the Linux desktop is pretty much well known by now -- I believe I mentioned "Window 95 as re-implemented by a Soviet Union that never fell"? A.k.a. clunky, incoherent, and obsolete? But recently two distributions have been released with a re-imagining of what the Linux desktop should look like. So today I installed Fedora 15 and Ubuntu 11.04 into Virtualbox on my Macbook Pro (into Virtualbox because it's currently the only virtualization environment with 3D support for Linux), and examined the two new systems -- Gnome 3's Gnome Shell, and Ubuntu's Harmony.

Gnome 3 was the first one I looked at, and by far the most revolutionary. Everything in the new Gnome Shell is set up to be doable with a couple of mouse swishes and one or two clicks. A swish to the upper left corner of the screen does three things -- does an Expose'-like scale of your windows so you can choose which window you want to activate by just clicking on it, sucks in a dock from the left side of the screen, and sucks in a screen list from the right side of the screen. You can grab a window and move it to the next screen in the screen list (there's always one blank screen at the end of the list -- sort of like the iPhone's applications screens), or you can click on the word "Applications" towards the upper left of your screen and see a list of applications in a format somewhat like an iPhone's application chooser. It is all easier to use than it sounds -- it really does make the whole thing swish swish swish point and click easy.

If you have applets running, like gDesklets, you can get to them by going to the bottom right corner of the screen. A sort of fuzzy menu bar then rises up from the bottom.

My general conclusion: Gnome 3 is currently incomplete -- it's barely configurable at all for example -- but it puts together the best ideas in UI's that have come down the pike over the past few years into an easy-to-use whole. My workflow falls out of the way Gnome 3 works naturally. If I want a screen for my browser windows, for example, swish to top left, swish to next screen on the right and click it, swish to left and select my browser on the dock, and voila, it pops open on the new screen and *another* blank screen is created for the *next* thing I want to do. Close all the windows on a screen, and it goes away, so I always have just the screens I need for my workflow -- no more, no less. I spent some time today doing software development with this system, and the usability compared to traditional Gnome is astounding.

Next up was Ubuntu's Unity. That, alas, turns out to be a disappointment. While Gnome 3 re-imagined the world to the point where some of the rumored Windows 8 functionality is going to be a clone of already-existing Gnome 3 functionality (like the iPhone-like application chooser), Unity simply attempts to clone Mac OS X without seeming too obvious about it. The problem is that simply moving the dock to the left side and modifying GTK+ to move application menus to the top like MacOS (except active and context sensitive, can't forget that!) isn't enough to make a quantum leap in functionality. Frankly, it ends up looking a bit of a mess. Gnome 3 re-imagined, Unity cloned, and like most clones, the clone isn't the equal of the original.

So: Does this mean the Linux desktop is usable now? Am I going to abandon my Macbook Pro and run a native Linux desktop again? Uhm... not hardly. I have a major investment in MacOS professional music software for which there is no Linux equivalent, and I still have significant difficulties viewing multimedia-based sites with Linux. Part of that is Microsoft and Apple's fault -- they release all these tools for "free" that produce (and view) multimedia content in their own proprietary formats like Quicktime and WMA, and don't release the viewing tools for Linux. I also can't read my corporate email using Linux -- neither of Evolution's Exchange plugins will handle proxied Exchange servers. That's sort of important too. Still, the fact that there is now a Linux UI which is actually innovative rather than a crude clone of other people's ideas is a sea change in an OS development process which all too often has ignored the desktop in favor of server optimizations. I don't know what happens next, but I suspect it'll be an improvement. Of course, given where the Linux UI started -- as an incoherent mess (inherited from MIT X11) -- that's sort of faint praise. So it goes.

-ELG

Sound, Flash, and VMware Player

This is some notes on how to get sound and flash working on a Scientific Linux 64-bit guest ( rebranded Red Hat Enterprise Linux 6 ) running inside VMware Player on Windows 7 64-bit:

Flash: Note that Chrome for 64-bit Linux does *not* include the integrated Flash player that all other platforms have. You'll need to download 64-bit Flash beta from Adobe. At the time of this writing, that's at Adobe Labs. Once you extract the tgz file, you'll be left with a file "libflashplayer.so", simply copy it to /usr/lib64/mozilla/plugins , restart Chrome or Firefox, and you're set.

Sound: VMware's sound system crashes with the default Ubuntu / Red Hat sound configuration. This is apparently because VMware doesn't bother emulating all pieces of the hardware they say they're emulating, and when ALSA touches the missing bits, VMware disables the sound device. That's easy enough to fix though. From the Gnome menu, go to System->Preferences->Sound. In the Sound preferences, click on Hardware. Change the profile at the bottom from "Analog Stereo Duplex" to "Analog Stereo Output". Then on the little icon of the speaker on the bottom margin of the VMware Player, right-click it and select "Connect". After about a minute it'll turn green and you'll be able to play sound again.

Why this happens: Though the Ensoniq device emulated is capable of stereo duplex operation (i.e., both recording and outputting at the same time), I know because I actually had one of the physical cards back in the day and used it for that purpose along with a multitrack recorder program, VMware's emulation of the device is not capable of such and thus you must disable that capability. Unless you were intending to record audio within the Linux virtual machine (*not* recommended, VMware's timing is not sufficiently good to get good results there), this has no actual effect -- you can still play Flash videos from Chrome (and Firefox presumably) and hear the sound.

So if you're running Windows 7 on your physical hardware because you need the graphics performance for games and aren't willing to do the dual-card IOMMU hack with Xen that I demonstrated previously (or your hardware simply doesn't have IOMMU support, or you're not willing to use the latest bleeding-edge OpenSUSE as your platform), you can still have the far more secure web browsing environment of Linux available, and with VMware Player you can give Linux any additional drives beyond your boot drive for Linux to manage. Linux works far better as a server than Windows 7 does -- it can provide CIFS, AFP, iSCSI, and do it all on a much better software RAID stack than Microsoft's, as well as using LVM to manage that space, for which Windows 7 has no equivalent. And VMware's Unity system actually works pretty well with SL6/RHEL6 on Windows, although not as well as it works in VMware Fusion on MacOS (the issue being that the little Unity menu icon gets put into the screen menu bar on MacOS and has a native look and feel, while shows up as a clunky little usually-invisible icon above the Start menu on Windows). Which means that you can mix Linux Chrome windows, Windows IE windows (ick! But there's a couple of applications I need for customer support that requires IE plugins that don't exist for any other browser), Linux shell windows, and hoary old Outlook all on the same screen and manage them using the normal Aero Peek icons at the bottom or, if you have a Logitech mouse with the Logitech drivers installed, by assigning one of the side buttons to Window Switcher (their clone of Apple's Expose'). And unlike earlier versions of Windows, Windows 7 is stable -- in fact, I've never managed to make it crash. It's just very annoying... but that's why Apple is still in business, after all. Because Apple makes computers that are not annoying. For a price. A big price, alas...

-ELG

Windows virtualization software

Previously I mentioned three bare metal hypervisor solutions -- Xen, KVM, and VMware ESXi -- and what was required to push through a video card to a WIndows VM living on those hypervisors. I have since managed to get that working with VMware ESXi, and I suspect if I install the very latest Fedora 14 host it'll work with KVM too.

That works fine with a desktop machine or rack server where I can mount multiple video cards into the box. The performance of Windows 7 virtualized is indistinguishable from Windows 7 raw. Here are my scores for Windows 7 raw on that box, with a Seagate Momentus XT 7200 RPM boot drive (due to the front-loading slot on my case that allows easy swapping of OS drives):

• Processor: 7.5
• Memory: 7.6
• Graphics: 7.3
• Gaming graphics: 7.3
• Primary hard disk: 5.9

Compare with the virtualized numbers.

The problem, however, is when you want to go mobile. You simply can't add a second video card to a laptop. So if I want to play games on one of those new Dell Sandy Bridge desktop-replacement laptops, I need to run WIndows native, and come up with a way to have Linux also running and handling my years of accumulated data, all of which is in a Unix filesystem tree and cannot simply be copied into Windows. I preferably want this Linux to be running on a raw partition -- not on a file within a Windows partition -- and it has to be able to access raw Linux-formated USB and SATA drives. So, let's look at the first candidate... VMware Player

VMware Player is VMware's entry level desktop virtualization program. At one point in time VMware player would only start up virtual machines that had been created by VMware Workstation, but now it's an almost full version of VMware Workstation with various functionality like snapshots stripped out and with a limitation of only four cores allowed. VMware Player is "free" -- it's free for personal use, if you want to deploy it in a corporate environment you can license it for a fairly trivial per-seat fee (quite trivial, it'll be lost in the noise of your IT budget).

The test machine is my Xen server with Windows 7 installed as described above. I installed VMware Server on it without any problem. I created my Scientific Linux 6.0 virtual machine with no problem, giving it 2 gigabytes of memory and a 16gb root filesystem. VMware Tools installed easily into SL6 and allowed me to treat the Linux "X" desktop as if it were just any other Windows window, I could click into it, my mouse pointer could be moved outside of the window while I was typing into Linux program, and so forth. Adding the two 2TB SATA physical hard drives to the virtual machine was as simple as point and click, though the VM had to be off to do so because VMware Player's hot-plug functionality apparently does not work with physical drives. Once I booted SL6 up, it saw the two 2TB drives and assembled the Linux software RAID arrays on it automagically, though I had to do vgchange -a -y to get SL6 to recognize the LVM volumes on the RAID arrays.

So how fast is access to those two SATA drives? On a subsequent reboot of my Linux VM, a RAID check got fired off. The two drives were being read at 105MB/sec apiece -- 210MB/sec total -- and it used less than 20% of one of my eight cores for VMware to virtualize this. My take on it is that VMware Player's fake SCSI device takes a fair chunk of CPU to virtualize, but modern multi-core CPU's are so bleepin' fast that you won't even notice (which I didn't, until I went to see).

The final thing I wanted to do was to export a NTFS-formatted volume to Windows via iSCSI. Windows 7 has Microsoft's iSCSI initiator built in. I gave both my Windows machine and my Linux VM fixed addresses (using the bridged mode for the Linux VM's virtual network card), and installed the iSCSI target daemon and utils with 'yum install scsi-target-utils'. Then I added the already existing logical volume /dev/datagroup/win7 volume to /etc/volumes/targets.conf (see that file for exact format of what you need to add) and started up the daemon with "service tgtd start". Then I went to the Windows Administrative Tools (you can get to them from the the Start Menu if you've configured them to appear there, or from the Control Panel), selected the iSCSI Initiator, told it to scan the IP address of my Linux VM, and voila, it popped up there and as a drive letter in the Windows Finder. Easy peasy! The only thing to remember is to poke a hole for iSCSI in both the Linux and Windows firewalls, or it doesn't work :). (Yes, been there, done that, heh!).

The final test was to attach a USB hard drive to the system and export it to my MacBook Pro via iSCSI and use it as a Time Machine device. When I attached the hard drive to the system, it popped up in Windows, but clicking Virtual Machine ->Removable Devices showed the new drive, and allowed me to add it to the virtual machine. I then added it to targets.conf and told the tgd daemon about it, then went to my iSCSI initiator on the MacBook (the globalSAN iSCSI initiator) and added it, then used Disk Utility to format it as a Mac volume. Then I went to Time Machine and told Time Machine to use it for backup, and... voila. It started backing up at about 20mb/sec -- or about 50% of theoretical USB2 speed, not too bad considering this is being done via WiFi, not a direct connection, and the iSCSI target is running in a virtual machine, not directly on the hardware. A copy in Windows of a 4GB file to a similar USB drive ran at 26mb/sec, and that should go faster than Time Machine writes because it's a big sequential write, not a lot of smaller files. So now I have the equivalent of one of those expensive Time Capsule thingies, except that a 2TB Western Digital drive in an external USB case costs a *lot* less! Why an external USB? Simple -- so if I ever have to restore my MacBook Pro after a disk crash (which has happened before), I can unplug it from my big server and plug it directly into the MBP for MacOS to restore the system back to pre-crash state.

So... it's clear that VMware Player will do everything I want it to do here. There's two more options to look at before calling this competition done, however: Oracle's VirtualBox, which recently released a brand new version (version 4.0.4), and Microsoft's own Windows Virtual PC, which doesn't officially support Linux but which has been made to do so. More on those later...

-ELG

Pushing a graphics card into a VM, part 5

Part 1 Part 2 Part 3 Part 4 Part 5

So here's the final thumbnail summary:

Hardware:

• Video card #1: ATI 5750 (but the 5770 should work too and is slightly faster, but the 5750 was on the Xen compatibility list).
• Video card #2: nVidia Corporation VGA G98 [GeForce 8400 GS] *PCI* card (BIOS set to use PCI as first card)
• Intel(r) Desktop Board DX58SO -- not a great motherboard, but it was available at Fry's and was on the Xen VT-d compatibility list
• Intel Core I7-950 processor
• 12GB of Crucial 3x4GB DDR3 RAM
• Hard drives: 2 Hitachi 7200 rpm SATA 2GB drives, configured as RAID1 via Linux software RAID.
• Antec Two Hundred V2 gamer case to handle swapping OS's via the front 2 1/2" drive port.
• Various 2 1/2" drives to hold the Linux OS's that I was experimenting with

Software:

• OpenSUSE 11.3, *stock*.
• Windows 7, *stock*.

By adding the PCI card, my Linux console remains my Linux console, and Xen properly starts up my Windows DomU. My configuration is now complete. I may extend my Windows LVM volume to 200G so I can install more games on it, but note that all of my personal files, ISO's, etc. live on Linux. Note that 5.9 is what the Windows Performance Index should be for that particular hard drive combo, so this Windows system is as good as most mid-range gaming systems performance-wise. I added the paravirtualization drivers for the networking and disk controller, but they didn't improve performance any -- all they did was reduce how much CPU the dom0 qemu was expending implementing virtual disk and network controllers. Given that I have a surplus of CPU on this system (8 threads, 3.2ghz), it's in retrospect no surprise that I saw no performance gain on the disk and network from going paravirtual -- all I did was free up more CPU for use for things like, say, video encoding.

Thoughts and conclusions:

One thing that was very clear through this entire process is that I'm very much pushing beyond the state of the art here. The software and hardware configurations needed for this to work were very twiddly -- there is exactly one (1) Linux distribution (OpenSuse 11.3) which will do it at this point in time, and there were no GUI tools for OpenSuse 11.3 which would create a Xen virtual machine with the proper PCI devices. Furthermore, the experimental Xen 4.01 software on OpenSuse is almost entirely undocumented -- or, rather, it has man pages provided with it, but the man pages document an earlier version of Xen which is significantly different from what's actually shipped with OpenSuse 11.3.

From a general virtualization perspective, comparing Xen, KVM, and ESXi, Xen currently wins on capabilities but only by a hair, and those capabilities are almost totally undocumented -- or worse yet, don't work the way the documentation says they work. Xen's only fundamental technological advantage over KVM and ESXi right now is its ability to run paravirtualized Linux distributions without needing the VT-x and VT-d extensions -- a capability which is important for ISP's with tens of thousands of older servers without these extensions, but becoming increasingly less important as VT-x is now everywhere except in the low-end Atom processors. Comparing my Xen installation at home with my KVM installation at work, both of which I have now used extensively and pushed their capabilities to their limits, I can see why Red Hat is pushing the KVM merger of hypervisor and operating system -- KVM gives you significantly greater ability to monitor the overall performance of your system, vs. Xen where 'xm top' is a poor substitute for being able to get detailed monitoring of your overall system performance, is significantly better at resource management since the same resource manager handles everything (core hypervisor/dom0 plus VM's), and the Linux scheduler can consider everything when deciding what to schedule, rather than having the Xen hypervisor out in the background making decisions about which Xen domain to schedule next based upon very little information.

In short, my general conclusion is that KVM is the future of Linux virtualization. Unfortunately my experience with both KVM and Xen 4.0 is that both are somewhat immature compared to VMware's ESX and ESXi products. They are difficult to manage, their documentation is persistently out of date and often incorrect, and both have a bad tendency to crash cryptically when doing things that they're supposed to be able to do. Their core functionality works well -- I've been running Internet services on Xen domains for over five years now and for that problem domain it is bullet-proof, while at work I am developing for several different variants of Linux using KVM virtual machines on Fedora 14 as well as running a Windows VM to handle the VSphere management tools, and it's been bullet-proof. But they decidedly are not as polished as VMware at this point, other than Citrix's XenServer, which lacks the PCI passthrough capability of ESXi and thus was not useful for the projects I was considering.

My take on this, however, is that VMware's time as head of the virtualization pack is going to be short. There isn't much more that they can add to their platform that the KVM and Xen people aren't already working on. Indeed, the graphics passthrough capability of Xen is already beyond where VMware is. At some point VMware is going to find themselves in the same position vs. open source virtualization that SGI and Sun found themselves in vs. open source POSIX. You'll note that SGI and Sun are no longer in business...

-ELG

Pushing a graphics card into a VM, part 4

Part 1 Part 2 Part 3 Part 4 Part 5

Okay, so virt-manager did pick up my new VM once I created it with xm create on a config file, but when I rebooted the system the VM was gone. So how can I fix this? Well, by taking advantage of functionality that OpenSUSE has had for auto-starting Xen virtual machines all along: Just move my config file into /etc/xen/auto and it'll auto start (and auto shutdown, if I have the xen tools installed) at system boot.

Of course, that requires a config file. Rather than paste it here, I'll let you view the config file as a text file. Note that 'gfx_passthru=1' is commented out. The Xen documentation says I need it, but if I put it there, my VM doesn't start up -- it crashes into the QEMU monitor. Also I ran into another issue, a timing issue. pciback is grabbing the console away from Linux and leaving the video card half-initialized, and when Xen grabs the video card and shoves it into the VM, the video card locks up the system solid when Windows tries to write to it. My solution to that was even simpler -- put the older of the nVidia cards back into the system, and load the 'nouveau' driver using YaST's System > Kernel > INITRD_MODULES and System > Kernel > MODULES_LOADED_ON_BOOT functionality. This flips the console away from the ATI card early enough that it doesn't conflict with Xen giving the video card to Windows. This also gives me a Linux console on the nVidia card that I can switch to by plugging in a second keyboard to the front USB on my chassis (the one I did *not* push into the Windows VM) and flipping my monitor to its DVI input (rather than the HDMI coming from the ATI card).

With all of this done, I can now reboot my system and get Windows on video card 0, and Linux on video card 1. I suppose I could reverse the video cards (to give the boot video card to Linux), unfortunately my board puts the second 16-lane PCIe slot too close to the bottom of the case and a double-width PCIe card won't fit there. Maybe when I upgrade to one of those spiffy SuperMicro server motherboards with the IMPI and such, at which point I won't need a second video card anyhow because the on-board video will suffice for Linux...

Next up in Part 5: Thoughts and conclusions.

Pushing a graphics card into a VM, Part 3

Part 1 Part 2 Part 3 Part 4 Part 5

OpenSUSE 11.3 was a quite easy install. I haven't used SUSE since the early 'oughts, but first impressions were pretty good. OpenSUSE 11.3 is KDE-based, which is a change from the other distributions I've been using for the past few years -- Ubuntu on my server at home, Debian on my web and email server, and various Red Hat derivations at work -- and seems to be pretty well put together. The latest incarnation of YAST makes it more easily managable from the command line over a slow network connection than the latest Ubuntu or Red Hat, which rely on GUI tools at the desktop. The biggest difference between Red Hat and SUSE was that SUSE uses a different package dependency manager, "zypper", which is roughly equivalent to Red Yat's "yast" and Debian's "apt-get" but with its own quirks. It appears to be slightly faster than "yast" and roughly the same speed as "apt-get". If you wonder why SUSE/Novell wrote "zypper", at the time they wrote it, "yast" was excruciatingly slow and utterly unusable unless you had the patience of Job. Red Hat has sped up "yast" significantly since that time, but SUSE has stuck with "zypper" nevertheless. I also set up the bridging VLAN configuration that I mention in my previous post about how to do it on Fedora. Again SUSE has slightly different syntax than Red Hat for how to do this in /etc/sysconfig/network/* (note *not* network-scripts), but again it was fairly easy to figure out via reading the ifup / ifdown scripts and consulting SUSE's documentation.

So anyhow, I installed the virtualization environment via "yast" and rebooted into the Xen kernel downloaded by that. At that point I created a "win7" LVM volume in LVM volume group "virtgroup" on my 2TB RAID array, and went into the Red Hat "virt-manager" and attached to my Xen domain, then told it to use that LVM volume as the "C" drive and installed Windows 7 on it. I'm using a LVM volume because at work with KVM, I find that this gives significantly better disk i/o performance in my virtual machine than pointing the virtual disk drive at file on a filesystem. Since both Xen and KVM use Qemu to provide the virtual disk drive to the VM, I figured that the same issue would apply to Xen, and adopted the same solution that I adopted at work -- just point it at a LVM volume, already. (More on that later, maybe).

Okay, so now I have Windows 7-64 bit installed and running, so I shut it down and went to attach PCI devices to it via virt-manager and... err. No. Virt-manager wouldn't do it. Red Hat strikes again, it claims that Xen can't do PCI passthrough! So I went back to the handy Xen Wiki and started figuring out via trial and error how to use the "xm" command line, where the 'man' page for xm doesn't in any way reflect the actual function of the program that you see when you type 'xm help'. So here we go...

First, claw back the physical devices you're going to use via booting with them attached to pciback. So my module line for the xen.gz kernel in /boot/grub/menu.lst looks like...

module /vmlinuz-2.6.34.7-0.5-xen root=/dev/disk/by-id/ata-WDC_WD5000BEVT-22ZAT0_WD-WXN109SE2104-part2 resume=/dev/datagroup/swapvol splash=silent showopts pciback.hide=(02:00.0)(02:00.1)(00:1a.0)(00:1a.1)(00:1a.2)(00:1a.7)(00:1b.0)

Note that while XenSource has renamed pciback to 'xen-pciback', OpenSUSE renames it back to 'pciback' for backward compatibility with older versions of Xen. So anyhow, on my system, this hides the ATI card and its sound card component, and the USB controller to which the mouse and keyboard are attached. I leave the other USB controller attached to Linux. I did not have any luck pushing USB devices directly to the VM, I had to push the entire controller instead, apparently the Xen version of QEMU shipped with OpenSUSE 11.3 doesn't implement the USB (or else I simply need to read the source). Note that you want to make sure your system boots *without* the pciback.hide before you boot *with* it, because once the kernel starts booting and sees those lines, your keyboard, mouse, and video go away!

Okay, so now I'm booted. I ssh into the system via the network port (err, make sure that's set up before you boot with the pciback.hide too!) and go into virt-manager (via X11 displaying back to my Macbook, again make sure you have some way of displaying X11 remotely before you start this) and start up the VM. At that point I can do:

• xm list

and see my domain running, as well as log into Windows via virt-manager. So next, I attach my devices...

• xm pci-attach win7 0000:02:00.0
• xm pci-attach win7 0000:02:00.1
• xm pci-attach win7 0000:00:1a.0
• xm pci-attach win7 0000:00:1a.1
• xm pci-attach win7 0000:00:1a.2
• xm pci-attach win7 0000:00:1a.7
• xm pci-attach win7 0000:00:1b.0

Windows detects the devices, loads drivers, and prompts me to reboot to activate. So I tell Windows to reboot, and it comes back up, but nothing's showing up on my real (as vs. virtual) video screen. then I go into device-manager in Windows and see what happened. The two USB devices (keyboard and mouse) show up just fine. But the ATI video card shows up with an error. I look at what Windows tells me about the video card, and Windows tells me that there is a resource conflict with another video card -- the virtual video card provided by QEMU. So I disable the QEMU video card, reboot and... SUCCESS! I now have Windows 7 on my main console with video and keyboard and mouse!

Windows Experience reports:

• Calculations per second: 7.6
• Memory: 7.8
• Graphics: 7.2
• Gaming graphics: 7.2
• Primary hard disk: 5.9

Those are pretty good, quite sufficient for gaming, except for the disk performance which is mediocre because we're going through the QEMU-emulated hard drive adapter rather than a paravirtualized adapter. When doing network I/O to download Civilization V via Steam I also notice mediocre performance (and high CPU utilization on the dom0 host) for the same reason. We'll fix that later. But for playing games, we're set! Civilization V looks great on a modern videocard on a 1080P monitor with a fast CPU!

Okay, so now I have a one-off boot, but I want this to come up into Windows every time my server boots. I don't want to have to muck around with a remote shell and such every time I want to play Windows games on my vastly over-powered Linux server (let's face it, a Core I7-950 with 12GB of memory is somewhat undertasked pushing out AFS shares to a couple of laptops). And that, friends, is where part 4 comes in. But we'll talk about that tomorrow.

-ELG

Pushing a video card into a VM, Part 2

Part 1 Part 2 Part 3 Part 4 Part 5

The first issue I ran into was that my hardware was inadequate to the task. My old Core-2 Duo setup lacked VT-d support. So I went to the Xen compatibility list and found a motherboard which supported VT-d and upgraded my motherboard. At the same time I also upgraded my case to an Antec case that has a slot on the front for plugging in 2 1/2 inch drives. This was to make it easier to swap operating systems. Theoretically you can hot-swap, but I've not tested that and don't plan to.

Since I am inherently a lazy penguin (much like Larry Wall), the next thing I did was try the "virtualization environments". I found that XenServer was a very well designed environment for virtualizing systems in the cloud. Unfortunately it was also running a release 3.x version of Xen rather than the new 4.0 release, and did not implement PCI passthrough or USB passthrough to fully virtualized VM's natively. There were hacks you could do, but once you start doing hacks, the XenServer environment is not really a nice place to do them. So I moved on.

ProxMox VE is a somewhat oversimplified front-end to KVM and OpenVZ. It looks like a nice environment for running a web farm via web browser, but unfortunately it does not support PCI passthrough natively either. Again, you can start hacking on it, but again once you start doing that you might as well go to a non-dedicated environment.

Ubuntu 10.10 with KVM was my next bet. I *almost* got it running, but the VM wouldn't attach the graphics card. It turns out that was another issue altogether, but looking at the versions of QEMU and KVM provided, it appeared that Fedora 14 had one version newer (as you'd expect, since Fedora 14 came out almost a month later), so I went to Fedora 14 instead.

I got close -- really close -- with Fedora 14. But two different video cards -- an old nVidia 7800GT and a new nVidia GTS450 -- both ended up with error messages in the libvirtd logs saying there was an interrupt conflict that prevented attaching the PCI device. I ranted to a co-worker, "I thought MSI was supposed to solve that!" So I looked at enabling MSI on these nVidia cards and found out that... err... no. Not a good idea, even if I wanted, the cards generally crashed things hard if you tried. So I went back to the XenSource.com wiki on VGA passthrough again and followed the link to the list of video cards, and... err, okay. an ATI Radeon 5750 has been reported as running wiith Xen's VGA passthrough.

So, I swapped that out, and tried again with Fedora 14. This time the KVM module crashed with a kernel oops.

At this point I'm thinking, otay, KVM doesn't seem to want to do this. Xen, on the other hand, seems to have a Wiki and all documenting how to do this. So let's use Xen instead of KVM. The problem is that Xen is an operating system. It relies on having a special paravirtualized kernel for its "Dom0" that handles the actual I/O driver work. Red Hat claims providing such a kernel would be too much work and that they won't do it until the Dom0 patches are rolled into upstream by Linus. This despite the fact that Red Hat has patched their kernels to the point where Linus would barely recognize them if someone plunked the source to them on his disk, but it's that whole Not Invented Here thingy again, Red Hat invented KVM and was looking for an excuse to not include a Xen dom0 kernel, and there you go. I looked at downloading a dom0 kernel for Fedora 14, but then... hmm. Look. OpenSUSE 11.3 *comes* with a XEN dom0 kernel. So let's just install OpenSUSE 11.3.

OpenSUSE 11.3 is what I eventually had success with. But to do that, I ended up having to fight Red Hat -- again. But more on that in Part 3.

-ELG