Eating the seed corn: Red Hat Software, CentOS, and the end of an era

Or: Why we are migrating to Ubuntu Linux.

So, why did Linux win the server wars? Why is virtually every cloud server running Linux now, rather than some variety of Windows Server? Why are probably the majority of corporate web servers running Linux now, and a large variety of other internal application servers are running Linux?

Basically, Linux won the server wars because Linux won the poor college student market. Poor college students couldn't afford a copy of Windows Server to run on their own computer. They could afford a copy of Linux to run on their own computer. And twenty years later, they are the decision makers in IT departments who decide what technology to use.

So anyhow: After being one of those poor college students some years ago I'm now one of those decision makers. For example, I spec'ed and got quotes for $16,000 worth of server equipment last week as well as $1500 worth of software, and over the next few years will be doing the same for a six-figure amount of equipment and software. Granted, this isn't huge by IT terms, but multiply me by thousands, and it adds up quickly.

And one of the things I spec'ed was the embedded OS in our appliance. In 2010 I chose to use CentOS Linux for the embedded distribution in our appliance for a number of reasons:

1. Licensing costs. The licensing terms for Red Hat Enterprise Linux really are oriented around servers. Their per-server costs would make the appliance unprofitable. Furthermore, we're using an extremely stripped down Linux distribution for our appliance. There really is no value we would get from Red Hat support. So we went with a community distribution (CentOS) rather than a commercial distribution(RHEL or SUSE) to reflect the value that we get from the support organization around the distribution.
2. Stream of security patches. We don't have a team to watch CVE's and figure out whether they apply to our stripped-down Linux distribution. Instead, we watch the stream of RPM's coming out of the distribution and apply the ones applicable to our distribution on a regular basis.
3. Length of support. Ten years is a long time. It means we don't have to re-base our software that runs on top of the embedded OS to a new Linux distribution very often. This means less work and more play. Playing with new technologies to add to our product is a lot more fun than the grunt work of porting our software to a new Linux distribution.
4. Continuity with earlier versions of Red Hat Enterprise Linux. If you knew how to set up advanced networking on EL5, you know how to set it up on EL7 (with some caveats such as having to disable NetworkManager first). This made the job of re-basing our software considerably easier than with distributions that moved things around in major ways with each release.
5. A commitment in writing and in all major press outlets by Red Hat Software, when they acquired the CentOS project in 2014, to continue the mission of the CentOS project unchanged, which led to transitioning from Centos 6 to Centos 7 when it arrive and stabilized.

Fundamentally, Red Hat Software has broken all of the above with recent decisions such as the decision to discontinue CentOS 8 as of the end of 2021.

1. CentOS Stream is not a community distribution. It is a beta test stream for RHEL. As such, it is not an appropriate base for an embedded appliance distribution. So we will have to move to another community distribution, and Ubuntu is the community distribution that has the highest mindshare right now.
2. One reason why CentOS Stream is not a community distribution is that CentOS Stream will not be a reliable source of security patches. Yes, I understand that Facebook has their own internal distribution based on CentOS Stream. We are not Facebook. We do not have infinite money like Facebook. We do not have a team to watch the list of RPM's and apply only the finalized tested versions to our appliance distribution. Yet another reason why CentOS Stream is not appropriate for us.
3. One thing that kept us on CentOS rather than on Ubuntu LTS was that CentOS releases were supported for twice as long as Ubuntu LTS releases. Since that's no longer true, there's no reason to stay with a RHEL-derived distribution.
4. Red Hat Software broke significant continuity with earlier versions of RHEL with Enterprise Linux 8. For example, they removed all Java applet containers from the distribution other than their own JBOSS applet container. You use Tomcat? Tough. You'll have to install it from scratch and keep it updated yourself when security patches happen. And those who use the BTRFS filesystem are completely left out.
5. And finally, Red Hat Software has proven that they will break their commitments. So.

And Red Hat's explanation? "It was putting price pressure on Red Hat’s ability to capture some of the value that we create."

Uhm, no. Because the users of CentOS, by and large, are not going to buy Red Hat Enterprise Linux. Buying RHEL does not comport with their business model if they're a business, or with their budget if they're a poor college student. They are going to move to another community distribution, either another rebuild of RHEL such as Oracle Linux or Rocky Linux, or to a different distribution altogether such as Debian or Ubuntu.

So why are we choosing Ubuntu LTS rather than some other distribution?

1. Oracle Linux: I've evaluated Oracle Linux. They've made sufficient changes to make it difficult to transition from CentOS, plus have broken a few things along the way. And they don't (can't) fix the fact that Enterprise Linux 8 broke significant continuity with earlier versions of RHEL. They can't do that and maintain compatibility with RHEL 8.
2. Then there's the general problem with all of the RHEL rebuilds: mindshare and market share is declining drastically. CentOS is really bad as a desktop operating system, so the students we get in as interns, if they know Linux at all, usually know Ubuntu Linux or one of its derivatives. Combined with its derivatives, that makes Ubuntu the most popular desktop Linux right now. For desktop Linux use, WSL has made Ubuntu the most common Linux on the desktop in our office despite the fact that we work with CentOS or CentOS-derived distributions as our target.
3. Debian Linux is quite stable, but if security fixes require effort to backport to the previous stable release, they usually aren't backported. This makes it hard for us to keep our product secure without transitioning to a newer distribution.
4. Fedora -- it moves too fast. Decent desktop OS, but not a good fit for an embedded appliance.
5. The ubiquity of Ubuntu. It is available as a supported option, and fully cloud-aware, on all major cloud platforms -- Google, Microsoft, and Amazon. To create a virtual machine with many of the other Linux distributions you must first locate a third-party image, then install that possibly untrusted third-party image. And often the third-party image is not cloud aware -- e.g. it doesn't have cloud-init installed or the native cloud tools installed, making it hard to bring up new virtual machines in an automated fashion. Ubuntu is a Tier 1 supported distribution everywhere that matters.

Thus Ubuntu.

In general, the mindshare issue -- where Red Hat's mindshare amongst potential employees is declining rapidly while Ubuntu's mindshare is becoming near ubiquitous -- makes it almost a no-brainer for us. Instead of attempting to port our software stack from Centos 7 to a RHEL 8 clone as Centos 7 approaches end-of-life, we will instead port it to Ubuntu 20.04 LTS. We use a configuration management tool to actually create our appliance (we plop the deliverables onto the CM server and it configures a copy of the base OS with our changes), and the CM tool we use hides most distribution details from us, so the job of porting is mostly going to be one of adjusting package names and occasional file locations in the manifest used to create the appliance. Furthermore, we avoid the Tomcat issue. Ubuntu ships with Tomcat, while RHEL 8 does not. Granted, that's a fairly minor issue (we already have things in our manifest that manually install various software) but why make work for ourselves?

And our student interns? The ones that aren't already Ubuntu users will become Ubuntu users. And ten years from now, they will be specifying Ubuntu Linux rather than Red Hat Enterprise Linux for their corporation's deployments. Because that's how it works. If you seed your field, crops will rise. If you instead eat your seed corn, as Red Hat's current IBM pointy headed management is doing ... well, you eat well for this quarter. But good luck reaping a crop at the end of the growing season.

- ELG