Open Source community and project failure

I had a nasty experience with an open source project recently where members were obnoxious and dismissive about a bug and possible work-arounds, and that reminded me of some things that happened back in must have been spring of 2000. I was working for a vendor of tape backup software then and we were scoping a new project that was going to take advantage of all the goodies that SCSI gave us. Prior to then, most tape backup for small computers was nasty QIC-80 type thingies attached to the floppy controller or parallel printer port, but SCSI gave us oodles of info and allowed us to do things the floppy streamers could never have dreamed of. Our goal was simple: To have plug-and-play use of tape drives and tape robots where the customer plugged in the backup hardware, installed our software, and it Just Worked. (There's those two words again!).

So anyhow, I was scoping out the tape format that we were going to use. I had a pile of tape hardware that filled an entire lab, and a pile of tape drive vendor's SCSI programming manuals that was stacked about four feet high on my desk. We had decided that we were going to keep the tape catalog in a MySQL or PostGreSQL database, but we also wanted a copy of the tape catalog on the tape itself so that tapes could be quickly re-imported into a new install of the software in the event, say, of the building burning down and the off-site backups needing to be pulled in on a new server. Without an on-tape copy of the catalog importing a tape would be a slow and painful process of reading the entire tape from front to back. The question was, where would we put the tape catalog? My boss said, "what about a partition at the beginning of the tape?" That seemed rather interesting, because usually once you start writing a tape, all you can do is either overwrite it, or append to the end of it -- you can't come back to the start and start writing again (this is because of the way the tape drives' internal compression works, amongst other things). Could we in fact do this?

So I investigated that, and found that the tape partitioning ioctl in Linux only worked with Seagate 4mm DAT tape drives. It didn't work with Exabyte drives, or Tandberg drives, or that brand new HP LTO drive, just with Seagate drives. I investigated the CDB that was being sent by the tape driver and looked at the Seagate manual, Exabyte manual, and Tandberg manual, as well as at the SCSI SMC standards document itself, and realized: That "standard" had enough holes to drive a bus through. There were basically two different ways you could interpret it, and Seagate interpreted it one way, and everybody else interpreted another way. I came up with a way of detecting in real time which partition CDB format that the tape drive used based on reading the partition table via the mode page for that, and modified the tape driver to do it correctly for every tape drive that implemented tape partitioning support -- every single SCSI drive, other than Quantum's DLT. Which, uhm, was the most popular enterprise drive. Oh well. So we didn't use tape partitions, instead we simply set a tape mark at the end of a data stream and appended the catalog to the end of the tape instead.

So anyhow, I contacted the maintainer of the Linux tape driver with my patch and an explanation of what it was doing and why the old way only worked with DDS and this would work for (long list), and he looked at it and he didn't like it, so he re-wrote it a slightly more elegant way and sent it back to me. I tried it out, it didn't work on one of the drives, I fixed one thing he'd slightly mis-interpreted and sent it back to him. I think we did a couple of iterations of this, and finally it passed both my tests for whether it actually worked on my tape drives, and his taste for what he wanted his tape driver code to look like, and it went into the next version of the Linux kernel that was released.

The other issue with tape format was that we wanted to be able to quickly retrieve a single file. It turned out that even the primitive DLT drives implemented both block location inquiry and block seek. Our archive format already aligned headers at the start of tape blocks so you could start seeking at the start of a tape block and locate the file, this wasn't a gzip'ed stream, we did things a bit smarter than that (we did something similar to what ZFS does when you turn on ZFS compression). So I hacked our data engine to issue a block position ioctl at the start of each file (and also print the file name and position) and hacked it to accept a block location and issue a block seek ioctl to find a file, as a proof of concept to verify that this would work. This worked fine on Linux and worked fine on SGI Irix. It did not work at all on Solaris -- Sun hadn't implemented *any* of the modern features of tape drives in their tape driver. We hacked our data engine to use the Solaris sgen (SCSI Generic) device directly (i.e. issue raw SCSI CDB's to read, write, seek, etc.) rather than use the tape driver, but never were satisfied with that and put that aside.

Then there was FreeBSD. I liked FreeBSD. I ran FreeBSD on my own server at home. I tried this on FreeBSD and writing tapes was very... very... slow when I had the 'print position' code enabled. Ridiculously slow. So I looked at the FreeBSD tape driver and realized what was happening. There was a call to cache flush right before issuing the position request CDB. This caused every tape drive in our lab to basically come to a screaming halt every time the ioctl happened. I commented out that single line of code, and everything worked at full speed on every single enterprise-class SCSI tape drive that was currently shipping to paying customers anywhere in the Western world. So I contacted the FreeBSD tape driver maintainer to remove this unnecessary cache flush, and...

"But the SCSI standard says that the reported tape position may not be accurate unless you issue the cache flush."

"We've tested every single enterprise drive currently shipping on the planet. They all stall and become unusably slow when you issue a cache flush anywhere but at the end of a dataset, and they all report the position accurately without the cache flush, we can go back to the reported position and always find the data we expected there."

"But... the SCSI standard..."

"Have you found any drive that actually complied with the SCSI standard? The fact is that the cache flush makes the ioctl useless because it makes it too slow, and every tape vendor seems to ignore that little note in the SCSI standard and just do what we expect." (Followed with performance data on actual drives with / without cache flush and noting that this worked properly *without* the cache flush on both Linux and Irix, neither of which had the cache flush in their tape position ioctl).

It went on and on and on. I was an interloper from outside their little community, so I had to be put into my place, despite the fact that all I wanted was for the stupid ioctl to Just Work, as it did on Linux and Irix, so that we could support FreeBSD as a Tier 1 platform rather than relegated to a Tier 2 platform (one which could not host tape drives but merely be backed up to another server that was hosting tape drives). Eventually after much grumbling about "well okay, it is wrong according to the SCSI standard but if that's how the tape drives actually work, we'll accept this change" they queued it up for a future release of FreeBSD, but by that time it was rather too late in the process -- we'd moved on to actually implementing our product, and FreeBSD wasn't scoped because there was no version of FreeBSD currently shipping that had the tape driver functionality we needed for our product.

Whenever I think about that difference between the Linux and FreeBSD communities in 2000, the reason Linux rules the world and FreeBSD has been relegated to a side note becomes a bit clearer. At the beginning, FreeBSD was clearly technically superior to Linux. For example, the FreeBSD scheduler could handle heavy loads without falling over, and the FreeBSD memory management system could handle heavy memory pressure without falling over, and neither was true of Linux in 2000. But the deal is, the world is not a meritocracy. The majority of the world will accept "Good Enough" if it means they don't have to put up with a lot of friction. The FreeBSD community in 2000, reeling from loss after loss to Linux with their biggest deployment (Hotmail) being converted to lowly Windows, was defensive and paranoid and not very pleasant to deal with (it's considerably better now BTW). Experienced IT people going into that environment and getting that kind of attitude simply rolled their eyes and went with that Linux thing, which was Good Enough for a lot of things even if it wasn't as good as FreeBSD.

So what are my takeways? Well, if you're involved in an Open Source community, whether as a code contributor, supporter, or just general advocate, here's my takeaways:

• Stowe the attitude. If you appear defensive and hostile, experienced IT folks simply won't use your stuff.
• If someone contacts you with a problem, and a proposed solution, don't dismiss them as an idiot until you actually evaluate the technical content of their message. If you don't agree with them, give technical reasons why their solution doesn't work, don't make personal attacks upon the person calling them an idiot or stupid or incompetent or whatever.
• If it's clear that someone is entirely off base, either reply "Thank you for your contribution, we will consider it" or simply hit the DEL key.
• Police your fanboys if they're trashing your mailing list or forums giving attitude to folks who have problems, suggestions, or contributions. One of the things that hindered uptake of Linux in that era was the Linux fanboyz with their "Linux rulez Windows drulez" attitude. IT people want to solve problems, they're not interested in penis measuring contests, and they're going to use the solution to their problem that comes with the least baggage. It doesn't matter whether your project really *is* the best or not, I repeat once more: The real world is not a meritocracy. If some other product is Good Enough, Just Works, and comes without the baggage, guess whose product is going to win?

And this is rather too long, so one final takeaway: Treat every person who contacts you as if they are the second coming of Dennis Ritchie. It doesn't matter if your first impression is that they're the biggest moron on the planet. They are your customer. They are the people who you want to use your product, assuming you're writing your open source software to be used by other people, not just as a private project for your own personal use. Even the dumbest customer has something to contribute, if only a lesson in how stupid customers can be so that you can engineer your product to survive their unkind ministrations. Treat them as if they have something to contribute, and you may be surprised to find out that they do have something to contribute. Treat them like cr*p, and they'll return the favor and your project will die on the vine.

And that's my takeaway. Sorry about the long history lesson, but there was a purpose in it -- and hopefully you've figured that out by now. So it goes.

-ELG

Do Not Annoy The Geek

What brought that up? Well, two things. The first is that I'm updating my report on virtualization systems with a new round of testing. This new round of testing includes some entrants that didn't exist during the first round -- Scientific Linux 6 / Centos 6.0, Red Hat Enterprise Linux 6.1, Ubuntu 11.04, and OpenSUSE 11.4. I'll write that up as soon as I finish with OpenSUSE 11.4, the only one left to evaluate. The second thing that happened was that my Macbook Pro decided it didn't like to sleep or shut down cleanly, which annoyed me because, reading forums, it appears that the only way to fix it is to do a clean wipe and re-install from scratch, re-install my applications from scratch, then restore only data, not configuration info. Makes me feel like a Windows user. Which annoys the geek.

So, how did the round of virtualization evaluations annoy the geek? Well: SL/Centos, Ubuntu, and OpenSUSE can be downloaded from their respective web sites. Red Hat requires you to sign up for an evaluation. Okay, so I already have an account on their site from the *last* version of their software that I evaluated back in December of last year (the original RHEL 6.0 release), so I went back in and signed up again. And promptly got rejected. "We do not accept personal email addresses for evaluations." I.e., they only want corporations to evaluate their software. I shrugged, changed my email address to a .com address that I registered over ten years ago but have never used (but which forwards email to my personal inbox), and downloaded the software. But I was annoyed.

So why is this important? Simple. I am not a product manager. I don't make final decisions about what to buy. But I am, more often than not, the person that the product manager or the IT manager comes to when they have a problem and want to know, "what technology do I need?". People like me are called influence leaders, because our combination of technical skills and communications skills, our understanding of both technology and actual business issues, means that folks who have one but not the other come to us to decide what the next round of innovation deployment is going to be. My estimate is that I've personally influenced or caused to happen over $5M/year of purchases over the past ten years. And that is true of geeks in general -- geeks may not be the people who sign the checks, but they're the people who evaluate the technology and tell the people who sign the checks what to buy.

So anyhow, back to Red Hat. That question on their web site is completely anti-geek. See, while evaluating virtualization technology is part of my job description (seriously -- it's there in black and white), for the most part I do it as a private citizen, not as a qualified lead. That's because otherwise I get bombarded with spam emails and phone calls asking me to buy stuff -- but I don't buy stuff, I evaluate stuff. Other people in my company buy stuff, and no, you are not getting their name or phone number from me because then they'll come to me and ask me questions about you that I can't answer until I finish evaluating your product. But Red Hat's marketing department is looking for qualified leads -- potential check-writers that they can bombard with phone calls and spam emails to sell stuff. But you won't get very far bombarding me with emails and phone calls, because I am a technical leader, I don't write checks and you'll do nothing but annoy me if you bombard me with phone calls and emails. If you want sales generated from me, you'll have to do it the old-fashioned way -- by letting me do a technical evaluation of your product for a month or so to see if it'll fulfil our company's needs better than the competition.

So how important is this sort of geek cred? Well, let's look at Sun Microsystems. Back in the day, they had geek cred out the yazoo. Geeks lusted after a Sun workstation on their desk, and fumed at the notion that they had to settle for a lowly PC, because Sun workstations with their BSD-based SunOS were geek nirvana with full geek programming toolkit including a "C" compiler and programmable shell scripting environment. Then Sun decided, upon releasing AT&T System V.4 as "Solaris", that they would no longer include the software development tools with their workstations. That was the beginning of the end for Sun, it just took fifteen years for it to catch up with them. The geek lust eventually moved to PC's running Linux. Linux went up and up and up, Sun lost market share to PC's running both Windows and Linux year after year, and to Apple once Apple moved to Unix and started including development tools to attract the geeks to their platform, until finally Sun got acquired for the few competitive products they had left. Sun made a last-ditch effort with OpenSolaris to regain geek cred, but it was too little, too late -- Linux had such a lock on geek mindshare that there was no way Sun could regain enough mindshare to make a difference. When geeks lust for Unix servers and workstations now, they lust for well-endowed Linux servers and MacOS workstations, not Sun ones -- but that was *not* true in 1990, in 1990 if you were a geek, you had serious Sun lust.

Of course, this points out two things about geek cred: a) it takes a long time to build up to the point where it makes or breaks technology companies (but in the end it always does), and b) once you lose it, getting it back is a real problem. Well, and c) if your product is pretty much the only thing that solves a particular problem, you don't need geek cred (thus why the IBM 370 architecture machines still sell plenty despite being utterly unfashionable in geek circles). So if you're a marketing type looking for the long term, what do you need to know?

1. Provide evaluation copies of your software without jumping through too many hoops. Basically, anything that involves delay is too many hoops for geeks, geeks are impatient and want instant gratification. At one employer they made it so difficult to download an evaluation copy of the software that virtually nobody did so -- the only way you could do it was to basically become a qualified lead and have one of their salesmen call you. But that annoys the geek. One reason why my employer uses VMware ESXi heavily is because we in the technical staff could download the evaluation version of the software, see that it worked quite well for our projects, and then, and only then, did we go to our superiors and have them negotiate whatever licensing was needed to make our projects work. If VMware had made it hard to get evaluation versions of their software, that would have never happened -- we would have used one of the Open Source virtualization platforms despite their limitations.
2. Describe what your product does on your web site. Seriously. I can't count the number of times I read some breathless but vague marketing hype on the web site, try to infer whether it solves the problem I'm trying to solve, download the actual eval product... and meh. It didn't do anything like what I thought it was when I read the breathless hype on the web site. Tell me what problem you're trying to solve with your product. Yes, I know you don't want to limit yourself, but please. Give me a clue *without* having to waste my time, because wasting time annoys the geek!
3. Provide technical documentation on your web site. At a previous employer I once wrote a blog entry for a customer-facing company blog that described, with some technical detail, exactly what it is that our product did. That blog entry got kiboshed by a PHB because "it lets out too many details of our secret sauce". Thing is, geeks don't like secrets. They want to know what it does and how it does it. If you don't tell them, they'll decide you don't have anything but smoke and mirrors and go elsewhere. It always frustrated me at that company that we had a cool product, but nobody knew it -- because we refused to tell it. But there's no such thing as secrets in technology. Everything can be reverse-engineered. So keeping technical details close to the vest doesn't accompany anything but annoying the geek -- and hurting your street cred, since it makes you look like a fly-by-night peddling smoke and mirrors.
4. Provide a programmable interface if reasonable and possible. End users won't care, of course. But one reason why Linux (and eventually MacOS X) won out over the other Unixes was because they were very open to programmers. Linux of course comes with the complete source code, while MacOS has source available for many components of the system plus has a free programming tool set available for it (XCode) as well as having that all-important Unix shell prompt with all the geeky Unix tools. Meanwhile, the other Unixes required you to fork over significant sums of money to get their programming toolkits. One of the things Microsoft has done right recently is to include PowerShell with all their systems and export most of their OS API's as PowerShell objects. It has decidedly helped their reputation in geek circles -- you'll notice that while you still have some geeks saying "Microsoft is Teh Eeeevil", most have moved to a position of neutrality on the whole subject of Microsoft. (Disclaimer: This is being written on Firefox running on Fedora 15 running under VirtualBox on a Windows 7 host :). I still wouldn't deploy a Windows server unless you paid me a significant sum of money to do so, but people who put Linux on the desktop because Microsoft is 'Teh Eeevil' are just being twits.
5. DON'T CRASH! Seriously. Crashes annoy geeks even more than missing features. If there's a choice between having a cool feature and having a reliable product, err on the side of a reliable product. Nobody ever lost customers by having a reliable product, but once you lose cred by getting a reputation for having a crashy product... well.
6. And -- participate in geek community. There were some tools at one employer that we could have contributed to the Open Source community that would have worked quite well at building geek cred, but management was totally against it because it would "divulge corporate secrets". Well, that company is out of business now. Openness sells. Secrecy doesn't. If you can't out-innovate your competition once they figure out your "secrets", you're in the wrong business -- because the technology industry moves so fast that any "secrets" you divulge *should* be obsolete long before any competitor can take advantage of them. If not, you're as doomed as the startup I worked for that took three years to create their first product, which was two years obsolete by the time they finally got it out the door late, slow, and buggy. You have to move *fast* in the technology industry, and if you can't out-run any "secrets" you divulge to the geek community, you're going to be out of business soon anyhow.

So anyhow, just one of those things to keep in mind if you're a marketing person wondering how to get market share. Find some geeks and ask them what annoys them about your product or about how your company sells your product. Then fix it. It won't create short term sales figures, but if you're wanting a long career for a stable company, that's how to do it -- something that Red Hat needs to remember, in their current rush to go complete pointy haired boss on the geeks when it comes to evaluating their enterprise products.

-ELG