End of the FreeBSD ZFS experiment

So my write performance with ZFS on FreeBSD was abysmal. The final straw was when I was copying from one pool to another and it was running at roughly the same speed as a floppy disk. I made numerous attempts to tune both ZFS and the iSCSI initiator, and nothing that I tried made any real long-term difference. Things would speed up after I tweaked stuff, then slowly settle back down to a tedious crawl.

Out of frustration I created a 64-bit Centos 6.2 image with the exact same stats as the FreeBSD image and installed the native Linux port of ZFS. This requires a stub that goes into the kernel to meet licensing requirements, then compiles against that stub code. I then shut down the FreeBSD virtual machine, installed the iSCSI initiator on the Linux machine and scanned both of my iSCSI storage arrays to tell it about the initiator, then went to the storage arrays and unassigned the volumes from the FreeBSD machine and assigned them to the Linux machine instead. Then I scanned and logged them in on the Linux machine, and did the following command at a root login:

zpool import -f -F -m -a

They imported cleanly and everything came up.

So the next thing I did was set off my copy going. I am ZFS-mirroring between the two iSCSI storage arrays and I only have a single gigabit Ethernet port from my ESXi box to the storage arrays, so this put a maximum throughput of roughly 100 megabytes per seconds for both read and write. ZFS did this throughput to the storage arrays handily.

So clearly the problem is not ZFS. And FreeBSD has been shown to have good ZFS performance with DASD (direct-attached storage devices). So the fundamental problem appears to be the FreeBSD iSCSI initiator. I don't care enough to diagnose why it's so terrible when used with ZFS despite the fact that I hit all the tuning flags to turn up the queue depth etc., but the end result is that ZFS combined with iSCSI on FreeBSD is a no-go.

On Linux, BTW, it Just Worked once I built the zfs RPM's and installed them. I'm performing at the full speed of my network. And remember, that's the ultimate role of computer technology -- to Just Work, leaving the hard stuff of deciding what's going to go onto that server to the humans. My goal was to move bytes from point A to point B as fast as my ESXi system could do so, given the fact that I need to set up Etherchannel on my antique SMC switch to do ESXi trunking to get data from point A to point B any faster than gigabit Ethernet will take me. (I don't know if the antique will even do it, this is a production ESXi box so I have to set an outage time at an oddball time before I can move the iSCSI network Ethernet wires to the supposedly Etherchannel-trunked ports and flip the ESXi vswitch bit to ip-ip hash to split the traffic between the two trunked ports). So while it sucks that I have to manually build and install ZFS on Linux, the final result appears to work far better than it really should, considering the very beta-quality appearance of that ZFS On Linux site and the rapid updates they're making to the software.

-ELG

ZFS caveats and grumbles

This is a followup of how ZFS has worked in my test deployment.

First of all, don't even try ZFS deduplication on any real data. You require about 2GB of memory for every 100GB of deduplicated data, which means that on a 2 terabyte filesystem you'd need around 400GB of memory. If you don't have that much memory, ZFS will still work... at about the same speed as a 1985-vintage Commodore 1541 floppy drive or a 9600 baud modem. So reluctantly I have to say that ZFS's deduplication capabilities are pretty much a non-starter in most production environments.

Compression, on the other hand, appears to work much better. When I turn on compression the system gets no slower as data gets added, it still remains the same level of slow.

Finally: ZFS supposedly is a "zero startup time" filesystem. But the reality is that if a pool gets shut down uncleanly, when you start up ZFS it does a potentially very lengthy integrity check, as in, can take as long as an hour to run. While still better than NTFS or Linux ext4 (both of which can take hours for any reasonably-sized filesystem), "quick" recovery from an unclean system outage is relative -- don't rely on the system being back up within minutes if someone managed to yank both power cords out of the back of your server while trying to decant some *other* system from the rack.

Next up: FreeBSD has an iSCSI initiator. But it is not integrated into system boot in any way, and ZFS if you enable it in rc.conf comes up first thing, long before networking, so you cannot enable ZFS in rc.conf (or use it as a system filesystem) if your storage array is iSCSI-connected. My rc.local looks like this now:

bsdback# cat rc.local
#!/bin/sh
iscontrol -c /etc/iscsi.conf -n tzpool1
iscontrol -c /etc/iscsi.conf -n mzpool1
iscontrol -c /etc/iscsi.conf -n tzpool2
iscontrol -c /etc/iscsi.conf -n mzpool2
sleep 5
/etc/rc.d/zfs onestart
/etc/rc.d/mountd stop
/etc/rc.d/mountd start

The mountd stop / start is necessary because mountd started up long before, noticed that the zfs mountpoints in /etc/exports didn't actually exist yet, and didn't export them. If you are exporting ZFS mountpoints via NFS on FreeBSD, this is the only way to make it happen correctly as far as I can tell -- even if you exported them via zfs, mountd starts up, looks at the zfs exports file, and refuses to export them if zfs isn't up yet.

And rc.shutdown.local looks like:

/etc/rc.d/mountd stop
/etc/rc.d/nfsd stop
/etc/rc.d/zfs onestop

This is the only way I can get the sequencing right.

Note that Red Hat Enterprise Linux has had the ability to properly sequence filesystem bringup so that iSCSI filesystems get mounted after networking (and iSCSI) comes up for quite some time -- since Red Hat Enterprise Linux 4, circa 2004, in fact. RHEL has also had the ability to automatically bring up the iSCSI targets after networking and before mounting network file systems via their equivalent of the rc.conf system (SysV Init) since 2004. This is an area in which FreeBSD lags, and should be justifiably flamed as lagging. You should not need to write custom rc.local scripting to bring up standard parts of the FreeBSD operating system in the correct order, it should Just Work(tm) after properly setting up the rc.d dependencies and ZFS flags to make it Just Work. ZFS needs to have a pre-networking and post-network two stage bringup so that any pools not located during the pre-networking stage can be searched for during the post-networking stage, and iSCSI needs to have a its own rc.d script that brings it up at the end of the networking stage.

All in all, ZFS on FreeBSD is working for me in my test deployment in my IT infrastructure, but it's not as seamless as I had hoped. When I look for IT technology I look for something that Just Works(tm). ZFS on FreeBSD would Just Work(tm) if I were using DASD's, but since I'm using network-attached storage, it's more of an erector set scenario than I'd like.

ZFS -- the killer app for FreeBSD?

Okay, so here's my issue. I have two iSCSI appliances. Nevermind why I have two iSCSI appliances, they were what was available, so that is what I'm using. So I want my backups mirrored between them. Furthermore, I want my backups to be versioned so I can access yesterday's backup or last month's backup without a problem. Furthermore, I want my backups to look like complete snapshots of the system that was backed up as of that specific point in time. Furthermore, because my data set is highly compressible in part and highly de-duplicable in part, I want it compressed as necessary and dedupe'ed as necessary.

So, how can I do this with Linux? Well... there is a LUFS version of ZFS that will sort of, maybe, do it in a way that regularly loses data and performs terribly. There is BTRFS which is basically a Linuxy re-implementation of ZFS that does compression but not deduplication, and which is still very much a beta-quality thing at present, they didn't even have a fsck program for it until this spring. And ... that's it. In short, I can only do it slowly and buggy.

So at present I have a FreeBSD virtual machine in my infrastructure happily digesting backups and bumping the snapshot counter along. And ZFS is a first-class citizen in FreeBSD land, not a castaway in LUFS-land like on Linux. I'd love to use BTRFS for this. But BTRFS today is at about the same stage as ZFS on Solaris in 2005, when it was an experimental feature in OpenSolaris, or ZFS on FreeBSD in 2008 when the first buggy port was released. ZFS on FreeBSD is stable and rock solid today, and BTRFS, realistically, isn't going to be stable and rock solid for another three or four years at least.

So if you haven't investigated ZFS on FreeBSD to manage large data sets in a versioned, compressed, and deduplicated fashion, perhaps you should. It solves this problem *today*, not a half decade from now. And a bird in hand is worth a dozen in four years.

- ELG

Random notes on iSCSI storage

This summary is not available. Please click here to view the post.

Best practices for virtualization

A series of notes...

1. vSphere/ESXi: Expensive. Inscrutable licensing scheme -- they have more SKU's than my employer, almost impossible to tell what you need for your application. Closest thing to It Just Works in virtualization. Call them the Apple of virtualization.
2. Xen : Paravirtualization gives it an advantage in certain applications such as virtualized Linux VM's in the cloud. Paravirtualization generally is faster than hypervirtualization, though most hypervisors now include paravirtualized device drivers to ease that pain. Xen doesn't Just Work, it's more an erector set. Citrix's XenServer is the closest that Xen gets to vSphere's 'Just Works', I need to download it and try it out.
3. KVM : The future. Integrating the hypervisor and the OS allows much better performance. That's why VMware wrote their own kernel with integrated hypervisor. Current issues: Management is the biggest difficulty. There is difficulty creating clustered filesystems for swift failover or migration of virtual machines (ESXi's VMFS is a cluster file system -- point several ESXi systems at a VMFS filesystem on a iSCSI or Fiber Channel block storage, and they'll all be able to access virtual machines on that system). Most KVM systems set up to do failover / migration in production use NFS instead, but NFS performs quite poorly for the typical virtualization workload for numerous reasons (may discuss later). Closest thing to VMFS performance for VM disks is using LVM volumes or clustered LVM (if using iSCSI block storage), but there are no management tools for KVM allowing you to set up LVM pools and manage them for virtual machine storage with snapshots and so forth. Virtual disk performance on normal Linux filesystems, via the qcow2 format, sucks whether you're talking ext4, xfs, or nfs. In short, the raw underlying bits and pieces are all there, but there is not a management infrastructure to use them. Best practice performance-wise for clustered setup: NFS share for metadata (xml description files of VM's, etc.), iSCSI or Fiber Channel block storage possibly sliced/diced with clustered LVM for the VM disks.

So what am I going to use today if I'm a busy IT guy who wants something that Just Works? VMware vSphere. Duh. If, on the other hand, I'm building a thousand-node cluster, a) it's probably my full time job so I have time to spend futzing with things like clustered LVM, and b) the cost of vSphere for a cluster that large would be astronomical so would decidedly make paying my salary to implement Xen or KVM on said cluster more palatable than paying VMware.

-ELG

Random notes on automating Windows builds

1. Install the version control system of choice. In this case, bitkeeper, but any other CLI-drivable vc system will work.
2. Check out your directory tree of the various products you are going to be building.
3. Install the *real* Microsoft Visual Studio 10
4. Create a solution (Microsoft-speak for "makefile" though it isn't) for each of the individual solutions you are building as part of your overall product and make sure each solution builds. This will be saved in the file Foo.vcxproj (for some solution named Foo) in each solution's root directory.
5. Add the directory that 'devend' and 'nmake' lives in to your PATH in your system config. Control Panel->System->Advanced SYstem Settings->Environment Variables, edit user variable Path. My Path looks like: C:\Users\eric.green\bitkeeper;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin
6. Create a standard Unix-style makefile in the parent directory that has a list of the subdirectories to recurse into, then in each subdirectory, a Makefile that has 'devenv /build Foo.vcxproj' to build and 'devend /clean Foo.vcxproj' to clean.
7. Test your make file with 'make', make sure that the proper .exe files are produced in each of your subdirectories.
8. With Visual Studio closed, install Wix and Votive
9. Use Votive to build a WiX project file and "compile" it to XML.

Once you've done this, then you can edit the Makefile at the root of your project so that after it recurses the directory, it runs the WiX commands:

1. candle product.wxs
2. light product.wixobj

The output should be product.msi.

Install Jenkins to integrate with the source control system, check from time to time for new checkins, then fire off a build via use of nmake when the checkins happen. Jenkins does run under Windows, with some caveats. See e.g. Setting up Jenkins as a Windows service. Biggest issue may be getting email to go out on Windows, will have to investigate that further once I get to that point.

--ELG

Microsoft and the epitomy of fail

Ever wonder why Microsoft lost the Internet? They did. Nobody uses Microsoft products for Internet services except a few losers that don't know any better. The cloud? It's a Linux world, baybee.

Well, here's the deal that I've found out this past six weeks or so: Developing for Microsoft products is a painful and expensive process. What that means is that Microsoft has lost the student and hobbyist markets, and that's where the people who create new stuff come from -- they don't come from stuffy old companies that have $10K hanging around for Microsoft licenses for a single engineering team.

So anyhow, I recently created my first tray app. It used a couple of MFC functions where I could not find an equivalent anywhere else in the system that would do what I needed done. Because of that, it would not compile with Microsoft's "student and hobbyist" product, Microsoft Visual C++ Express. Once I did a workaround for that (temporary install of the demo version of the full Microsoft Visual Studio just to get access to the MFC include files and library), then there was another issue: Visual C++ Express won't create a .msi installer file for distributing your program. I worked around this by using an open source program called WiX to build an XML template for my package, and successfully managed to create a .msi file. I installed it. I used the control panel and uninstalled it. Yay.

But here's the problem. I'm a hobbyist when it comes to Microsoft software. As a hobbyist I only write Open Source software (if you want closed source software out of me I'm happy doing it, just pay me $$$, but that's not my thing with hobby software). As a hobbyist I don't release software that can't be compiled unless you pay Microsoft hundreds of dollars for the full Microsoft Visual Studio, or do possibly-illegal workarounds like installing demo versions on top of crippleware. So you'll never see this tray app and it's probably the last time I write anything for Windows that I'm not paid to write.

Multiply that decision by tens of thousands of hobbyists who look at the same situation and instead go write software for Linux, and you understand why Microsoft lost the Internet. I did this because I needed to learn how to work with Microsoft's software to do some stuff at work, but most people simply aren't as driven as me when it comes to learning new things. They take the easy path... and that's Linux.

Too bad, Microsoft... your stuff actually isn't that bad, it's ugly and a bit incoherent but then so is Linux. But if you make it hard for people to get used to writing software for your platform on a hobby basis, you lose. It's that simple. Microsoft Visual Studio Express is useless -- you can't write real programs for Microsoft without MFC, and creating MSI files should Just Work rather than having to go search for a third-party tool. With Linux, it Just Works -- fire up Eclipse and start developing C++ or Java programs, that simple. With MacOS, it Just Works -- fire up XCode and start developing Mac programs in Objective C or other supported languages, that simple. Windows? It's a painful experience trying to get around cripple-ware. And once that happens, there's one word to describe your company: Fail.

What's Microsoft's future? Microsoft really has no future other than as a legacy company. They lost the Internet and the Cloud because they preferred wringing money out of their devtools division (and it can't be a huge amount of money even) to fostering the next generation of innovators with free or inexpensive tools to use to write software for Microsoft's platforms, and they'll lose the next major innovation to happen in computing too. And one of these days, the accumulated sum total of these innovations will render Microsoft as irrelevant as Unisys -- just another legacy company milking its legacy products for service income long after they've become irrelevant to the majority of the industry.

-ELG

Gnome 3, Mac OS, and the Geeks (Part 2)

So in part two of this comparison, we're going to talk about workspaces.

Both Gnome 3 and Mac OS Lion have the concept of a linear ribbon of "workspaces" or virtual desktops. Gnome 3 lays out its workspaces vertically, while Mac OS Lion lays out its workspaces horizontally. So how do they compare on some routine workspace operations? Let's see...

Switching to Workspaces:

Gnome 3: There are two basic ways to do this: 1) Press the Windows key or swoosh mouse to left top of screen in order to activate the Activities screen. Move the mouse to the right side of the screen to make the workspace list pop out and select the workspace you wish to be in. OR: 2) Press CTRL-ALT-down to go to the next workspace down, or CTRL-ALT-up to go to the next workspace up. Mac: 1. Press F3 button on a recent Mac to go into Mission Control. You can also set a multi-touch gesture to do this (mine is three fingers up on the trackpad). Your workspaces will be listed horizontally at the top of the screen, move your mouse to and click on the one you want to go to. 2. Assign a multi-touch gesture to next-workspace and previous-workspace. Mine is three fingers left/right on the trackpad. 3. Assign a key sequence to next-workspace and previous-workspace. Mine is CTRL-left and CTRL-right. Because I have an Apple laptop, I can use multi-touch gestures to move left and right and to activate the Workspace switcher. This gives me one more option on the Mac. But reality is that navigating to a workspace is ridiculously easy on both systems, either from keyboard or via mouse/trackpad.

Creating Workspaces:

Gnome 3: There is always one "blank" workspace at the end of the list of workspaces. If you move to that blank workspace and open a window there, a new blank workspace is created after it automatically, without you having to do anything.

Mac OS Lion: Press the F3 function key or use the multi-touch gesture to get to Mission Control. Move your mouse pointer to the right top of the screen. A new shadowed-out workspace will pop out of the ether. Click on it. You will now have a blank workspace to work in.

The Gnome approach can be done without touching the mouse, and actually requires no intervention on your part to create the new workspace -- it simply gets created when you need it. It Just Works, which is what a computer is supposed to be -- something that Just Works, without you having to do fiddly things to make it work. It will be interesting to see whether Apple or Microsoft copy this feature in their next release.

Deleting workspaces

Gnome 3: When you close the last window on a workspace, the workspace is deleted and you are then placed on the Activities screen, from whence you can select another workspace to work in using either CTRL-ALT-up/down or the workspace pop-out at the right of the screen. This prevents workspace clutter where you have lots of those automatically-created workspaces hanging around. (Note: I am aware of the "persistent workspace" plugin, I am comparing stock configurations).

Lion: Activate Mission Control. Move to the top left corner of the little workspace icon you want to zap. A little X will appear. Click on that X. The workspace will go bye-bye, and any windows on it will be moved to the first workspace.

Again the Gnome 3 approach to this appears to me to be much simpler than the Lion approach. There's no fine motor skill needed to move the mouse pointer to the exact point where the X will appear, you simply close your windows and poof, you're done. Nothing to remember, nothing to discover, it Just Works.

Moving Windows to a Workspace

For both Lion and Gnome 3 you simply move to the workspace containing the window you want to move, trigger the Activities or Mission Control screen, grab the window you want by clicking on it, and drop it on the icon of the workspace you want to move it to.

-------------------

So that's workspaces. As you can see, Gnome 3 is quite competitive with the state of the art that is Mac OS Lion. So that brings up the next question: Why do so many of the hard-core Linux geeks hate Gnome 3? I'll discuss that in the next installation of this series.

-ELG

Gnome 3, Mac OS Lion, and the geeks (Part 1)

I've been using Gnome 3 on Fedora 15/16 and Mac OS Lion on a new Macbook Pro for several months now. What I see are two platforms that have done a significant re-thinking of the user interface to deal with some unpleasant facts:

1. The old paradigm of using menus to select programs has reached its expire-by date because the menus have attained a depth that nuclear submarines would love to attain
2. The population of most major technological countries is aging, and our old eyes simply cannot read the print on those tiny little pull-down menus anymore,
3. Fine motor control of old folks is pretty bad. We can manage to swoosh the cursor to the corner of the screen, or hit a big icon, but fiddly little menu items are hard for us to nudge a mouse into the right box to navigate a pull-down menu, and finally,
4. Pull-down menus simply aren't compatible with small touch-screens, because fingers are too fat to select tiny little things and you can't see them on a small touch screen like on a tablet anyhow.

In the process of dealing with these unpleasant facts, Mac OS Lion and Gnome Shell have also done a major re-thinking of how you do many common tasks in order to reduce the number of mouse movements / key strokes needed to do them. So let's look at some common tasks...

Program selection, Gnome 3:

Method a: Swoosh the mouse to the top left of the screen (one movement). Swoosh the mouse to the 'Applications' tab (one movement). Click left button. Select program from list of icons (possibly using the scroll wheel to scroll up and down the list).

Method b: Press the Windows key. Type the first couple of characters of the program you want to run. Use the arrow up-down to move the highlight to the icon of the program you want to run. Press ENTER.

Program selection, Mac OS Lion

Method A: Move the mouse to the icon of a rocketship on the toolbar. Click. Move mouse pointer to icon of program to run, possibly using left-right wobble wheel on your mouse or left-right two finger swipes on trackpad to go to next page of programs. Click on program.

Method B: Set a hot corner in preferences (I set bottom left), swipe mouse to there, move mouse to program to run, click.

Method C: Set a hot key in preferences such as control-opt-l, use left-right arrow keys to navigate pages. Unfortunately Mac OS has no way that I can find of using the keyboard to actually run one of those programs, you must actually navigate your mouse pointer to it and click it.

Verdict: If using a keyboard, Gnome 3 is very easy to navigate and uses a minimum of keystrokes to locate and run a program, requiring no mouse input at all. If using a mouse, the fact that you can get to the Applications icons immediately in MacOS, vs. having to click on the Applications tab after swooping to the corner, makes MacOS require one less mouse movement and one less mouse click. Score: TIE.

Select A Window, Gnome 3

Method 1: Ye olde alt-TAB, with a kick. For applications with multiple windows, you are shown a down-arrow and a down-cursor will then open window previews. Select the window you want.

Method 2: Hit the Windows key or swoop mouse to left top of screen. The windows will then swoosh out into a thumbnail pane view. Click on the window you're interested in.

Method 3: Hit the Windows key or swoop mouse to top left of screen. Select the dock icon representing the program you want to switch to. Either click it to go to the topmost window, or right-click it to select which window you want.

Select A Window, Mac OS Lion:

Method 1: Command-tab to get a list of running programs. While still holding down Command-tab, then use the arrow keys to move left-right to program whose window you want to see. Release command-tab while that program is highlighted. Note that unlike with Gnome 3, you do *not* get to choose which exact window of the program is going to be switched to -- you get whatever MacOS feels like giving you.

Method 2: Set a mouse button or gesture (I use the Page Forward button on my Logitech mouse or a triple-finger-up gesture on the trackpad). Invoke said mouse button or gesture. The windows will zoom out to pane/thumbnail view. Navigate mouse to the window you want and click on it.

Method 3: Move mouse to bottom of screen, click the dock icon of the program you want to switch to. Right-click will allow you to choose which of the windows to switch to.

Verdict: Gnome 3 wins on command-tab, its command-tab function is full-featured and works well. It ties on mouse button or gesture. It loses on dock, but only barely because its dock is not always visible and you have to move your mouse to the top right of the screen to show it, but that's only one mouse movement extra so not a huge loss. So Gnome 3 by a nose.

-----------------------------------------------------------------------

So what have we learned thus far? Well, 1) For two specific tasks, both Gnome 3 and Mac OS Lion have done a lot of work on reducing the amount of mouse movements and/or keystrokes needed to do these tasks, and almost completely eliminated any necessity for fine motor movements or reading of tiny print, and 2) Gnome 3 is quite competitive with Mac OS Lion in doing these tasks. In the next part of this series I will compare some other common operations, and finally I will summarize the results and examine one of the more bizarre things that has happened since the release of Gnome 3 -- the rabid condemnation of it by early Linux developers (including Linus Torvalds) who appear to despise it, and what that means for Linux.

Do Not Annoy The Geek

What brought that up? Well, two things. The first is that I'm updating my report on virtualization systems with a new round of testing. This new round of testing includes some entrants that didn't exist during the first round -- Scientific Linux 6 / Centos 6.0, Red Hat Enterprise Linux 6.1, Ubuntu 11.04, and OpenSUSE 11.4. I'll write that up as soon as I finish with OpenSUSE 11.4, the only one left to evaluate. The second thing that happened was that my Macbook Pro decided it didn't like to sleep or shut down cleanly, which annoyed me because, reading forums, it appears that the only way to fix it is to do a clean wipe and re-install from scratch, re-install my applications from scratch, then restore only data, not configuration info. Makes me feel like a Windows user. Which annoys the geek.

So, how did the round of virtualization evaluations annoy the geek? Well: SL/Centos, Ubuntu, and OpenSUSE can be downloaded from their respective web sites. Red Hat requires you to sign up for an evaluation. Okay, so I already have an account on their site from the *last* version of their software that I evaluated back in December of last year (the original RHEL 6.0 release), so I went back in and signed up again. And promptly got rejected. "We do not accept personal email addresses for evaluations." I.e., they only want corporations to evaluate their software. I shrugged, changed my email address to a .com address that I registered over ten years ago but have never used (but which forwards email to my personal inbox), and downloaded the software. But I was annoyed.

So why is this important? Simple. I am not a product manager. I don't make final decisions about what to buy. But I am, more often than not, the person that the product manager or the IT manager comes to when they have a problem and want to know, "what technology do I need?". People like me are called influence leaders, because our combination of technical skills and communications skills, our understanding of both technology and actual business issues, means that folks who have one but not the other come to us to decide what the next round of innovation deployment is going to be. My estimate is that I've personally influenced or caused to happen over $5M/year of purchases over the past ten years. And that is true of geeks in general -- geeks may not be the people who sign the checks, but they're the people who evaluate the technology and tell the people who sign the checks what to buy.

So anyhow, back to Red Hat. That question on their web site is completely anti-geek. See, while evaluating virtualization technology is part of my job description (seriously -- it's there in black and white), for the most part I do it as a private citizen, not as a qualified lead. That's because otherwise I get bombarded with spam emails and phone calls asking me to buy stuff -- but I don't buy stuff, I evaluate stuff. Other people in my company buy stuff, and no, you are not getting their name or phone number from me because then they'll come to me and ask me questions about you that I can't answer until I finish evaluating your product. But Red Hat's marketing department is looking for qualified leads -- potential check-writers that they can bombard with phone calls and spam emails to sell stuff. But you won't get very far bombarding me with emails and phone calls, because I am a technical leader, I don't write checks and you'll do nothing but annoy me if you bombard me with phone calls and emails. If you want sales generated from me, you'll have to do it the old-fashioned way -- by letting me do a technical evaluation of your product for a month or so to see if it'll fulfil our company's needs better than the competition.

So how important is this sort of geek cred? Well, let's look at Sun Microsystems. Back in the day, they had geek cred out the yazoo. Geeks lusted after a Sun workstation on their desk, and fumed at the notion that they had to settle for a lowly PC, because Sun workstations with their BSD-based SunOS were geek nirvana with full geek programming toolkit including a "C" compiler and programmable shell scripting environment. Then Sun decided, upon releasing AT&T System V.4 as "Solaris", that they would no longer include the software development tools with their workstations. That was the beginning of the end for Sun, it just took fifteen years for it to catch up with them. The geek lust eventually moved to PC's running Linux. Linux went up and up and up, Sun lost market share to PC's running both Windows and Linux year after year, and to Apple once Apple moved to Unix and started including development tools to attract the geeks to their platform, until finally Sun got acquired for the few competitive products they had left. Sun made a last-ditch effort with OpenSolaris to regain geek cred, but it was too little, too late -- Linux had such a lock on geek mindshare that there was no way Sun could regain enough mindshare to make a difference. When geeks lust for Unix servers and workstations now, they lust for well-endowed Linux servers and MacOS workstations, not Sun ones -- but that was *not* true in 1990, in 1990 if you were a geek, you had serious Sun lust.

Of course, this points out two things about geek cred: a) it takes a long time to build up to the point where it makes or breaks technology companies (but in the end it always does), and b) once you lose it, getting it back is a real problem. Well, and c) if your product is pretty much the only thing that solves a particular problem, you don't need geek cred (thus why the IBM 370 architecture machines still sell plenty despite being utterly unfashionable in geek circles). So if you're a marketing type looking for the long term, what do you need to know?

1. Provide evaluation copies of your software without jumping through too many hoops. Basically, anything that involves delay is too many hoops for geeks, geeks are impatient and want instant gratification. At one employer they made it so difficult to download an evaluation copy of the software that virtually nobody did so -- the only way you could do it was to basically become a qualified lead and have one of their salesmen call you. But that annoys the geek. One reason why my employer uses VMware ESXi heavily is because we in the technical staff could download the evaluation version of the software, see that it worked quite well for our projects, and then, and only then, did we go to our superiors and have them negotiate whatever licensing was needed to make our projects work. If VMware had made it hard to get evaluation versions of their software, that would have never happened -- we would have used one of the Open Source virtualization platforms despite their limitations.
2. Describe what your product does on your web site. Seriously. I can't count the number of times I read some breathless but vague marketing hype on the web site, try to infer whether it solves the problem I'm trying to solve, download the actual eval product... and meh. It didn't do anything like what I thought it was when I read the breathless hype on the web site. Tell me what problem you're trying to solve with your product. Yes, I know you don't want to limit yourself, but please. Give me a clue *without* having to waste my time, because wasting time annoys the geek!
3. Provide technical documentation on your web site. At a previous employer I once wrote a blog entry for a customer-facing company blog that described, with some technical detail, exactly what it is that our product did. That blog entry got kiboshed by a PHB because "it lets out too many details of our secret sauce". Thing is, geeks don't like secrets. They want to know what it does and how it does it. If you don't tell them, they'll decide you don't have anything but smoke and mirrors and go elsewhere. It always frustrated me at that company that we had a cool product, but nobody knew it -- because we refused to tell it. But there's no such thing as secrets in technology. Everything can be reverse-engineered. So keeping technical details close to the vest doesn't accompany anything but annoying the geek -- and hurting your street cred, since it makes you look like a fly-by-night peddling smoke and mirrors.
4. Provide a programmable interface if reasonable and possible. End users won't care, of course. But one reason why Linux (and eventually MacOS X) won out over the other Unixes was because they were very open to programmers. Linux of course comes with the complete source code, while MacOS has source available for many components of the system plus has a free programming tool set available for it (XCode) as well as having that all-important Unix shell prompt with all the geeky Unix tools. Meanwhile, the other Unixes required you to fork over significant sums of money to get their programming toolkits. One of the things Microsoft has done right recently is to include PowerShell with all their systems and export most of their OS API's as PowerShell objects. It has decidedly helped their reputation in geek circles -- you'll notice that while you still have some geeks saying "Microsoft is Teh Eeeevil", most have moved to a position of neutrality on the whole subject of Microsoft. (Disclaimer: This is being written on Firefox running on Fedora 15 running under VirtualBox on a Windows 7 host :). I still wouldn't deploy a Windows server unless you paid me a significant sum of money to do so, but people who put Linux on the desktop because Microsoft is 'Teh Eeevil' are just being twits.
5. DON'T CRASH! Seriously. Crashes annoy geeks even more than missing features. If there's a choice between having a cool feature and having a reliable product, err on the side of a reliable product. Nobody ever lost customers by having a reliable product, but once you lose cred by getting a reputation for having a crashy product... well.
6. And -- participate in geek community. There were some tools at one employer that we could have contributed to the Open Source community that would have worked quite well at building geek cred, but management was totally against it because it would "divulge corporate secrets". Well, that company is out of business now. Openness sells. Secrecy doesn't. If you can't out-innovate your competition once they figure out your "secrets", you're in the wrong business -- because the technology industry moves so fast that any "secrets" you divulge *should* be obsolete long before any competitor can take advantage of them. If not, you're as doomed as the startup I worked for that took three years to create their first product, which was two years obsolete by the time they finally got it out the door late, slow, and buggy. You have to move *fast* in the technology industry, and if you can't out-run any "secrets" you divulge to the geek community, you're going to be out of business soon anyhow.

So anyhow, just one of those things to keep in mind if you're a marketing person wondering how to get market share. Find some geeks and ask them what annoys them about your product or about how your company sells your product. Then fix it. It won't create short term sales figures, but if you're wanting a long career for a stable company, that's how to do it -- something that Red Hat needs to remember, in their current rush to go complete pointy haired boss on the geeks when it comes to evaluating their enterprise products.

-ELG