Confessions of a Linux Penguin

Gnome 3, Mac OS, and the Geeks (Part 2)

2012-02-05T21:44:00.000-08:00

So in part two of this comparison, we're going to talk about workspaces.

Both Gnome 3 and Mac OS Lion have the concept of a linear ribbon of "workspaces" or virtual desktops. Gnome 3 lays out its workspaces vertically, while Mac OS Lion lays out its workspaces horizontally. So how do they compare on some routine workspace operations? Let's see...

Switching to Workspaces:

Gnome 3: There are two basic ways to do this: 1) Press the Windows key or swoosh mouse to left top of screen in order to activate the Activities screen. Move the mouse to the right side of the screen to make the workspace list pop out and select the workspace you wish to be in. OR: 2) Press CTRL-ALT-down to go to the next workspace down, or CTRL-ALT-up to go to the next workspace up. Mac: 1. Press F3 button on a recent Mac to go into Mission Control. You can also set a multi-touch gesture to do this (mine is three fingers up on the trackpad). Your workspaces will be listed horizontally at the top of the screen, move your mouse to and click on the one you want to go to. 2. Assign a multi-touch gesture to next-workspace and previous-workspace. Mine is three fingers left/right on the trackpad. 3. Assign a key sequence to next-workspace and previous-workspace. Mine is CTRL-left and CTRL-right. Because I have an Apple laptop, I can use multi-touch gestures to move left and right and to activate the Workspace switcher. This gives me one more option on the Mac. But reality is that navigating to a workspace is ridiculously easy on both systems, either from keyboard or via mouse/trackpad. Creating Workspaces:

Gnome 3: There is always one "blank" workspace at the end of the list of workspaces. If you move to that blank workspace and open a window there, a new blank workspace is created after it automatically, without you having to do anything.

Mac OS Lion: Press the F3 function key or use the multi-touch gesture to get to Mission Control. Move your mouse pointer to the right top of the screen. A new shadowed-out workspace will pop out of the ether. Click on it. You will now have a blank workspace to work in.

The Gnome approach can be done without touching the mouse, and actually requires no intervention on your part to create the new workspace -- it simply gets created when you need it. It Just Works, which is what a computer is supposed to be -- something that Just Works, without you having to do fiddly things to make it work. It will be interesting to see whether Apple or Microsoft copy this feature in their next release.

Deleting workspaces

Gnome 3: When you close the last window on a workspace, the workspace is deleted and you are then placed on the Activities screen, from whence you can select another workspace to work in using either CTRL-ALT-up/down or the workspace pop-out at the right of the screen. This prevents workspace clutter where you have lots of those automatically-created workspaces hanging around. (Note: I am aware of the "persistent workspace" plugin, I am comparing stock configurations).

Lion: Activate Mission Control. Move to the top left corner of the little workspace icon you want to zap. A little X will appear. Click on that X. The workspace will go bye-bye, and any windows on it will be moved to the first workspace.

Again the Gnome 3 approach to this appears to me to be much simpler than the Lion approach. There's no fine motor skill needed to move the mouse pointer to the exact point where the X will appear, you simply close your windows and poof, you're done. Nothing to remember, nothing to discover, it Just Works.

Moving Windows to a Workspace

For both Lion and Gnome 3 you simply move to the workspace containing the window you want to move, trigger the Activities or Mission Control screen, grab the window you want by clicking on it, and drop it on the icon of the workspace you want to move it to.

-------------------

So that's workspaces. As you can see, Gnome 3 is quite competitive with the state of the art that is Mac OS Lion. So that brings up the next question: Why do so many of the hard-core Linux geeks hate Gnome 3? I'll discuss that in the next installation of this series.

-ELG

Gnome 3, Mac OS Lion, and the geeks (Part 1)

2012-02-01T21:08:00.000-08:00

I've been using Gnome 3 on Fedora 15/16 and Mac OS Lion on a new Macbook Pro for several months now. What I see are two platforms that have done a significant re-thinking of the user interface to deal with some unpleasant facts:

The old paradigm of using menus to select programs has reached its expire-by date because the menus have attained a depth that nuclear submarines would love to attain
The population of most major technological countries is aging, and our old eyes simply cannot read the print on those tiny little pull-down menus anymore,
Fine motor control of old folks is pretty bad. We can manage to swoosh the cursor to the corner of the screen, or hit a big icon, but fiddly little menu items are hard for us to nudge a mouse into the right box to navigate a pull-down menu, and finally,
Pull-down menus simply aren't compatible with small touch-screens, because fingers are too fat to select tiny little things and you can't see them on a small touch screen like on a tablet anyhow.

In the process of dealing with these unpleasant facts, Mac OS Lion and Gnome Shell have also done a major re-thinking of how you do many common tasks in order to reduce the number of mouse movements / key strokes needed to do them. So let's look at some common tasks...

Program selection, Gnome 3:

Method a: Swoosh the mouse to the top left of the screen (one movement). Swoosh the mouse to the 'Applications' tab (one movement). Click left button. Select program from list of icons (possibly using the scroll wheel to scroll up and down the list).

Method b: Press the Windows key. Type the first couple of characters of the program you want to run. Use the arrow up-down to move the highlight to the icon of the program you want to run. Press ENTER.

Program selection, Mac OS Lion

Method A: Move the mouse to the icon of a rocketship on the toolbar. Click. Move mouse pointer to icon of program to run, possibly using left-right wobble wheel on your mouse or left-right two finger swipes on trackpad to go to next page of programs. Click on program.

Method B: Set a hot corner in preferences (I set bottom left), swipe mouse to there, move mouse to program to run, click.

Method C: Set a hot key in preferences such as control-opt-l, use left-right arrow keys to navigate pages. Unfortunately Mac OS has no way that I can find of using the keyboard to actually run one of those programs, you must actually navigate your mouse pointer to it and click it.

Verdict: If using a keyboard, Gnome 3 is very easy to navigate and uses a minimum of keystrokes to locate and run a program, requiring no mouse input at all. If using a mouse, the fact that you can get to the Applications icons immediately in MacOS, vs. having to click on the Applications tab after swooping to the corner, makes MacOS require one less mouse movement and one less mouse click. Score: TIE.

Select A Window, Gnome 3

Method 1: Ye olde alt-TAB, with a kick. For applications with multiple windows, you are shown a down-arrow and a down-cursor will then open window previews. Select the window you want.

Method 2: Hit the Windows key or swoop mouse to left top of screen. The windows will then swoosh out into a thumbnail pane view. Click on the window you're interested in.

Method 3: Hit the Windows key or swoop mouse to top left of screen. Select the dock icon representing the program you want to switch to. Either click it to go to the topmost window, or right-click it to select which window you want.

Select A Window, Mac OS Lion:

Method 1: Command-tab to get a list of running programs. While still holding down Command-tab, then use the arrow keys to move left-right to program whose window you want to see. Release command-tab while that program is highlighted. Note that unlike with Gnome 3, you do *not* get to choose which exact window of the program is going to be switched to -- you get whatever MacOS feels like giving you.

Method 2: Set a mouse button or gesture (I use the Page Forward button on my Logitech mouse or a triple-finger-up gesture on the trackpad). Invoke said mouse button or gesture. The windows will zoom out to pane/thumbnail view. Navigate mouse to the window you want and click on it.

Method 3: Move mouse to bottom of screen, click the dock icon of the program you want to switch to. Right-click will allow you to choose which of the windows to switch to.

Verdict: Gnome 3 wins on command-tab, its command-tab function is full-featured and works well. It ties on mouse button or gesture. It loses on dock, but only barely because its dock is not always visible and you have to move your mouse to the top right of the screen to show it, but that's only one mouse movement extra so not a huge loss. So Gnome 3 by a nose.

-----------------------------------------------------------------------

So what have we learned thus far? Well, 1) For two specific tasks, both Gnome 3 and Mac OS Lion have done a lot of work on reducing the amount of mouse movements and/or keystrokes needed to do these tasks, and almost completely eliminated any necessity for fine motor movements or reading of tiny print, and 2) Gnome 3 is quite competitive with Mac OS Lion in doing these tasks. In the next part of this series I will compare some other common operations, and finally I will summarize the results and examine one of the more bizarre things that has happened since the release of Gnome 3 -- the rabid condemnation of it by early Linux developers (including Linus Torvalds) who appear to despise it, and what that means for Linux.

Do Not Annoy The Geek

2011-07-28T20:54:00.000-07:00

What brought that up? Well, two things. The first is that I'm updating my report on virtualization systems with a new round of testing. This new round of testing includes some entrants that didn't exist during the first round -- Scientific Linux 6 / Centos 6.0, Red Hat Enterprise Linux 6.1, Ubuntu 11.04, and OpenSUSE 11.4. I'll write that up as soon as I finish with OpenSUSE 11.4, the only one left to evaluate. The second thing that happened was that my Macbook Pro decided it didn't like to sleep or shut down cleanly, which annoyed me because, reading forums, it appears that the only way to fix it is to do a clean wipe and re-install from scratch, re-install my applications from scratch, then restore only data, not configuration info. Makes me feel like a Windows user. Which annoys the geek.

So, how did the round of virtualization evaluations annoy the geek? Well: SL/Centos, Ubuntu, and OpenSUSE can be downloaded from their respective web sites. Red Hat requires you to sign up for an evaluation. Okay, so I already have an account on their site from the *last* version of their software that I evaluated back in December of last year (the original RHEL 6.0 release), so I went back in and signed up again. And promptly got rejected. "We do not accept personal email addresses for evaluations." I.e., they only want corporations to evaluate their software. I shrugged, changed my email address to a .com address that I registered over ten years ago but have never used (but which forwards email to my personal inbox), and downloaded the software. But I was annoyed.

So why is this important? Simple. I am not a product manager. I don't make final decisions about what to buy. But I am, more often than not, the person that the product manager or the IT manager comes to when they have a problem and want to know, "what technology do I need?". People like me are called influence leaders, because our combination of technical skills and communications skills, our understanding of both technology and actual business issues, means that folks who have one but not the other come to us to decide what the next round of innovation deployment is going to be. My estimate is that I've personally influenced or caused to happen over $5M/year of purchases over the past ten years. And that is true of geeks in general -- geeks may not be the people who sign the checks, but they're the people who evaluate the technology and tell the people who sign the checks what to buy.

So anyhow, back to Red Hat. That question on their web site is completely anti-geek. See, while evaluating virtualization technology is part of my job description (seriously -- it's there in black and white), for the most part I do it as a private citizen, not as a qualified lead. That's because otherwise I get bombarded with spam emails and phone calls asking me to buy stuff -- but I don't buy stuff, I evaluate stuff. Other people in my company buy stuff, and no, you are not getting their name or phone number from me because then they'll come to me and ask me questions about you that I can't answer until I finish evaluating your product. But Red Hat's marketing department is looking for qualified leads -- potential check-writers that they can bombard with phone calls and spam emails to sell stuff. But you won't get very far bombarding me with emails and phone calls, because I am a technical leader, I don't write checks and you'll do nothing but annoy me if you bombard me with phone calls and emails. If you want sales generated from me, you'll have to do it the old-fashioned way -- by letting me do a technical evaluation of your product for a month or so to see if it'll fulfil our company's needs better than the competition.

So how important is this sort of geek cred? Well, let's look at Sun Microsystems. Back in the day, they had geek cred out the yazoo. Geeks lusted after a Sun workstation on their desk, and fumed at the notion that they had to settle for a lowly PC, because Sun workstations with their BSD-based SunOS were geek nirvana with full geek programming toolkit including a "C" compiler and programmable shell scripting environment. Then Sun decided, upon releasing AT&T System V.4 as "Solaris", that they would no longer include the software development tools with their workstations. That was the beginning of the end for Sun, it just took fifteen years for it to catch up with them. The geek lust eventually moved to PC's running Linux. Linux went up and up and up, Sun lost market share to PC's running both Windows and Linux year after year, and to Apple once Apple moved to Unix and started including development tools to attract the geeks to their platform, until finally Sun got acquired for the few competitive products they had left. Sun made a last-ditch effort with OpenSolaris to regain geek cred, but it was too little, too late -- Linux had such a lock on geek mindshare that there was no way Sun could regain enough mindshare to make a difference. When geeks lust for Unix servers and workstations now, they lust for well-endowed Linux servers and MacOS workstations, not Sun ones -- but that was *not* true in 1990, in 1990 if you were a geek, you had serious Sun lust.

Of course, this points out two things about geek cred: a) it takes a long time to build up to the point where it makes or breaks technology companies (but in the end it always does), and b) once you lose it, getting it back is a real problem. Well, and c) if your product is pretty much the only thing that solves a particular problem, you don't need geek cred (thus why the IBM 370 architecture machines still sell plenty despite being utterly unfashionable in geek circles). So if you're a marketing type looking for the long term, what do you need to know?

Provide evaluation copies of your software without jumping through too many hoops. Basically, anything that involves delay is too many hoops for geeks, geeks are impatient and want instant gratification. At one employer they made it so difficult to download an evaluation copy of the software that virtually nobody did so -- the only way you could do it was to basically become a qualified lead and have one of their salesmen call you. But that annoys the geek. One reason why my employer uses VMware ESXi heavily is because we in the technical staff could download the evaluation version of the software, see that it worked quite well for our projects, and then, and only then, did we go to our superiors and have them negotiate whatever licensing was needed to make our projects work. If VMware had made it hard to get evaluation versions of their software, that would have never happened -- we would have used one of the Open Source virtualization platforms despite their limitations.
Describe what your product does on your web site. Seriously. I can't count the number of times I read some breathless but vague marketing hype on the web site, try to infer whether it solves the problem I'm trying to solve, download the actual eval product... and meh. It didn't do anything like what I thought it was when I read the breathless hype on the web site. Tell me what problem you're trying to solve with your product. Yes, I know you don't want to limit yourself, but please. Give me a clue *without* having to waste my time, because wasting time annoys the geek!
Provide technical documentation on your web site. At a previous employer I once wrote a blog entry for a customer-facing company blog that described, with some technical detail, exactly what it is that our product did. That blog entry got kiboshed by a PHB because "it lets out too many details of our secret sauce". Thing is, geeks don't like secrets. They want to know what it does and how it does it. If you don't tell them, they'll decide you don't have anything but smoke and mirrors and go elsewhere. It always frustrated me at that company that we had a cool product, but nobody knew it -- because we refused to tell it. But there's no such thing as secrets in technology. Everything can be reverse-engineered. So keeping technical details close to the vest doesn't accompany anything but annoying the geek -- and hurting your street cred, since it makes you look like a fly-by-night peddling smoke and mirrors.
Provide a programmable interface if reasonable and possible. End users won't care, of course. But one reason why Linux (and eventually MacOS X) won out over the other Unixes was because they were very open to programmers. Linux of course comes with the complete source code, while MacOS has source available for many components of the system plus has a free programming tool set available for it (XCode) as well as having that all-important Unix shell prompt with all the geeky Unix tools. Meanwhile, the other Unixes required you to fork over significant sums of money to get their programming toolkits. One of the things Microsoft has done right recently is to include PowerShell with all their systems and export most of their OS API's as PowerShell objects. It has decidedly helped their reputation in geek circles -- you'll notice that while you still have some geeks saying "Microsoft is Teh Eeeevil", most have moved to a position of neutrality on the whole subject of Microsoft. (Disclaimer: This is being written on Firefox running on Fedora 15 running under VirtualBox on a Windows 7 host :). I still wouldn't deploy a Windows server unless you paid me a significant sum of money to do so, but people who put Linux on the desktop because Microsoft is 'Teh Eeevil' are just being twits.
DON'T CRASH! Seriously. Crashes annoy geeks even more than missing features. If there's a choice between having a cool feature and having a reliable product, err on the side of a reliable product. Nobody ever lost customers by having a reliable product, but once you lose cred by getting a reputation for having a crashy product... well.
And -- participate in geek community. There were some tools at one employer that we could have contributed to the Open Source community that would have worked quite well at building geek cred, but management was totally against it because it would "divulge corporate secrets". Well, that company is out of business now. Openness sells. Secrecy doesn't. If you can't out-innovate your competition once they figure out your "secrets", you're in the wrong business -- because the technology industry moves so fast that any "secrets" you divulge *should* be obsolete long before any competitor can take advantage of them. If not, you're as doomed as the startup I worked for that took three years to create their first product, which was two years obsolete by the time they finally got it out the door late, slow, and buggy. You have to move *fast* in the technology industry, and if you can't out-run any "secrets" you divulge to the geek community, you're going to be out of business soon anyhow.

So anyhow, just one of those things to keep in mind if you're a marketing person wondering how to get market share. Find some geeks and ask them what annoys them about your product or about how your company sells your product. Then fix it. It won't create short term sales figures, but if you're wanting a long career for a stable company, that's how to do it -- something that Red Hat needs to remember, in their current rush to go complete pointy haired boss on the geeks when it comes to evaluating their enterprise products.

-ELG

Accessing raw drives from VirtualBox

2011-07-20T19:59:00.000-07:00

In my previous virtualization series, I avoided VirtualBox because there was no GUI support for accessing raw drives, and I have a pair of 2TB Linux RAID drives that I wanted to use a Linux VM on my Windows host to assemble and export to my network as a set of network shares. However, when I wanted to see the Gnome Shell functionality of Fedora 15, I had no choice but to use VirtualBox -- it's the only virtualization solution out there that currently supports OpenGL acceleration for Linux virtual machines.

So given that incentive, I actually did it. The secret is VirtualBox's VMDK support for importing VMWare virtual drives. The VMDK header format allows creating a virtual disk that is actually just a pointer to a physical drive. So, I used the directions on the VirtualBox site to create two vmdk files pointing at physical drives and then add them to my freshly installed Fedora 15 virtual machine as "existing" drives.

The first thing to note is that on Windows Vista or Windows 7, you MUST run VirtualBox as Administrator to access physical drives. Otherwise your VM simply won't start (and you can't even create your virtual VMDK's if you're not doing it as Administrator).

The next thing to note is where Oracle puts all the binaries you'll need. So you'll need to pop open a Terminal window as the administrative user (i.e. right-click and "run as Administrator") and:

C:> path "%path%;C:\Program Files\Oracle\VirtualBox"

Then you'll need to find your virtual machines. For me:

C:> cd "\Users\eric\VirtualBox VMs"

did the trick.

Now you can run the commands. I knew from my VMware Player experiment what my two physical drives were identified as in Windows. They were identified as drives 0 and 1, while my boot drive (which plugs into the front) is identified as drive 2. So I went ahead and did it. Otherwise you may need to do a bit of poking around to figure out which drives to push. So anyhow:

C:> VBoxManage internalcommands createrawvmdk -filename Fedora15/Disk0.vmdk -rawdisk \\.\PhysicalDrive0
C:> VBoxManage internalcommands createrawvmdk -filename Fedora15/Disk1.vmdk -rawdisk \\.\PhysicalDrive1

Then I right-clicked my VirtualBox icon and ran it as Administrator (you will probably want to go into its settings via right-click on its icon and make that permanent for left-double-clicks too), clicked the Fedora15 virtual machine, then its Settings icon, hit "+", and added the two hard drives as "existing" virtual hard drives. I then started the VM and... success! I saw my two drives in /proc/partitions.

Well.... *almost* success. Fedora 15 didn't activate my arrays on that first boot. So mdadm to activate them, vgscan (to detect my volume groups after activating the RAID arrays with mdadm), then lvchange -ay to activate the detected logical volumes. But once I did all that I could mount my filesystems and add them to /etc/fstab, and Fedora 15 properly assembled everything on my next reboot.

So what's the downside of VirtualBox so far? Well... it's hard to say. One thing I *have* noticed is that YouTube videos do not play properly. Multimedia playback in virtual machines is always problematic because of timing jitter, and I suspect that being on a Windows host with its really lousy clock system doesn't help. But then, I can just view multimedia on the Windows host -- that's why I have it, for games and stuff that doesn't work well under Linux. Other than that, everything else seems to be working... and you can't beat the price: Free (for personal use).

-ELG

Re-inventing the Linux desktop

2011-07-18T20:56:00.000-07:00

My opinion of the Linux desktop is pretty much well known by now -- I believe I mentioned "Window 95 as re-implemented by a Soviet Union that never fell"? A.k.a. clunky, incoherent, and obsolete? But recently two distributions have been released with a re-imagining of what the Linux desktop should look like. So today I installed Fedora 15 and Ubuntu 11.04 into Virtualbox on my Macbook Pro (into Virtualbox because it's currently the only virtualization environment with 3D support for Linux), and examined the two new systems -- Gnome 3's Gnome Shell, and Ubuntu's Harmony.

Gnome 3 was the first one I looked at, and by far the most revolutionary. Everything in the new Gnome Shell is set up to be doable with a couple of mouse swishes and one or two clicks. A swish to the upper left corner of the screen does three things -- does an Expose'-like scale of your windows so you can choose which window you want to activate by just clicking on it, sucks in a dock from the left side of the screen, and sucks in a screen list from the right side of the screen. You can grab a window and move it to the next screen in the screen list (there's always one blank screen at the end of the list -- sort of like the iPhone's applications screens), or you can click on the word "Applications" towards the upper left of your screen and see a list of applications in a format somewhat like an iPhone's application chooser. It is all easier to use than it sounds -- it really does make the whole thing swish swish swish point and click easy.

If you have applets running, like gDesklets, you can get to them by going to the bottom right corner of the screen. A sort of fuzzy menu bar then rises up from the bottom.

My general conclusion: Gnome 3 is currently incomplete -- it's barely configurable at all for example -- but it puts together the best ideas in UI's that have come down the pike over the past few years into an easy-to-use whole. My workflow falls out of the way Gnome 3 works naturally. If I want a screen for my browser windows, for example, swish to top left, swish to next screen on the right and click it, swish to left and select my browser on the dock, and voila, it pops open on the new screen and *another* blank screen is created for the *next* thing I want to do. Close all the windows on a screen, and it goes away, so I always have just the screens I need for my workflow -- no more, no less. I spent some time today doing software development with this system, and the usability compared to traditional Gnome is astounding.

Next up was Ubuntu's Unity. That, alas, turns out to be a disappointment. While Gnome 3 re-imagined the world to the point where some of the rumored Windows 8 functionality is going to be a clone of already-existing Gnome 3 functionality (like the iPhone-like application chooser), Unity simply attempts to clone Mac OS X without seeming too obvious about it. The problem is that simply moving the dock to the left side and modifying GTK+ to move application menus to the top like MacOS (except active and context sensitive, can't forget that!) isn't enough to make a quantum leap in functionality. Frankly, it ends up looking a bit of a mess. Gnome 3 re-imagined, Unity cloned, and like most clones, the clone isn't the equal of the original.

So: Does this mean the Linux desktop is usable now? Am I going to abandon my Macbook Pro and run a native Linux desktop again? Uhm... not hardly. I have a major investment in MacOS professional music software for which there is no Linux equivalent, and I still have significant difficulties viewing multimedia-based sites with Linux. Part of that is Microsoft and Apple's fault -- they release all these tools for "free" that produce (and view) multimedia content in their own proprietary formats like Quicktime and WMA, and don't release the viewing tools for Linux. I also can't read my corporate email using Linux -- neither of Evolution's Exchange plugins will handle proxied Exchange servers. That's sort of important too. Still, the fact that there is now a Linux UI which is actually innovative rather than a crude clone of other people's ideas is a sea change in an OS development process which all too often has ignored the desktop in favor of server optimizations. I don't know what happens next, but I suspect it'll be an improvement. Of course, given where the Linux UI started -- as an incoherent mess (inherited from MIT X11) -- that's sort of faint praise. So it goes.

-ELG

Virtualization solution #3: Windows Virtual PC

2011-03-24T20:49:00.000-07:00

Note that my Windows platform is Windows 7, which basically includes a free Windows XP virtual machine for Windows Virtual PC. But I was wanting to add physical hard drives. And... err... no. It won't do it.

My basic take: Windows Virtual PC works well if you're wanting some Windows "sandboxes" to play in. The drop-and-drag integration in particular is fairly impressive. But for what I want -- to create a virtual machine that's given ownership of a bunch of disks in order to software RAID-6 them and divvy slices of the resulting arrays out via iSCSI, CIFS, and AFP, i.e., to create a virtual storage appliance -- it simply lacks the basic functionality I need. Thus far my 64-bit Scientific Linux 6 is working quite well as a JBOD-manager with VMware Player... and there's not many virtualization solutions that can do that, and nothing other than VMware Player on the desktop.

-ELG

Sound, Flash, and VMware Player

2011-03-24T08:01:00.000-07:00

This is some notes on how to get sound and flash working on a Scientific Linux 64-bit guest ( rebranded Red Hat Enterprise Linux 6 ) running inside VMware Player on Windows 7 64-bit:

Flash: Note that Chrome for 64-bit Linux does *not* include the integrated Flash player that all other platforms have. You'll need to download 64-bit Flash beta from Adobe. At the time of this writing, that's at Adobe Labs. Once you extract the tgz file, you'll be left with a file "libflashplayer.so", simply copy it to /usr/lib64/mozilla/plugins , restart Chrome or Firefox, and you're set.

Sound: VMware's sound system crashes with the default Ubuntu / Red Hat sound configuration. This is apparently because VMware doesn't bother emulating all pieces of the hardware they say they're emulating, and when ALSA touches the missing bits, VMware disables the sound device. That's easy enough to fix though. From the Gnome menu, go to System->Preferences->Sound. In the Sound preferences, click on Hardware. Change the profile at the bottom from "Analog Stereo Duplex" to "Analog Stereo Output". Then on the little icon of the speaker on the bottom margin of the VMware Player, right-click it and select "Connect". After about a minute it'll turn green and you'll be able to play sound again.

Why this happens: Though the Ensoniq device emulated is capable of stereo duplex operation (i.e., both recording and outputting at the same time), I know because I actually had one of the physical cards back in the day and used it for that purpose along with a multitrack recorder program, VMware's emulation of the device is not capable of such and thus you must disable that capability. Unless you were intending to record audio within the Linux virtual machine (*not* recommended, VMware's timing is not sufficiently good to get good results there), this has no actual effect -- you can still play Flash videos from Chrome (and Firefox presumably) and hear the sound.

So if you're running Windows 7 on your physical hardware because you need the graphics performance for games and aren't willing to do the dual-card IOMMU hack with Xen that I demonstrated previously (or your hardware simply doesn't have IOMMU support, or you're not willing to use the latest bleeding-edge OpenSUSE as your platform), you can still have the far more secure web browsing environment of Linux available, and with VMware Player you can give Linux any additional drives beyond your boot drive for Linux to manage. Linux works far better as a server than Windows 7 does -- it can provide CIFS, AFP, iSCSI, and do it all on a much better software RAID stack than Microsoft's, as well as using LVM to manage that space, for which Windows 7 has no equivalent. And VMware's Unity system actually works pretty well with SL6/RHEL6 on Windows, although not as well as it works in VMware Fusion on MacOS (the issue being that the little Unity menu icon gets put into the screen menu bar on MacOS and has a native look and feel, while shows up as a clunky little usually-invisible icon above the Start menu on Windows). Which means that you can mix Linux Chrome windows, Windows IE windows (ick! But there's a couple of applications I need for customer support that requires IE plugins that don't exist for any other browser), Linux shell windows, and hoary old Outlook all on the same screen and manage them using the normal Aero Peek icons at the bottom or, if you have a Logitech mouse with the Logitech drivers installed, by assigning one of the side buttons to Window Switcher (their clone of Apple's Expose'). And unlike earlier versions of Windows, Windows 7 is stable -- in fact, I've never managed to make it crash. It's just very annoying... but that's why Apple is still in business, after all. Because Apple makes computers that are not annoying. For a price. A big price, alas...

-ELG

Virtualization solution #2: VirtualBox

2011-03-08T00:28:00.001-08:00

So the next piece of virtualization software that I was going to try out is VirtualBox. Remember, my Linux is installed on an entirely separate hard drive pair, that I mapped into VMware Player as two drives then installed Scientific Linux 6 onto one of the RAID arrays previously configured on that drive pair for that purpose (a 20GB RAID1 pair). 'grub' handles that situation just fine, it skips the RAID header and loads the Linux kernel and does its thing. At that point, I can see the remaining 1.8 terabytes of RAID'ed LVM volumes.

So I fire up VirtualBox and go to create a virtual machine via its GUI and... err... it doesn't allow me to assign physical drives to my VM. Which is one of those "WTF?!" moments, because the underlying QEMU that VirtualBox is based upon certainly has the *capability* to add physical drives into a virtual machine, but a bit of Googling around finds that you must do some cryptic command line hacking to make VirtualBox do it. The GUI won't do it.

At that point, realizing that VMware was point and click and ridiculously easy to set up and did what I wanted it to do without said hacking, I said "F*** that" and uninstalled VirtualBox. It may be that VirtualBox could perform better than VMware Player. But my time is more valuable than any tiny increment of performance that VirtualBox could potentially give me compared to VMware Player.

Next up: I check out Windows Virtual PC and see what I can do with it. For one thing, it'll be cool to try Windows XP Mode, even if it turns out not to be useful for virtualizing Linux...

-- ELG

Bizarre Windows behavior

2011-03-06T20:53:00.000-08:00

Some bizarre things I've seen with Windows 7 Ultimate thus far:

No loopback device. I had an ISO of Office 2010 on my network share, I copied it over to Windows 7 and... err... now what? I ended up exporting it to the Linux virtual machine as a virtual CD, mounting it in the Linux VM, then re-exporting it via Samba to Windows as a share.
Windows 7 won't browse my workgroup. No way, no how. I can go directly to the Run prompt and type in "\\linserver\office" and get my Office share, but it won't automagically populate its network neighborhood thingy in the left margin of Windows Explorer with my workgroup shares. It's not that Samba is misconfigured. My MacBook properly populates *its* left margin with the Samba server and expands to show the shares when I click on it, as well as showing the share I exported from Windows 7.f
Windows 7 won't do a thing when I click "Map Network Drive". Nothing. Nada. I remember that in the Xen installation of Windows 7 it certainly worked fine, it popped up a dialogue and all that, but now that it's native nothing -- zero -- happens when I click on that item.
Probably related to Homegroups. This is the only Windows 7 system on my network, so Homegroups is utterly useless. I attempt to change it and... nada. Nothing. Won't let me leave the homegroup.

Computers are supposed to be deterministic. Windows 7 isn't -- its behavior appears to be semi-random, operating upon a logic that Boole and Von Neumann would not have recognized. So have you had weird things happen with Windows 7? Curious penguins want to know!

-ELG

Windows virtualization software

2011-03-05T11:56:00.000-08:00

Previously I mentioned three bare metal hypervisor solutions -- Xen, KVM, and VMware ESXi -- and what was required to push through a video card to a WIndows VM living on those hypervisors. I have since managed to get that working with VMware ESXi, and I suspect if I install the very latest Fedora 14 host it'll work with KVM too.

That works fine with a desktop machine or rack server where I can mount multiple video cards into the box. The performance of Windows 7 virtualized is indistinguishable from Windows 7 raw. Here are my scores for Windows 7 raw on that box, with a Seagate Momentus XT 7200 RPM boot drive (due to the front-loading slot on my case that allows easy swapping of OS drives):

Processor: 7.5
Memory: 7.6
Graphics: 7.3
Gaming graphics: 7.3
Primary hard disk: 5.9

Compare with the virtualized numbers.

The problem, however, is when you want to go mobile. You simply can't add a second video card to a laptop. So if I want to play games on one of those new Dell Sandy Bridge desktop-replacement laptops, I need to run WIndows native, and come up with a way to have Linux also running and handling my years of accumulated data, all of which is in a Unix filesystem tree and cannot simply be copied into Windows. I preferably want this Linux to be running on a raw partition -- not on a file within a Windows partition -- and it has to be able to access raw Linux-formated USB and SATA drives. So, let's look at the first candidate... VMware Player

VMware Player is VMware's entry level desktop virtualization program. At one point in time VMware player would only start up virtual machines that had been created by VMware Workstation, but now it's an almost full version of VMware Workstation with various functionality like snapshots stripped out and with a limitation of only four cores allowed. VMware Player is "free" -- it's free for personal use, if you want to deploy it in a corporate environment you can license it for a fairly trivial per-seat fee (quite trivial, it'll be lost in the noise of your IT budget).

The test machine is my Xen server with Windows 7 installed as described above. I installed VMware Server on it without any problem. I created my Scientific Linux 6.0 virtual machine with no problem, giving it 2 gigabytes of memory and a 16gb root filesystem. VMware Tools installed easily into SL6 and allowed me to treat the Linux "X" desktop as if it were just any other Windows window, I could click into it, my mouse pointer could be moved outside of the window while I was typing into Linux program, and so forth. Adding the two 2TB SATA physical hard drives to the virtual machine was as simple as point and click, though the VM had to be off to do so because VMware Player's hot-plug functionality apparently does not work with physical drives. Once I booted SL6 up, it saw the two 2TB drives and assembled the Linux software RAID arrays on it automagically, though I had to do vgchange -a -y to get SL6 to recognize the LVM volumes on the RAID arrays.

So how fast is access to those two SATA drives? On a subsequent reboot of my Linux VM, a RAID check got fired off. The two drives were being read at 105MB/sec apiece -- 210MB/sec total -- and it used less than 20% of one of my eight cores for VMware to virtualize this. My take on it is that VMware Player's fake SCSI device takes a fair chunk of CPU to virtualize, but modern multi-core CPU's are so bleepin' fast that you won't even notice (which I didn't, until I went to see).

The final thing I wanted to do was to export a NTFS-formatted volume to Windows via iSCSI. Windows 7 has Microsoft's iSCSI initiator built in. I gave both my Windows machine and my Linux VM fixed addresses (using the bridged mode for the Linux VM's virtual network card), and installed the iSCSI target daemon and utils with 'yum install scsi-target-utils'. Then I added the already existing logical volume /dev/datagroup/win7 volume to /etc/volumes/targets.conf (see that file for exact format of what you need to add) and started up the daemon with "service tgtd start". Then I went to the Windows Administrative Tools (you can get to them from the the Start Menu if you've configured them to appear there, or from the Control Panel), selected the iSCSI Initiator, told it to scan the IP address of my Linux VM, and voila, it popped up there and as a drive letter in the Windows Finder. Easy peasy! The only thing to remember is to poke a hole for iSCSI in both the Linux and Windows firewalls, or it doesn't work :). (Yes, been there, done that, heh!).

The final test was to attach a USB hard drive to the system and export it to my MacBook Pro via iSCSI and use it as a Time Machine device. When I attached the hard drive to the system, it popped up in Windows, but clicking Virtual Machine ->Removable Devices showed the new drive, and allowed me to add it to the virtual machine. I then added it to targets.conf and told the tgd daemon about it, then went to my iSCSI initiator on the MacBook (the globalSAN iSCSI initiator) and added it, then used Disk Utility to format it as a Mac volume. Then I went to Time Machine and told Time Machine to use it for backup, and... voila. It started backing up at about 20mb/sec -- or about 50% of theoretical USB2 speed, not too bad considering this is being done via WiFi, not a direct connection, and the iSCSI target is running in a virtual machine, not directly on the hardware. A copy in Windows of a 4GB file to a similar USB drive ran at 26mb/sec, and that should go faster than Time Machine writes because it's a big sequential write, not a lot of smaller files. So now I have the equivalent of one of those expensive Time Capsule thingies, except that a 2TB Western Digital drive in an external USB case costs a *lot* less! Why an external USB? Simple -- so if I ever have to restore my MacBook Pro after a disk crash (which has happened before), I can unplug it from my big server and plug it directly into the MBP for MacOS to restore the system back to pre-crash state.

So... it's clear that VMware Player will do everything I want it to do here. There's two more options to look at before calling this competition done, however: Oracle's VirtualBox, which recently released a brand new version (version 4.0.4), and Microsoft's own Windows Virtual PC, which doesn't officially support Linux but which has been made to do so. More on those later...

-ELG

Surviving with Windows

2011-03-04T21:21:00.000-08:00

Even the most fervent Linux penguin may be required to use Windows for some purpose or another. For example, I have been doing a lot of work with VMware VSphere lately. And VSphere Client runs on... err... *WINDOWS*. Then there's games. Linux gaming is a non-starter due to the fact that the "X" Window System simply doesn't have the performance to do games. Mac gaming is better now that Steam has arrived, but Windows is still "the" PC gaming platform for most games.

So if you have to use Windows, what makes it less stressful for someone who prefers more elegant operating systems than the 16 years of hacks and kludges that is Windows today? I'll make a brief list here...

Microsoft Security Essentials. Don't even think of running Windows without an antivirus. This program is free, works as well as the others, and does *not* spam you or install spyware on your system like many of the other antivirus programs do.
Google Chrome. Internet Explorer on Windows 7 is a lot more secure than in days of yore, but Google Chrome is the most secure web browser for Windows, period.
Microsoft Office 2010, Standard version. It drives me batty with its incoherent user interface but Outlook still is the best here. However, if you do *not* need compatibility with Exchange Server and its calendar, install OpenOffice and Thunderbird instead -- their user interface is much better than Office 2010's.
Apple Quicktime. For playing Quicktime videos. Duh.
Foxit Reader for Windows. For reading PDF files, which is what everybody's documentation is sent out as nowadays. Do *NOT* install Adobe's own PDF reader, it is evil, it has a security breach every other day.
On the other hand, there's no alternative to Adobe's own Flash Player. Though thankfully Chrome doesn't need it, but IE will, so go ahead and install it -- using IE, because the installer will complain that Chrome already has Flash (duh).
GNU Emacs for Windows. Because sometimes you need to edit text files, and Emacs is the best way to do that.
Xmarks. The best way to keep all your bookmarks in sync between Chrome, IE, and any other computers you have.
Evernote, to write things like shopping lists and todo lists and keep your notes in sync with your smartphone, your tablet, and the cloud.
Steam client. Of course :).

To be continued...

Note one of the things I did *not* mention: Cygwin. The reality is that it will constantly annoy you due to the fact that the Unix API doesn't map well onto the Windows API. You're using Windows, not Unix. You're much better off learning how to use the native Windows tools like Powershell. They will annoy you too, but not as much as trying to shove a square peg into a round hole.

-ELG

Disk encryption and LUKS

2011-03-02T07:08:00.000-08:00

One of the interesting features of recent Linux distributions such as Ubuntu is LUKS, which allows you to encrypt (most of) your data on the hard drive. So the question I was asked is: How secure is LUKS?

Let's look at the Ubuntu implementation. It uses AES-256 to encrypt the disk volume using an appropriate cipher feedback mode to deal with frequency attacks and other such attacks against statically encrypted data. First, is the cipher secure? The answer there is, yes. AES (Rijndael) has been subjected to extensive cryptanalysis both as part of the AES cipher competition and afterwards, and there are no known compromises against full AES. The reality is that even AES-128 would be sufficient for this purpose, because the weakness of the Ubuntu implementation lies elsewhere: the passphrase. A NSA employee once stated in public that they didn't care anymore about how well encryption algorithms worked, because there was an infinity of methods to obtain the passphrases of anybody they really wanted to spy on.

Remember, a cipher is *not* a cryoptosystem. A cryptosystem consists of both the cipher, and the software required to feed it keys and data. In this case, the biggest weakness is in the keystore (stored in the header of the volume) and the method of securing it. The keystore is secured by a passphrase. Brute force dictionary attacks against the passphrase to decrypt the keystore might work, but usually won't if you chose a sufficiently complex passphrase incorporating non-word "words". The most likely attack here is either a passphrase sniffer injected into the system or a phishing exploit where a prompt is put up for the passphrase, but it is not by the login process, it is by software previously injected into the system via other means (typically a network-based exploit that then uses a privilege elevation exploit to gain permission to insert itself into the system boot sequence at the correct place).

Or, black bag cryptanalysis could be used if done by a determined organization -- a hardware keystroke scanner placed in your system, a pin mike pointed at your keyboard that then can be used to determine what keys were pressed based upon sound (each key has a subtly different sound when pressed, and which keys were pressed can be determined via frequency analysis for your language and assigned to the appropriate sound given sufficient keystroke sounds recorded), a pin camera can be mounted somewhere aimed at your keyboard to record your key presses. If you cannot guarantee that your system and computing environment has been physically secured, no cryptosystem is going to be sufficient. Thus the various exploits against ATM machines, where the machines are physically compromised (via scanner equipment physically added to them) to gain access to ATM card codes and PIN numbers for later use by thieves.

The Ubuntu setup, in other words, is sufficient for dealing with the issue of casual theft of computer equipment by random thieves -- they will not have previously captured passphrases thus will be reduced to brute force dictionary attacks upon the keystore encryption, which should fail as long as you choose a sufficiently complex passphrase -- but is insufficient to deal with a determined attack by someone who is willing to go to the trouble of rootkit'ing or black-bagging you. In particular, it is useless for dealing with network-based exploits, which occur after the passphrase has already been entered and can extract your data via the network accordingly. So it is useful, but adjust your expectations -- it is not going to stop a determined attacker (as vs. a casual theft of your drives or computer).

So, could this setup be made more secure against even passphrase interception attacks? Yes and no. You would need to load the Linux kernel and all pre-boot software from secure read-only media and also have the keystore reside there, then physically secure this media in some location other than with your physical hardware. A dongle in your laptop bag, for example, is not sufficient. You would need to have this media physically secure in your presence at all times to avoid having it compromised by black bag attacks, and have some method of physically destroying it if impending rubber hose cryptography seems likely and the data in question is so sensitive that the repercussions of it being decrypted is dire. But even there, you're still susceptible to ordinary network-based attacks that trick you into installing malware onto your drive or which exploit holes in your network software and which then steal your data via the network.

The reality is that the only computer that is completely and utterly secure is one located in a vault with no network access. Unfortunately, said computer is also not very useful for our day-to-day computing needs. The LUKS setup sufficiently deals with the problem of casual theft of computer equipment but will not stop a determined attacker with the resources of a major criminal syndicate or government behind it. It is a reasonable compromise between security and usability. So adjust your expectations accordingly.

-ELG

User interfaces and the Office 2010 problem

2011-02-18T21:24:00.000-08:00

Microsoft has some excellent user interface guidelines. The committee that wrote these guidelines made many of them unnecessarily complex and verbose, and I disagree with quite a few of them because I feel programmers will abuse them to do things that shouldn't be done in user interfaces, but by and large, I believe that if you keep these guidelines in mind, you will have a better product.

How, then, do we explain the mess that is Microsoft Office 2010? I can't. Take the mess that is Outlook, for example. Okay, so you want to add an email account. What's the first thing that Outlook tells you to do? Err.... EXIT OUTLOOK! And that makes sense because... err... why?

Yes, I know someone's going to pipe in with technical reasons for why Microsoft does this. But my point is, the user doesn't care. If Microsoft wants to keep a separate database with email accounts so that any email client they support (Outlook, Windows Live, or any future client) can access it, fine and dandy. But don't require people to run another program just to set up their email accounts when they're already in their email program.

Then there is the horrifically slow speed of Outlook 2010. I'm running Office 2010 on a 3.07ghz quad-core Intel Core I7 Extreme 950, which is not a slow processor -- indeed, it's one of the fastest consumer processors on the market. My Windows Experience indexes are near the top for everything except disk i/o, where it's constrained by the mediocre limits of a 7200 rpm SATA drive. I also have 12 gigabytes of RAM. But Outlook 2010 is always lagging. I have my Macbook Core I7, a dual-processor Core I7 at 2.6ghz, also. It has a 5400 rpm SATA drive. The Macbook is running Apple's Mail with the exact same accounts set up as the Windows machine. I hit the "sync" button on Outlook. I hit the "sync" button on Mail. And the winner is... the slower computer. By a landslide. Yes, Apple Mail on a MacBook blows Outlook 2010 out of the water in terms of responsiveness, displaying the count of new messages one by one in its left-side inbox tray long before Outlook 2010 finishes its clumsy sync cycle and starts deciding to display the counts.

Here's a clue: responsiveness does *not* necessarily require faster algorithms. Mail isn't doing anything extra-special compared to Outlook. Mail simply does more in parallel and returns results to the user as they come in, rather than batching them up. Responsiveness matters. Caching values so that you don't have to fetch them every time from the server, displaying results as they come in, and otherwise making it look fast is, in many cases, as good as being fast. I'm not privy to the innards of Outlook, but it appears to me that Outlook performs its sync cycle, tediously gathering new emails from each server, and then -- and only then -- decides to update counts and headers in the email window. I may be wrong about that, but I shouldn't even be guessing in the first place, because, like with Apple Mail, it should just happen.

And finally, let's look at Word 2010. Please. Word 2010 goes full-ribbon. Microsoft's new "ribbon interface" concept is a tabbed icon tray. It suffers from the same problem as Apple's Dock -- icons simply are not descriptive. They're so much hieroglyphics to me. Thing is, Apple's Dock is just one row of icons, and even that is confusing enough that I sometimes hit Preferences when I was aiming for System Performance. Word 2010, on the other hand, is tabbed row after tabbed row of icons. The end result is that it's utterly unusable to me because I can't find anything, any functionality I want is hidden in row after row of hieroglyphics. I instead go into OpenOffice and create my documents there. Way to drive people to your competition, Microsoft!

There's two points here:

Icons do *not* replace text. Don't even think about it. Icons are a *supplement* to text.
Microsoft doesn't follow their own user interface guidelines, adding gratuitous complexity for the sake of complexity, hiding functionality in row after row of "ribbon" icons to the point of rendering their product basically unusable to anybody who doesn't live in it. Look at page 19 of their user interface document (linked to above). Look at Office 2010. Sigh and think of what could have been, if Microsoft had only followed their own guidelines.

Note that Windows 7 itself, while not an outstanding user interface, is understandable and usable. I might laugh at how you must use the search box to find anything in the Control Panel because there's so many of the freakin' things, but there is a logic to it that is easy to understand. Office 2010, on the other hand, has a logic to it... but one that reminds me of the story I presented here a few months back, about the physicist who made a logical interface for our product -- but an interface that only made sense if you were a physicist. Or as he exclaimed when we told him that our users wouldn't understand his UI, "then your users are idiots!" Why... yes! And they're idiots with *money*, who want to get a job done, and who are willing to *pay* us if we give them something that they can use to get the job done that doesn't require them to be nuclear physicists to use it!

Sadly, Microsoft's been stuck in their own hermetic world for so long with no strong leadership from the top to force all the various divisions to comply with things like, say, user interface standards, that there is just no consistency to their product line. Even Windows 7, probably the best Microsoft user interface since Windows 95 introduced their "new" user interface to the world, suffers from this syndrome a bit -- the various vintages of programs in their control panel, for example, have user interfaces that are all over the place. It's a shame, really, because Microsoft has the technology to do it right, and even the people like those who participated in writing their user interface design document. Just not the leadership.

I guess Microsoft chairman Uncle Fester figures that as long as the majority of people need to use his workstation OS because it's the standard, he really just needs to sit back and rake in the money with occasional gratuitously incompatible upgrades to force people to buy replacements for their old stuff. And he may be right. But unless your company is Microsoft, it behooves you to do as Microsoft says -- not as they do. Because your competition is going to come out with a clean, simple, easy to understand user interface for their product... and if your product looks like a mess, if it exposes technical details that customers don't want to know about rather than Just Working, well.

-ELG

Denial is more than a river in Egypt

2011-01-26T20:34:00.000-08:00

A Linux fanboy asks, "Why do so few people use Linux on the desktop? After all, it's superior to Windows."

My response is: Are you joking? Surely you're joking, right?

First of all, people don't use operating systems. They use applications. And most end-user applications run on Windows. If, for example, I want to manage a ESXi server, I have to use vSphere Client to do that. And vSphere Client runs on... err... Windows. As does pretty much every other specialty application on the planet that people use, or even many non-specialty ones -- try viewing WMC or QuickTime videos on Linux. You can't do it. They simply don't work. You can (illegally) hack your Linux system to do this by copying components from Windows, but really, how many end users are going to do this? And they certainly aren't going to do it in a corporate environment, where systems are locked down to prevent end users from installing illegal software.

Secondly, as I've repeatedly pointed out, the Linux desktop environment is a mess. It's like if the Soviet Union had not fallen in the early 90's, had seen Windows 95, and decided to create a Soviet version of Windows 95. It's clunky, creaky, overly complex, makes little sense from an end user perspective, and things that are easy on the Mac or Windows are ridiculously difficult to do with the Linux desktop. For example, to assign one of the side mouse buttons to the window switcher on Windows or Mac is a simple tool in the Mac control panel or Logitech mouse manager on Windows. To do so on Linux, on the other hand, is an adventure.

So anyhow, what inspired this rant? Well, simple: I was annoyed at the slow speed of the QEMU console used by KVM and Xen. This appears to be an issue with QEMU's console driver, it does the same thing with both KVM on Fedora and Xen on OpenSUSE. QEMU's console driver screen-scrapes video memory, then stuffs it into a VNC session, but does this so ridiculously slow as to render it basically unusable. I know it's possible to do virtualized console drivers that operate quickly -- VMware does it *over a network*, for cryin' out loud, install ESXi on one of your spare systems and point vSphere Client at it from a Windows system on your network if you disbelieve me -- but apparently nobody in the Xen or KVM communities cares, since this has been a problem for literally years. I guess it's because Xen and KVM are typically used to virtualize things like web and email servers, where nobody cares about how fast the console is.

The workaround is to a) use ssh if you need CLI access to the system, and b) spawn off a vnc session if you need GUI access to the system. For example, for my Fedora guest, I have this line in rc.local:

su -c "vncserver -geometry 1440x900 -depth 16" egreen > ~egreen/vnc-log 2>&1 &

From there on, I access it via a VNC viewer from my Macbook Pro or from the Linux host system.

[flame on]
This problem has been there since the beginning of the QEMU project, but nobody cares to fix it because, well, it's "good enough" to get in and start vncserver, so what's the problem? This disregard for end user experience is the #1 reason why Linux is a sad also-ran in the desktop operation system competition. Even netbooks have switched away from Linux to Windows, because the end-user experience with Linux is so pathetic that even Windows -- sad pathetic WINDOWS -- does it better. It's the whole XKCD 619 problem. It's not because Linux doesn't have the technical capability to have a decent end-user experience, it's because nobody *cares* because what exists is good enough for geeks and if you point out that end users don't like it, you get oodles of pushback from the Linux fanboys about how you don't *really* need feature X because there is workaround Y. Denial is more than a river in Egypt, folks. I've been using Linux for 15 years. My name is in the Linux source code. I've written at least half a million lines of userland code for Linux in the past 15 years and while my kernel contributions are minor driver contributions, they're there. And my desktop is a Mac because I find the Linux desktop environments to be so clumsy, clunky, and annoying. Q.E.D.
[/flame off]

So anyhow, that's my gripe of the day. I'm sure I'd get some pushback from Linux fanboys on it if they bothered reading it, which of course they won't, because they're too busy making sure Linux runs well on 4096-core processors. All I'll point out is that refusing to admit you have a problem is a guarantee that the problem will continue. The Linux community is like a drunk that refuses to admit he has a drinking problem. Linux has a user interface problem, people -- and like the drunk who won't go to rehab because he refuses to admit he has a problem, Linux's user interface problem is not going to get fixed as long as Linux geeks continue to insist they have no problem.

-- ELG

Pushing a graphics card into a VM, part 5

2010-11-15T21:47:00.000-08:00

Part 1 Part 2 Part 3 Part 4 Part 5

So here's the final thumbnail summary:

Hardware:

Video card #1: ATI 5750 (but the 5770 should work too and is slightly faster, but the 5750 was on the Xen compatibility list).
Video card #2: nVidia Corporation VGA G98 [GeForce 8400 GS] *PCI* card (BIOS set to use PCI as first card)
Intel(r) Desktop Board DX58SO -- not a great motherboard, but it was available at Fry's and was on the Xen VT-d compatibility list
Intel Core I7-950 processor
12GB of Crucial 3x4GB DDR3 RAM
Hard drives: 2 Hitachi 7200 rpm SATA 2GB drives, configured as RAID1 via Linux software RAID.
Antec Two Hundred V2 gamer case to handle swapping OS's via the front 2 1/2" drive port.
Various 2 1/2" drives to hold the Linux OS's that I was experimenting with

Software:

OpenSUSE 11.3, *stock*.
Windows 7, *stock*.

By adding the PCI card, my Linux console remains my Linux console, and Xen properly starts up my Windows DomU. My configuration is now complete. I may extend my Windows LVM volume to 200G so I can install more games on it, but note that all of my personal files, ISO's, etc. live on Linux. Note that 5.9 is what the Windows Performance Index should be for that particular hard drive combo, so this Windows system is as good as most mid-range gaming systems performance-wise. I added the paravirtualization drivers for the networking and disk controller, but they didn't improve performance any -- all they did was reduce how much CPU the dom0 qemu was expending implementing virtual disk and network controllers. Given that I have a surplus of CPU on this system (8 threads, 3.2ghz), it's in retrospect no surprise that I saw no performance gain on the disk and network from going paravirtual -- all I did was free up more CPU for use for things like, say, video encoding.

Thoughts and conclusions:

One thing that was very clear through this entire process is that I'm very much pushing beyond the state of the art here. The software and hardware configurations needed for this to work were very twiddly -- there is exactly one (1) Linux distribution (OpenSuse 11.3) which will do it at this point in time, and there were no GUI tools for OpenSuse 11.3 which would create a Xen virtual machine with the proper PCI devices. Furthermore, the experimental Xen 4.01 software on OpenSuse is almost entirely undocumented -- or, rather, it has man pages provided with it, but the man pages document an earlier version of Xen which is significantly different from what's actually shipped with OpenSuse 11.3.

From a general virtualization perspective, comparing Xen, KVM, and ESXi, Xen currently wins on capabilities but only by a hair, and those capabilities are almost totally undocumented -- or worse yet, don't work the way the documentation says they work. Xen's only fundamental technological advantage over KVM and ESXi right now is its ability to run paravirtualized Linux distributions without needing the VT-x and VT-d extensions -- a capability which is important for ISP's with tens of thousands of older servers without these extensions, but becoming increasingly less important as VT-x is now everywhere except in the low-end Atom processors. Comparing my Xen installation at home with my KVM installation at work, both of which I have now used extensively and pushed their capabilities to their limits, I can see why Red Hat is pushing the KVM merger of hypervisor and operating system -- KVM gives you significantly greater ability to monitor the overall performance of your system, vs. Xen where 'xm top' is a poor substitute for being able to get detailed monitoring of your overall system performance, is significantly better at resource management since the same resource manager handles everything (core hypervisor/dom0 plus VM's), and the Linux scheduler can consider everything when deciding what to schedule, rather than having the Xen hypervisor out in the background making decisions about which Xen domain to schedule next based upon very little information.

In short, my general conclusion is that KVM is the future of Linux virtualization. Unfortunately my experience with both KVM and Xen 4.0 is that both are somewhat immature compared to VMware's ESX and ESXi products. They are difficult to manage, their documentation is persistently out of date and often incorrect, and both have a bad tendency to crash cryptically when doing things that they're supposed to be able to do. Their core functionality works well -- I've been running Internet services on Xen domains for over five years now and for that problem domain it is bullet-proof, while at work I am developing for several different variants of Linux using KVM virtual machines on Fedora 14 as well as running a Windows VM to handle the VSphere management tools, and it's been bullet-proof. But they decidedly are not as polished as VMware at this point, other than Citrix's XenServer, which lacks the PCI passthrough capability of ESXi and thus was not useful for the projects I was considering.

My take on this, however, is that VMware's time as head of the virtualization pack is going to be short. There isn't much more that they can add to their platform that the KVM and Xen people aren't already working on. Indeed, the graphics passthrough capability of Xen is already beyond where VMware is. At some point VMware is going to find themselves in the same position vs. open source virtualization that SGI and Sun found themselves in vs. open source POSIX. You'll note that SGI and Sun are no longer in business...

-ELG

Pushing a graphics card into a VM, part 4

2010-11-14T18:41:00.000-08:00

Part 1 Part 2 Part 3 Part 4 Part 5

Okay, so virt-manager did pick up my new VM once I created it with xm create on a config file, but when I rebooted the system the VM was gone. So how can I fix this? Well, by taking advantage of functionality that OpenSUSE has had for auto-starting Xen virtual machines all along: Just move my config file into /etc/xen/auto and it'll auto start (and auto shutdown, if I have the xen tools installed) at system boot.

Of course, that requires a config file. Rather than paste it here, I'll let you view the config file as a text file. Note that 'gfx_passthru=1' is commented out. The Xen documentation says I need it, but if I put it there, my VM doesn't start up -- it crashes into the QEMU monitor. Also I ran into another issue, a timing issue. pciback is grabbing the console away from Linux and leaving the video card half-initialized, and when Xen grabs the video card and shoves it into the VM, the video card locks up the system solid when Windows tries to write to it. My solution to that was even simpler -- put the older of the nVidia cards back into the system, and load the 'nouveau' driver using YaST's System > Kernel > INITRD_MODULES and System > Kernel > MODULES_LOADED_ON_BOOT functionality. This flips the console away from the ATI card early enough that it doesn't conflict with Xen giving the video card to Windows. This also gives me a Linux console on the nVidia card that I can switch to by plugging in a second keyboard to the front USB on my chassis (the one I did *not* push into the Windows VM) and flipping my monitor to its DVI input (rather than the HDMI coming from the ATI card).

With all of this done, I can now reboot my system and get Windows on video card 0, and Linux on video card 1. I suppose I could reverse the video cards (to give the boot video card to Linux), unfortunately my board puts the second 16-lane PCIe slot too close to the bottom of the case and a double-width PCIe card won't fit there. Maybe when I upgrade to one of those spiffy SuperMicro server motherboards with the IMPI and such, at which point I won't need a second video card anyhow because the on-board video will suffice for Linux...

Next up in Part 5: Thoughts and conclusions.

Pushing a graphics card into a VM, Part 3

2010-11-13T23:16:00.000-08:00

Part 1 Part 2 Part 3 Part 4 Part 5

OpenSUSE 11.3 was a quite easy install. I haven't used SUSE since the early 'oughts, but first impressions were pretty good. OpenSUSE 11.3 is KDE-based, which is a change from the other distributions I've been using for the past few years -- Ubuntu on my server at home, Debian on my web and email server, and various Red Hat derivations at work -- and seems to be pretty well put together. The latest incarnation of YAST makes it more easily managable from the command line over a slow network connection than the latest Ubuntu or Red Hat, which rely on GUI tools at the desktop. The biggest difference between Red Hat and SUSE was that SUSE uses a different package dependency manager, "zypper", which is roughly equivalent to Red Yat's "yast" and Debian's "apt-get" but with its own quirks. It appears to be slightly faster than "yast" and roughly the same speed as "apt-get". If you wonder why SUSE/Novell wrote "zypper", at the time they wrote it, "yast" was excruciatingly slow and utterly unusable unless you had the patience of Job. Red Hat has sped up "yast" significantly since that time, but SUSE has stuck with "zypper" nevertheless. I also set up the bridging VLAN configuration that I mention in my previous post about how to do it on Fedora. Again SUSE has slightly different syntax than Red Hat for how to do this in /etc/sysconfig/network/* (note *not* network-scripts), but again it was fairly easy to figure out via reading the ifup / ifdown scripts and consulting SUSE's documentation.

So anyhow, I installed the virtualization environment via "yast" and rebooted into the Xen kernel downloaded by that. At that point I created a "win7" LVM volume in LVM volume group "virtgroup" on my 2TB RAID array, and went into the Red Hat "virt-manager" and attached to my Xen domain, then told it to use that LVM volume as the "C" drive and installed Windows 7 on it. I'm using a LVM volume because at work with KVM, I find that this gives significantly better disk i/o performance in my virtual machine than pointing the virtual disk drive at file on a filesystem. Since both Xen and KVM use Qemu to provide the virtual disk drive to the VM, I figured that the same issue would apply to Xen, and adopted the same solution that I adopted at work -- just point it at a LVM volume, already. (More on that later, maybe).

Okay, so now I have Windows 7-64 bit installed and running, so I shut it down and went to attach PCI devices to it via virt-manager and... err. No. Virt-manager wouldn't do it. Red Hat strikes again, it claims that Xen can't do PCI passthrough! So I went back to the handy Xen Wiki and started figuring out via trial and error how to use the "xm" command line, where the 'man' page for xm doesn't in any way reflect the actual function of the program that you see when you type 'xm help'. So here we go...

First, claw back the physical devices you're going to use via booting with them attached to pciback. So my module line for the xen.gz kernel in /boot/grub/menu.lst looks like...

module /vmlinuz-2.6.34.7-0.5-xen root=/dev/disk/by-id/ata-WDC_WD5000BEVT-22ZAT0_WD-WXN109SE2104-part2 resume=/dev/datagroup/swapvol splash=silent showopts pciback.hide=(02:00.0)(02:00.1)(00:1a.0)(00:1a.1)(00:1a.2)(00:1a.7)(00:1b.0)

Note that while XenSource has renamed pciback to 'xen-pciback', OpenSUSE renames it back to 'pciback' for backward compatibility with older versions of Xen. So anyhow, on my system, this hides the ATI card and its sound card component, and the USB controller to which the mouse and keyboard are attached. I leave the other USB controller attached to Linux. I did not have any luck pushing USB devices directly to the VM, I had to push the entire controller instead, apparently the Xen version of QEMU shipped with OpenSUSE 11.3 doesn't implement the USB (or else I simply need to read the source). Note that you want to make sure your system boots *without* the pciback.hide before you boot *with* it, because once the kernel starts booting and sees those lines, your keyboard, mouse, and video go away!

Okay, so now I'm booted. I ssh into the system via the network port (err, make sure that's set up before you boot with the pciback.hide too!) and go into virt-manager (via X11 displaying back to my Macbook, again make sure you have some way of displaying X11 remotely before you start this) and start up the VM. At that point I can do:

xm list

and see my domain running, as well as log into Windows via virt-manager. So next, I attach my devices...

xm pci-attach win7 0000:02:00.0
xm pci-attach win7 0000:02:00.1
xm pci-attach win7 0000:00:1a.0
xm pci-attach win7 0000:00:1a.1
xm pci-attach win7 0000:00:1a.2
xm pci-attach win7 0000:00:1a.7
xm pci-attach win7 0000:00:1b.0

Windows detects the devices, loads drivers, and prompts me to reboot to activate. So I tell Windows to reboot, and it comes back up, but nothing's showing up on my real (as vs. virtual) video screen. then I go into device-manager in Windows and see what happened. The two USB devices (keyboard and mouse) show up just fine. But the ATI video card shows up with an error. I look at what Windows tells me about the video card, and Windows tells me that there is a resource conflict with another video card -- the virtual video card provided by QEMU. So I disable the QEMU video card, reboot and... SUCCESS! I now have Windows 7 on my main console with video and keyboard and mouse!

Windows Experience reports:

Calculations per second: 7.6
Memory: 7.8
Graphics: 7.2
Gaming graphics: 7.2
Primary hard disk: 5.9

Those are pretty good, quite sufficient for gaming, except for the disk performance which is mediocre because we're going through the QEMU-emulated hard drive adapter rather than a paravirtualized adapter. When doing network I/O to download Civilization V via Steam I also notice mediocre performance (and high CPU utilization on the dom0 host) for the same reason. We'll fix that later. But for playing games, we're set! Civilization V looks great on a modern videocard on a 1080P monitor with a fast CPU!

Okay, so now I have a one-off boot, but I want this to come up into Windows every time my server boots. I don't want to have to muck around with a remote shell and such every time I want to play Windows games on my vastly over-powered Linux server (let's face it, a Core I7-950 with 12GB of memory is somewhat undertasked pushing out AFS shares to a couple of laptops). And that, friends, is where part 4 comes in. But we'll talk about that tomorrow.

-ELG

Pushing a video card into a VM, Part 2

2010-11-13T22:51:00.000-08:00

Part 1 Part 2 Part 3 Part 4 Part 5

The first issue I ran into was that my hardware was inadequate to the task. My old Core-2 Duo setup lacked VT-d support. So I went to the Xen compatibility list and found a motherboard which supported VT-d and upgraded my motherboard. At the same time I also upgraded my case to an Antec case that has a slot on the front for plugging in 2 1/2 inch drives. This was to make it easier to swap operating systems. Theoretically you can hot-swap, but I've not tested that and don't plan to.

Since I am inherently a lazy penguin (much like Larry Wall), the next thing I did was try the "virtualization environments". I found that XenServer was a very well designed environment for virtualizing systems in the cloud. Unfortunately it was also running a release 3.x version of Xen rather than the new 4.0 release, and did not implement PCI passthrough or USB passthrough to fully virtualized VM's natively. There were hacks you could do, but once you start doing hacks, the XenServer environment is not really a nice place to do them. So I moved on.

ProxMox VE is a somewhat oversimplified front-end to KVM and OpenVZ. It looks like a nice environment for running a web farm via web browser, but unfortunately it does not support PCI passthrough natively either. Again, you can start hacking on it, but again once you start doing that you might as well go to a non-dedicated environment.

Ubuntu 10.10 with KVM was my next bet. I *almost* got it running, but the VM wouldn't attach the graphics card. It turns out that was another issue altogether, but looking at the versions of QEMU and KVM provided, it appeared that Fedora 14 had one version newer (as you'd expect, since Fedora 14 came out almost a month later), so I went to Fedora 14 instead.

I got close -- really close -- with Fedora 14. But two different video cards -- an old nVidia 7800GT and a new nVidia GTS450 -- both ended up with error messages in the libvirtd logs saying there was an interrupt conflict that prevented attaching the PCI device. I ranted to a co-worker, "I thought MSI was supposed to solve that!" So I looked at enabling MSI on these nVidia cards and found out that... err... no. Not a good idea, even if I wanted, the cards generally crashed things hard if you tried. So I went back to the XenSource.com wiki on VGA passthrough again and followed the link to the list of video cards, and... err, okay. an ATI Radeon 5750 has been reported as running wiith Xen's VGA passthrough.

So, I swapped that out, and tried again with Fedora 14. This time the KVM module crashed with a kernel oops.

At this point I'm thinking, otay, KVM doesn't seem to want to do this. Xen, on the other hand, seems to have a Wiki and all documenting how to do this. So let's use Xen instead of KVM. The problem is that Xen is an operating system. It relies on having a special paravirtualized kernel for its "Dom0" that handles the actual I/O driver work. Red Hat claims providing such a kernel would be too much work and that they won't do it until the Dom0 patches are rolled into upstream by Linus. This despite the fact that Red Hat has patched their kernels to the point where Linus would barely recognize them if someone plunked the source to them on his disk, but it's that whole Not Invented Here thingy again, Red Hat invented KVM and was looking for an excuse to not include a Xen dom0 kernel, and there you go. I looked at downloading a dom0 kernel for Fedora 14, but then... hmm. Look. OpenSUSE 11.3 *comes* with a XEN dom0 kernel. So let's just install OpenSUSE 11.3.

OpenSUSE 11.3 is what I eventually had success with. But to do that, I ended up having to fight Red Hat -- again. But more on that in Part 3.

-ELG

Pushing a graphics card into a Xen VM, Part 1

2010-11-13T22:26:00.000-08:00

Part 1 Part 2 Part 3 Part 4 Part 5

One of the eternal bummers for Linux fanboys is the paucity of games for Linux. This is, in part, because Linux is not an operating system, Linux is a toolkit for building operating systems -- and each operating system built with the Linux toolkit is different, but all of them claim to be "Linux". Well, from a game designer's perspective there is no such thing as "Linux" -- each of the variants puts files in different places, each of the variants has a different way of configuring X11, and so forth. And talking about X11, that's another issue. Mark Shuttleworth got a lot of heat for saying that desktop Linux was never going to be competitive as long as it was saddled with the decades of fail that are X11, when he proposed moving Ubuntu Linux to Wayland. But the only Unix variant that has ever gotten any traction on the desktop -- Mac OS X -- did so by abandoning X11 and going to their own lighter-weight GUI library that forced a common interface upon all programs that ran on the platform (except for ported X11 programs, which were made deliberately ugly by the Mac OS X11 server that ran on top of the native UI in order to encourage people to port them to the native UI). Linux fanboys might talk about how OpenGL over X11 isn't theoretically incapable of handling gaming demands, etc. etc., but the proof is in the pudding -- if it's so easy, why isn't anybody doing it?

So anyhow, one of the interesting things about the Intransa Video Appliance is that it looks like Windows if you sit down at the console... but behind the scenes, it's actually VMware on top of a Linux-based storage system. So why not, I wondered, just push the entire video subsystem into Windows via VT-d? I mean, it's not as if Linux user interfaces run any slower remotely displayed over VNC than they do locally, they're pretty light-weight by modern standards. So if you could push the display, keyboard, and mouse into a Windows virtual machine that was started up pretty much as soon as enough of Linux was up and going to support it, you could have a decently fast gaming machine, *and* have a good Linux development and virtualization server -- all on the same box.

So, I assembled my selection of operating systems and started at it. I assembled the bleeding edge of Linux -- Ubuntu 10.10,Fedora 14, Citrix XenServer 5.6.0, ProxMox VE version 1.6, and OpenSUSE 11.3, and set to work seeing what I could do with them...

Next up: Part 2: The distributions.

-ELG

Microsoft in a nutshell

2010-11-02T21:18:00.000-07:00

It's no secret that Microsoft is a company in trouble. At one time they had a significant portion of the smartphone market, now they're an also-ran with single-digit market share. Their attempt to buy consumer marketshare in the gaming console market has generated some marketshare, but also significant losses. Their Zune Phone experiment lasted only two months before ignoble abandonment. The only things they have that make money right now are their core Windows and Office franchises -- the entire rest of the company is one big black hole of suck, either technologically, financially, or both. And while their market share in desktop operating systems is secure for the foreseeable future, with no viable competitor anywhere in sight (don't even mention Linux unless you want to cause gales of laughter, Linux on the desktop is a mess), Office faces a threat from OpenOffice. Plus, their very profitable Windows Server franchise, which accounts for a small percentage of their unit sales but a large percentage of their revenue, is steadily eroding as it becomes clear to almost everyone who isn't tied to Microsoft Exchange that Linux rules the world. Amazon EC3 runs on Linux, not Windows -- as does every other cloud play on the Internet. 'Nuff said.

Today something happened which epitomized this suck. I opened up an email in Microsoft Hotmail. At the top of the email, in red, was the following message: "This message looks very suspicious to our SmartScreen filters, so we've blocked attachments, pictures, and links for your safety."

The title of the email: "TechNet Subscriber News for November".
The sender of the email: technote@microsoft.com

Siiiiigh... even their own spam filter thinks they suck.

-ELG

Setting up bridging and vlans in Fedora 13

2010-10-28T23:38:00.000-07:00

In the previous half of this, I talked about how Fedora's NetworkManager interfered with complex configurations, and discussed how to disable it. Now I'll show you how to define a network that consists of:

Two Ethernet ports, bridged:
- eth0 - to public network
- eth1 - to internal network transparently bridged to public network
VLAN 4000 on internal network, *NOT* bridged to public network
Bridge to VLAN 4000 for my KVM virtual machines to connect to, *NOT* bridged to public network.

Now, to do all of this we take advantage of an interesting feature of the Linux networking stack -- bridges aren't "really" bridges. If a packet arrives at the physical eth1 hardware, it gets dispatched to either eth1.4000 or eth1 based upon the VLAN tag. Only those packets that are dispatched to eth1 actually make it onto the bridge to go to eth0. In other words, the Linux bridging code is a *logical* bridge, not a *physical* bridge -- it is not the physical ports that are connected, it is the logical Ethernet devices inside the Linux networking stack that are connected, and eth1.4000 and eth1 just happen to be connected to the same physical port but otherwise are logically distinct with the dispatching to the logical Ethernet device happening based upon the VLAN header (or lack thereof).

So, here we go:

/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BRIDGE=br0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1:
DEVICE=eth1
BRIDGE=br0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1.4000:
VLAN=yes
DEVICE=eth1.4000
BRIDGE=br1
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-br0:
DEVICE=br0
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=dhcp

/etc/sysconfig/network-scripts/ifcfg-br1:
DEVICE=br1
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.22.2
NETMASK=255.255.255.0

And there you are. Two bridges, one of which has a single port (eth1.4000) for virtual machines to attach to, one of which bridges eth0 and eth1 so that the machine plugged to eth1 on this cluster can also make it out to the outside world (via that bridge) as well as having the VM's on both systems communicate with each other (via the bridge attached to eth1.4000 on the 192.168.22.x network). Internal VM cluster communications stay internal, either within br1 or on VLAN 4000 that never gets routed to the outside world (we'd need an eth0.4000 to bridge it to the outside world -- but we're not going to do that). This does introduce a single point of failure for the cluster -- the cluster controller -- but it's one that's managable, if we need to talk to the outside world after the cluster controller dies we can simply plug in the red interconnect cable to a smart switch that blocks VLAN 4000 rather than into the cluster controller if the cluster controller goes down.

Now, there's other things that can be done here. Perhaps br1 could be given an IP alias that migrates to another cluster controller if the current cluster controller goes down. Perhaps we have multiple gigabit Ethernet ports that we want to bond together into a high speed bond device. There's all sorts of possibilities allowed by Red Hat's "old school" /etc/sysconfig/network-scripts system, and it won't stop you. The same, alas, cannot be said of the new "better" NetManager system, which would simply throw up its hands in disgust if you asked it to do anything more complicated than a single network port attached to a single network.

-ELG

Opaque Linux

2010-10-17T00:16:00.000-07:00

One of the things that is starting to annoy me is the increasing cluttering of Linux with opaque subsystems that have annoying bugs that are difficult to diagnose. Some are easy enough to work around -- udev's persistent-net-dev rule, for example, might make replicated virtual machines have no working network devices, but it's easy enough to simply remove the file (and fix the /etc/sysconfig/network-scripts files to remove any hardwired HMAC there too, of course, since using ovftool to push a virtual machine to ESX automatically gives it a new HMAC). But others -- like the NetworkManager subsystem that I mentioned last week -- either work or don't work for you. They're opaque black boxes that are pretty much impossible to do workarounds with, other than just completely disabling them.

I just finished rebuilding my system with the latest goodies to do virtualization -- I now have a quad-core processor with 12 gigs of memory and VT-d support. As part of that I just upgraded to Ubuntu 10.10, their latest and greatest. I've been using Fedora 13, Red Hat's latest and greatest, at work for the past month. My overall conclusion is... erm. Well. Fedora has its issues, but Ubuntu is getting to the point of utter opaqueness. For example, Ubuntu has a grand new grub system that generates elaborate boot menus. The only problem: Said elaborate boot menus are *wrong* for my system, they all say (hd2) where they should say (hd0). And the system that generates these elaborate boot menus is entirely opaque.... though at least I can go into the very elaborate grub.cnf file and manually edit it. Well, unless a software update has happened, at which point the elaborate boot menu generator subsystem runs again and whacks all your hd0's back to hd2s... even in entries that are old. Red Hat's grubby might be old and creaky, but at least it's never done *that* kind of silliness.

Now, granted, I am running rather unusual hardware in a far different configuration from what a desktop Linux user would want. If you want a desktop Linux system, I still believe Ubuntu is the best Linux you can put onto your system, I've run Ubuntu on the desktop for years and it serves well there. I especially like Ubuntu 10.10's ability to transparently encrypt your home directory similar to the way MacOS can, this will resolve a lot of issues with lost laptops and stolen data, this is an opaque subsystem but necessarily so. You can also put the proprietary Nvidia video drivers onto your system with a simple menu item, while with Fedora 13 you have to fight to put the proprietary driver onto the system (the GPL driver is loaded *in the initrd*, which makes it a PITA to get rid of). In short, if I were running Linux on the desktop, 10.10 is hard to beat. But for my purposes, doing virtualization research with KVM and VT-d, I'm wiping out Ubuntu in the morning and installing Fedora 13.

-ELG

NetworkManager Sucks

2010-10-10T10:04:00.000-07:00

Do a Google on the above search term, and you get over 20,000 results. NetworkManager is yet another attempt by the inept Linux desktop crew to make the Linux desktop look Windows-ish, and is as Fail as you might imagine anything desktop-ish from that gang. I mean, c'mon. We're still talking about an OS that cannot dynamically reconfigure its desktop when you plug an external monitor into your laptop, something that both Windows 7 and MacOS have absolutely no problem with. You expect them to get something as simple as a network manager correct? Uhm, no.

Stuck in Red Hat Enterprise Linux server-land for so many years, I didn't have to deal with NetworkManager. But now I'm doing some work with kvm/qemu which requires me to have the latest and greatest kvm/qemu, which requires me to get bleeding edge on my Linux distribution because mismatches between your kernel version and the userland tools can cause some weird issues, like seemingly random kernel panics where old tools don't fill out new fields and system calls end up randomly crashing (this should never happen, BTW, the kernel should check all inputs and reject any calls that would result in a kernel panic, but I have first-hand proof that this is happening). And bleeding edge either Ubuntu or Fedora means you get to deal with NetworkManager.

By and large folks who have a single network card plus a single WiFi adapter in their Linux box and don't intend to do anything unusual will have no problems with NetworkManager. But I wanted to set up a configuration that was unusual. My Linux server has two gigabit NIC's in it. One goes out to the corporate network. I wanted the other to be a direct connection to the Ethernet port on my Macbook Pro, and then have the two be bridged so it also appears I'm directly on the corporate network. I also wanted to set up a private non-routed VLAN tied to the specific network port between my Macbook Pro and the Linux server so I could set up a private network for file sharing between the two systems -- it's still far more pleasant to do editing, email, word processing, etc. on the Mac and use the Linux box as just a big bucket of bytes. Netatalk is not perfect but works "good enough" for this particular application.

All of this is functionality that Red Hat Enterprise Linux had implemented correctly by RHEL4 days (I know that because I backported most of the RHEL4 stuff to RHEL3 to get all of this functionality working in RHEL3 back in the old days for the Resilience firewalls). That is, we're talking about things that have worked correctly for over six years. Unless -- unless you're using a system that has its network cards managed by NetworkManager, at which point you're SOL unless you disable NetworkManager, because the system absolutely refuses to bring up bridges and vlans if you have NetworkManager enabled.

So my first inclination was to simply take an axe to NetworkManager and go back to the old way of doing things, which as I point out has been working for years. That is, "rpm -e NetworkManager". At which point I find out that half of bloody Gnome seems to have a dependency upon NetworkManager, albeit two dependencies removed, and while Gnome is evil it's the lesser of two evils (hmm, sounds like politics there, eh?). So, I settled for the simple way of disabling it:

# chkconfig NetworkManager off
# service NetworkManager stop

At that point I no longer have any networks configured other than localhost (eep!), so then I get to set up the bridging and VLAN by hand in the /etc/sysconfig/network-scripts files and bring everything up (GUI tools? Linux penguins don't need no freepin' GUI tools, we just cat - >somefile when we want to configure Linux ;-). But I'll talk about that in my next post.

-ELG

And the winner is...

2010-09-29T07:46:00.000-07:00

The iPhone 4.

The Droid X is an awesome piece of hardware. But my experiments with the various Android-based phones said to me that Android is still a work in progress. Each of them had odd bits of user interface that seemed unfinished or clunky or just plain badly thought out. After my brief experience with the Google hiring process it's pretty clear why that's true -- Google's hiring process, other than for a few superstars, has a built-in bias towards young mathematical types who recently took an algorithms course in college and also has a built-in filter to get rid of those of us who've been around long enough to know what we like to do and are good at doing. Youth has its advantages, but also its disadvantages -- young arrogant mathematical types rarely give much thought to user experience.

Which reminds me of an incident at a prior job. Me and my office-mate combined had maybe 15 years experience in the industry at the time, had actually used our product in production environments before joining the company that made it, and were now working on taking it to the next level with a new GUI and a new management infrastructure around our core data engine to make it easier to use in modern network environments. Our boss had about 20 years experience in the industry. So my boss assigns one of these young arrogant math PhD types to mock up a user interface for our project, we gave him the basic architecture and workflow and told him "make it easy to use." So he produces this mock up and calls me in to take a look at it and I scratch my head because I can't make heads or tails of what he's done, it isn't oriented around the workflow of any site admin that I've ever encountered. I call in my office-mate. He can't make heads or tails of it either. We ask this young brilliant mathematical type just out of college questions about how to do various site-admin-ish kinds of things, and he takes us on this long complicated set of procedures through a number of incomprehensible dialogs. My office-mate and I say "This doesn't seem like an easy to use interface for site admins." He goes, "but it's obvious! It's simple!"

So we look at each other, think, "hmm, he seems really sure about this, maybe it's just us," and call in our boss, just telling him "You have to look at this" but not why we want him to look at it. By this time we have 35 years of experience in the room, people who've actually used the technology in question in production environments. He runs through the same thing as we did, and comes to the same conclusion. By this time the arrogant young mathematical type is in pure snit mode. How *dare* we question his impeccable user interface! I explain to him that there's 35 years of experience in this room who've actually used the technology in production environments, so if we can't make heads or tails of it our customers will be utterly lost. "Then your customers are idiots!" he shouts.

Indeed. Indeed. But they are *paying* idiots. Which is what Apple understands. The customer may not always be right, but the customer is what keeps you in business, and customers want something that doesn't require a math PhD to understand or use. And in that regard, the iPhone despite its occasional glitches is still the one to beat, and Android still has a lot of growing up to do. As for that math PhD guy? Eventually after another design disagreement months later (on internals, not GUI -- we knew not to put him anywhere near the GUI by then) he stomped off and turned in his resignation because we didn't properly respect his brilliance, and my manager's manager talked him into working in another area of the company on another product rather than resigning outright, and eventually he turned in his resignation from *that* position after his tastes in user interface proved to be equally daunting for them for the same basic reason. Oddly enough, after a few years of seasoning elsewhere in the industry to brush the edges off his arrogance to turn it into less-obnoxious self confidence, he turned out to be a decent engineer... just don't put him anywhere near a user interface, for cryin' out loud. Which Google would have done immediately, if the Android user interface is any indicator.

-ELG

Droid or iPhone?

2010-08-24T17:11:00.001-07:00

On the left side, weighing in at 4.3 inches, the Motorola Droid X. This massive hunka hunka burnin' cell phone love is running a 1ghz TI OLED processor and has 8gb of built-in memory for programs and 16gb of Microsd memory for data, as well as an 8 megapixel camera.

On the right side, weighing in at 3.8 inches, the Apple iPhone 4. This sleek little beauty is running an 800mhz Samsung A4 with a 5 megapixel camera whose sensor is the same size as the Droid's (i.e., fewer, but more sensitive, pixels).

So who is the winner? The iPhone 4's camera, despite fewer pixels, is definitely better than the Droid X camera. As in, ridiculously good. On the other hand, the iPhone 4 is more tightly-walled than any previous iPhone. As in, jailbreaking it to run "unauthorized" software is ridiculously difficult. Given the way AT&T and Apple cripple the thing, that's a major problem. The iPhone's screen is too teensy for my not-as-young-as-they-used-to-be eyes. On the other hand, it also has the iPod ecosphere with it, and tight integration with my Macbook Pro, and all my current software will continue working with it.

The Droid X, on the other hand, is easily "rooted". Motorola has created their own ecosphere of sorts, with a car dock and charger, home docks, etc. so that you can do pretty much the same things as with the Apple ecosphere. It doesn't by default have any integration with my Macbook Pro, but a third-party program called The Missing Sync will do most of that. The big screen is nice for older eyes. But Android itself is ugly and clunky, though serviceable.

So who's the winner? Call it a draw -- for now. Which presents a problem, because my aging iPhone 3G really is not liking iOS 4.0, everything runs really slow and clunky. Maybe I ought to flip a coin... or maybe if I wait a few more months, the horse will sing. Hmm...

-ELG

Migration concluded

2010-08-24T17:07:00.000-07:00

The penguin has now landed at a new employer. I'll update my LinkedIn profile with that information after I've had a few days to de-stress and relax... the past couple of weeks have been a wild, wild ride, reminding me a bit of the last couple of weeks before the deadline for a major new product. But having too much interest in your skills definitely beats the alternative :).

-- ELG

Action items

2010-08-09T20:12:00.000-07:00

I had joined the company a few weeks earlier and was sitting in yet another raucous meeting. The latest attempt at a new product had failed, and the blame-casting and finger-pointing were at full tilt. Finally I sighed, and added my own say. "Look. I'm new here and I don't know what all has gone on, and really don't care who's to blame for what, blame isn't going to get anything done. What I want to know is, what do we need to do now?"

Person 1: "Well, we failed because we weren't using software engineering system X" (where X is some software engineering scheme that was popular at the time).

"Okay, so we'll use software engineering system X, I have no objection to using any particular system, as long as we use one. What's the first thing we need to do, in that system?"

Person 2: "We need to figure out what we want the product to do."

"Okay, let's do that. What is the product supposed to do?"

We discussed it for a while, then from there the meeting devolved into a list of action items, and eventually broke up with another meeting scheduled to work on the detailed functional requirements. But on the whiteboard before we left, I had already sketched out the basics of "what we want it to do", and eventually that turned into an architecture and then a product that is still being sold today, many years later.

So what's my point? Simple: Meetings must be constructive. One of the things my teacher supervisors told me, when I first entered the classroom, was to always ask myself, what do I want the students to be doing? And then communicate it. A classroom where every student knows what he's supposed to be doing at any given time is a happy classroom. Idle hands being the devil's workshop and all that. The same applies to meetings. Unless it's intended to be an informational meeting, meetings should always be about, "what do we want to do". And meetings should never be about blame-casting, finger-pointing, or any of the other negative things that waste time at meetings. No product ever got shipped because people pointed fingers at each other.

Everybody should have a takeaway from a development meeting -- "this is what I am supposed to be doing." Otherwise you're simply wasting time. So now you know why one of my favorite questions, when a meeting has gone on and on and on and is now drawing to a close but without any firm conclusion, is "what do we need to be doing? What are our action items?" We all need to know that we're on the same page and that we all know what we're supposed to be doing. That way there are no surprises, there are no excuses like "but I thought Doug was supposed to do that task!" when the meeting minutes show quite well that Doug was *not* assigned that action item, and things simply get done. Which is the point, after all: Get the product done, and out the door.

--ELG

* Usual disclaimer: The above is at least slightly fictionalized to protect the innocent. If you were there, you know what really happened. If you weren't... well, you got my takeaway, anyhow.

Architectural decisions

2010-08-08T19:31:00.000-07:00

Let's look at two products. The first product is a small 1U rackmount firewall device with a low-power Celeron processor and 256 megabytes of memory. It can be optionally clustered into a high availability cluster so that if one module fails, the other module takes over. Hard drive capacity is provided by a 120gb hard drive or a 64GB SSD. The second is a large NAS file server with a minimum configuration of 4 gigabytes of memory and with a minimum hard drive configuration of 3.8 terabytes. The file system on this file server is inherently capable of propagating transactions due to its underlying design.

So: How are we going to handle failover on these two devices? That's where your architectural decisions come into play, and your architectural decisions are going to in large part influence how things are going to be done.

The first thing to influence our decisions is going to be how much memory and CPU we have to play with. This directly influences our language choices, because the smaller and more limited the device, the lower level we have to go in order to a) fit the software into the device, and b) get acceptable performance. So for the firewall, we chose "C". The architect of the NAS system also chose "C". As an exercise for the reader, why do you think I believe the architect of the NAS system was wrong here? In order to get acceptable performance with the small module, we chose a multi-threaded architecture where monitor threads were associated with XML entries of what to monitor, and faults and alerts were passed through a central event queue handler which used that same XML policy database to determine which handler module (mechanism) to execute for a given fault or alert event, nothing was hard-wired, everything could be reconfigured simply by changing the XML. The architect of the NAS system had an external process sending faults and alerts to the main system manager process via a socket interface using a proprietary interface, and the main system manager process then spawned off agent threads to perform whatever tasks were necessary -- but the main system manager process had no XML database or any other configurable way to associate mechanism with policy. Rather, policy for handling faults and alerts was hard-wired. Is hard-wiring policy into software wise or necessary if there is an alternative?

The next question is, what problem are we going to solve? For the firewall system, it's simple -- we monitor various aspects of the system, and execute the appropriate mechanism specified by XML-configured policies when various events happen with the goal of maintaining service as much as possible. One possible mechanism could be to ask the slave module to take over. Tweaking policy so that this only happens when there's no possibility of recovery on the active module is decidedly a goal because there is a brief blink of service outage as the upstream and downstream switches get GARP'ed to redirect gateway traffic to a different network port, and service outages are bad. We don't have to worry about resyncing when we come back up -- we just resync from the other system at that point, if we had any unsynced firewall rules or configuration items that weren't on the other system at the point we went down, well, so what. It's no big deal to manually re-enter those rules again. And in the unlikely event that we manage to dual-head (not very likely because we have a hardwired interconnect and differential backoffs where the current master wins and does a remote power-down of the slave before the slave can do a remote power-down of the master), no data gets lost because we're a firewall. We're just passing data, we're not serving it ourselves. All that happens if we dual-head is that service is going to be problematic (to say the least!) until one of the modules gets shut down manually.

For the NAS system, it's quite a bit harder. Data integrity is a must. Dual-heading -- both systems believing they are the master -- requires either advanced transaction merge semantics when partitioning is resolved (transaction merge semantics which are wicked hard to prove do not lead to data corruption), or must be avoided at all costs by having all systems associated with a filesystem immediately cease providing services if they've not received an "I'm going down" from the missing peer(s), have no ability to force the missing peer to shut down (via IPMI or other controllable power), and no way of assuring (via voting, or other mechanisms) that the missing peers are going down. Still, we're talking about the same basic principle, with one caveat -- dual-heading is a disaster and it is better to serve nothing at all than risk dual-heading.

For the NAS system, the architectural team chose not to incorporate programmable power (such as IPMI) to allow differential backoffs to assure that dual-heading couldn't happen. Rather, they chose to require a caucus device. If you could not reach the caucus device, you failed. If you reached the caucus device but there were no update ticks on the caucus device from your peer(s), you provided services. This approach is workable, but a) requires another device, and b) provides a single point of failure. If you provide *multiple* caucus devices, then you still have the potential for a single point of failure in the event of a network partition. That is because when partition happens (i.e. you start missing ticks from your peers), if you cannot reach *all* caucus devices, you cannot guarantee that the missing peers are not themselves updating the missing caucus device and thinking *you* are the down system. How did the NAS system architectural team handle that problem? Well, they didn't. They just had a single caucus device, and if anybody couldn't talk to the caucus device, they simply quit serving data in order to prevent dual-heading, and lived with the single point of failure. I have a solution that would allow multiple caucus devices while guaranteeing no dual-heading, based on voting (possibly weighted in case of a tie), but I'll leave that as an exercise to the reader.

So... architectural decisions: 1) Remember your goals. 2) Make things flexible. 3) Use as high-level an architecture as possible on your given hardware to ensure that #2 isn't a fib, i.e., if what you're doing is doable in a higher-level language like Java or Python, for heaven's sake don't do it in "C"!. 4) Separate policy from mechanism -- okay, so this is same as #2, but worth repeating. 5) Document, document, document! I don't care whether it's UML, or freehand sketches, or whatever, but your use cases and data flows through the system *must* be clear to everybody in your team at the time you do the actual design or else you'll get garbage, 6) Have good taste.

Have good taste? What does that mean?! Well, I can't explain. It's like art. I know it when I see it. And that, unfortunately, is the rarest thing of all. I recently looked at some code that I had written when I was in college, that implemented one of the early forum boards. I was surprised and a bit astonished that even this many years later, the code was clearly well structured and showed a clean and well-conceived architecture to it. It wasn't because I had a lot of skill and experience, because, look, I was a college kid. I guess I just had good taste, a clear idea of what a well-conceived system is supposed to look like, and I don't know how that can be taught.

At which point I'm rambling, so I'm going to head off to read a book. BTW, note that the above NAS and firewall systems are, to a certain extent, hypothetical. Some details match systems I've actually worked on, some do not. If you worked with me at one of those companies, you know which is which. If you didn't, well, try not to read exact details as gospel of how a certain system works, because you'll be wrong :).

-ELG

The migration of the penguin

2010-08-01T21:02:00.000-07:00

I have added a link to my resume in the left margin, in case someone is interested in hiring a long-time Linux guy who knows where the skeletons are buried and, if you need something Linux done, probably has already done it at least once...

-ELG

The world's most reluctant Linux advocate

2010-07-31T12:35:00.000-07:00

In the fall of 1995, I had successfully brought to completion the project to comply with new federal and state reporting standards for school discipline for the consortium of school districts that our consulting firm served, and was busy cleaning up the master student demographics suite to properly incorporate the new discipline screens rather than have it be a stand-alone subsystem reaching into the student database. It was a hard slog -- the code was a mess. The guy who had written it, who I had replaced, was a math guy, not a computer guy, and he had no inkling of simple things like comments or code reuse, the product life cycle was a mystery to him. His notion of code reuse was to cut and paste the same code multiple places, and there were some significant bugs that I was cleaning up. The only good news was that the code was heavily componetized -- it was basically a hundred small programs tied together by a menu system and a common database, though some of the programs seemed to be bigger programs because they forked out to other programs to provide more screens to school secretaries. All of this was running on SCO Xenix or Unix, depending upon the school and when it had bought our software suite.

So during all this my boss calls me into his office and says, "one of our districts has asked if we're investigating Linux as a possible way to bring down costs for school districts. What do you think?" Now, one thing to remember is that my boss was a big old ex-IBM bull, a no-BS kind of guy, and we got on about like you'd expect from two people with strong opinions but mutual respect. "Linux is freeware downloaded off the Internet," I replied. "Don't we have enough trouble maintaining our own code right now without having to maintain some freeware downloaded off the Internet too?" And that was pretty much that. Still, I thought, "hmm, I have that new Windows 95 machine at home, I bet it'd run Linux." I'm a geek, and what geek wouldn't want to play with a free operating system?

So, after work I headed off to the local Barnes & Noble to grab a book about Linux. The one I bought had something called "Slackware 95" on CD in the back. I took it home with me that night and installed it on a partition on my home computer. So I installed it and figured out how to get "X" running and... well, it worked okay. fvwm was ugly and crude and limited, and there wasn't much desktop software, no real word processor, but I knew LaTex and it had LaTex, so that was good. It drove my laser printer fine too. So the next day at work, I went ahead and installed it on our eval machine at work, where we'd also installed Windows 95 to see what we could do about porting our UI to it. I compiled our source tree on Linux and... hmm, it just compiles, just like it compiles on SCO Unix? And it actually ran!

So I started developing on Linux instead of on SCO Unix, mostly because it was much easier to get Emacs up and going and I prefer Emacs to 'vi' (let the flame wars begin!), not to mention that the GNU tool suite is a lot nicer than the old-school Unix tools. When I finished a module and did initial smoke testing I'd then copy the code over to SCO Unix and compile again there. From time to time I'd also go into the menu system and create a Linux version of one of the SCO Unix system administration programs that we'd accumulated over the years to allow school technology coordinators to manage the system. But I still hadn't considered actually deploying Linux at schools. While it seemed the technology held up okay -- our software actually ran faster on Linux than on SCO Unix -- the business objections were formidable. "We don't want to trust our critical student data to some hackerware downloaded off the Internet!" was the least of it.

That changed in the Spring of 1996, however, when Red Hat Software came out with their 3.0.3 version of Red Hat Linux, which they marketed as "Linux for business". It came in a box! With a manual! From a real company! Complete with a shadow businessman logo wearing a red hat marching off to do business with his briefcase! For the first time, the possibility of actually using Linux as part of our business was not ridiculous. The only thing I really didn't like about 3.0.3 was that all of the system administration tools were TCL/TK GUI scripts, but given that I'd already written a number of menu-based system administration scripts, that didn't seem a fatal objection. I switched from Slackware to Red Hat 3.0.3, and kept on developing under Linux rather than SCO Unix.

So, early June 2006 came along, and we got another school district as a customer. My boss called me into his office again. "What would it take to port our software to Linux?" he asked. "I pretty much already have it ported," I said. "Maybe two weeks to do a thorough job of testing and filling in any system administration scripts that aren't yet rewritten, and it would be ready." "We have this new customer. With our winning bid we could make more money selling Linux rather than SCO Unix, the OS isn't specified in the bid, should we do SCO Unix or Linux?" "Well, Linux has some risks involved in it," I replied. "We still haven't tested it with real data, it should work, but there's no guarantee." He then said that we hadn't won the hardware bid, and there was no guarantee that it would work with SCO. I then suggested a dual-OS strategy -- plan on using *either* of the operating systems, depending upon which one worked with the hardware when the hardware came to us for us to install the OS and administrative software. Given the wholesale pricing I'd gotten from Red Hat Software, we could purchase an official copy of Red Hat Linux 3.0.3 for each school to counter the "hackerware downloaded from the Internet!" objection and basically it was lost in the noise compared to the significant cost of SCO Unix.

So the hardware came in, and the tape drive was supported by Linux, while it was not supported by SCO Unix. We had two options at that point -- delay the deployment until tape drives supported by SCO Unix could be procured, or deploy with Linux. We were scheduled in two weeks to have the machines at a high school gymnasium at the school district to train the school secretaries on how to use the software. It would take at least two weeks to argue with the school district and the hardware vendor about tape drives.

"We go Linux," I told my boss. And we did. I spent the next two weeks sweating the details, making sure everything worked, using real data from a real school district (with the student ID information masked out) to validate that all functions of the software itself did what they were supposed to do, going through all the management screens to make sure they worked properly with the hardware on the systems, and so forth. On the appointed day I drove the main Linux development machine to the school district myself, and stayed on hand while the secretaries all booted their machines, just in case something broke, and... it didn't. Everything Just Worked, without a hitch, all the demos went off as planned, and my inservice training on the discipline system went on as usual, the secretaries were quite attentive, laughed at the right points (i.e., when I produced an official state discipline form with a ridiculous discipline infraction for them to punch into their computers and made an offhand humorous comment about it), and... phew!

That's always the moment of truth: when the product hits the customer's hands. You either pass or fail at that point. I'm proud to say that we passed, and became one of the first of what eventually became a thundering storm of people migrating away from proprietary Unix systems to Linux. Over the next three years we transitioned all of our schools to Linux -- it simply made things easier only having to maintain one set of administrative tools, and it wasn't as if it cost any money, we usually did it when they were upgrading old hardware so we were getting paid for that service and Linux came along for the ride, and it Just Worked. And what more can you say?

I suppose there's a couple of lessons there. First, don't dismiss Open Source software just because it's "some hackerware downloaded off the Internet." Secondly: don't use Open Source software just because it's Open Source if you can't make a business case for it in terms of risks vs. benefits. We couldn't make a business case for it in the fall of 1995, we simply did not have the engineering cycles to handle a transition to Linux given the state of Linux at that time, the risks outweighed the possible benefits. By the summer of 1996, when the code base issues had been resolved, the primary objection of customers about using Linux had been resolved, and the issues of hardware compatibility and profit became key, Linux simply Made Sense. It still wasn't the safe choice. But the risks were limited enough at that time compared to the benefits to justify taking the risk.

-ELG

The value of an education

2010-07-15T23:16:00.000-07:00

For some reason Americans seem to believe education is something people receive. People go to college to "receive an education". This implies that students are simply receptacles. The professor opens up their heads and drops knowledge in, then sews them back up. This worries me when I'm looking at the quality of the young people entering the computer science field today, because while they've had exposure to a lot of technology, by and large it's been as users, not as participants. The actual technology is something they don't even think about -- it's transparent, just a part of their world, not something they actually see and think about.

The problem is that education isn't something you receive. Education is something you do. I graduated from a middle-tier university. Which means nothing at all, actually, because I knew my **** when I left there, I'd actually designed bit-slice CPU's and microcode for them and built hardware and written programs in microprocessor assembly language *for fun* while the guy across the street with the 4.0 GPA knew nothing except what was on the test, I mean, he'd been writing software on Unix minicomputers for four years and he didn't even know what 'nroff' was or that he was using Unix! So yeah, it's all about what use you make of the experience. I spent as much time in professor's offices talking about my latest projects as I spent studying, which hurt my GPA, but (shrug). I'm still employed in the computer field today. The guy across the street? Nope.

That is one reason why Open Source is exciting to me, and why people who have a background in the Open Source community interest me far more than people who have a 4.0 average from Big Name University. I'm looking for doers, not regurgitators. What gets software shoved over the transom isn't the ability to memorize what's going to be on tests, it's what, for lack of a better term, I call "get'r'done". The problem I see is that the technology has become so capable, so complex, so difficult to grasp, that the number of people who could learn the basics of some simple technology like a Commodore 64 then build up to writing significant Linux kernel subsystems has basically slowed to a dribble. Simple and relatively open technology like the Commodore 64 where you could grasp the entire design all by your lonesome (the programming manual came with a schematic of the computer in it!) simply doesn't exist anymore. For good reason in most cases, today's computers have far better functionality, but how are we going to get the people with the "big picture" today when there's no "little picture" like a Commodore 64 to build up from?

So anyhow: That's a problem. It's a problem I find with a lot of the younger software engineers. I've managed some very bright youngsters, but that lack of what I'll call big picture thinking hinders them greatly. They simply don't understand why a busy loop waiting for input is not acceptable unless there is no alternative and why it should have a timer to put the process to sleep between samples, or what the hardware looks like and how to program the front panel that's driven by a PIC processor. They're like ferrets, it's all "oooh, shiney!" to them, with no rhyme or reason or understanding of what's actually happening under the surface. And I have no idea at all what's going to happen when all us older farts get put out to pasture either via corporate executives calling us "too old and expensive" or simply getting too tired and retired... there just isn't enough of the young folks who have the slightest clue. Not that we were the majority even when I was 21, but at least there was a sizable number who *did* have a clue then... and you can find a lot of their names looking at the early Linux kernel patch sets. But even the Linux kernel crowd is graying today... and what happens when we're no longer around, given that the number of young people today who understand technology at the same comprehensive level we do -- or that we did at age 21 -- is essentially zero?

-ELG

That XKCD 619 feeling

2010-06-29T10:44:00.000-07:00

A Linux advocate says:

The case for using Apple software of Microsoft Windows for something is so slim it tends to sound like the techno lust (sooo shiny ...) or the machinations of a mad man (I HAVE TO HAVE IE!!!!!).

Ah yes, I'm getting that XKCD 619 feeling again, where Linux advocates say about usable user interfaces, "why would anybody want that?!". I've been using and developing for Linux since 1995, so I'm not exactly a newbie. I have the latest Ubuntu on my big Linux development machine (the latest Fedora is similar in my experience) and you know the latest Ubuntu desktop with a high-end graphics card reminds me of? It's as if someone had described MacOS and Windows 7 to engineers in the old Soviet Union, and they sat down and wrote their own clunky half-a** clone based upon nothing but those descriptions. You can practically hear the clunks of heavy metal and whirring of primitive gyroscopes as you operate it. I'm sorry, but anybody who says that Linux has the usability of MacOS or Windows 7 on the desktop is drinkin' some mighty strong kool-aid.

KDE is a bloated incoherent resource-hogging mess (consider it the Windows Vista of Linux desktops), and Gnome's primitive old-school Windows 95 Meets Motif style desktop is usable compared to the competition only if you have a high-end graphics card and can enable 3D Effects and their CCCP-style Expose' and Spaces clones (I say CCCP-style because they have significant usability issues compared to the real thing). And both are limited by "X" which has significant problems dealing with the modern world and hot-pluggable monitors. As in, it doesn't do it. On the day when you can plug an external monitor into your Linux laptop and have the desktop automagically just extend onto the new monitor, with no "dead spaces" and no problems dragging and dropping things between monitors, let me know. Right now, due to Xinerama basically being abandonware, the only multiple-monitor setup that works properly is nVidia's, and only with two same-sized monitors (otherwise there are created "dead spaces" that can eat your windows so you can't get at them), and only if you manually set it up using nVidia's own setup program. Wow, how competitive with MacBooks (where it Just Works) or Windows 7 (one right-click to get Display Settings, then select "extends desktop" rather than "mirrors desktop" from the Displays options) is that? Err... not!

I use Linux where it is appropriate -- my web and email server is running Linux, and I'm developing on Linux for embedded servers that run Linux. But to say there's no reason to use anything other than Linux is just koolaid-drinking ... and, uhm, for the guy who says his HTC Evo 4G proves FOSS rocks, I might point out that the EVO 4G is running a proprietary closed-source "skin" (HTC's "Sense" UI). Yeah, that's "proof" alright... but maybe not of what the original commenter claimed :).

--ELG

The new alternative to VMware: KVM

2010-06-22T15:17:00.001-07:00

Both the latest Ubuntu and the latest Red Hat are shipping with a new alternative to VMware Server called QEMU-KVM. I've been playing with it, and it is much faster and lighter weight than VMware Server, as well as being more flexible and easier to use.

To get started with QEMU-KVM on Ubuntu 10.04, first install kvm and qemu-kvm from aptitude. Then install virt-manager. After that, System Tools->Virtual Machine Manager will bring up your virtual machine management console.

You'll see two entries when you do this:

localhost (QEMU Usermode) - Not Connected
localhost (QEMU)

Double-click on localhost (QEMU) and it'll connect to the local root virtual machine manager. You could also connect to other machine's managers, if you're wanting to, say, manage the virtual machines on a host in your data center, by using File->Add Connection. Now you'll probably want to set up a data pool for use by your new virtual machines. Most of us put the virtual machines on their own partition, not on the root partition, but the default data pool is in /var/lib/libvirt/images -- which is on the root partition. Ick. Never fear, right-click on the localhost(QEMU) and select 'Details', then click on the 'Storage' tab when you get the details. Click "+" to add your new storage pool, once you define its location click the green 'play' button to make it active, then hit the red delete button to get rid of the 'default' pool. You now have a new default storage pool at the location you desire.

Okay, so you have your data pool, now what about creating a virtual machine? Easiest way to do that is to use an ISO image of your favorite distribution. Just right-click on the localhost(QEMU) entry again, and select 'New'. The resulting wizard is ridiculously easy to navigate as long as you remember that it's going to create it in whatever your enabled data pool is when you tell it to 'create a disk image on the computer's hard drive'.

So, after this you should be able to run the virtual machine and install your ISO on it. Remember that ctrl-alt gets you out of the QEMU console back into the regular Linux desktop environment, and you'll be fine. To open a console, just right-click and select 'open'. Or once you have a VM set up and installed, you can shut it

Okay, so what's the limits of QEMU/KVM right now? First of all, don't expect to run graphical environments via the normal console with any kind of responsiveness. It emulates a very slow/old display card which is then screen-scraped by a vnc server. KVM is mostly useful for running non-GUI setups, such as Asterisk servers or hosted virtual web servers. Secondly, some operating systems might not install at all into KVM due to driver support issues. Finally, there is no equivalent of "VMware tools" to integrate with your host environment so you can move your mouse freely between the virtual machine terminal and the host OS. Your best bet there, if you want a graphical console inside a virtual machine, is to install VNC in the virtual machine and then use VNC to view your graphical console.

But aside from those limitations, KVM appears to be working quite well. It is definitely better on Linux than VMware Server, and if you need to create a vmdk to import into VMware on some other non-Linux host, it's easy enough to just 'qemu-img convert -O vmdk VbAst32.img VbAst32.vmdk' and voila, the new virtual machine will import cleanly into VMware. And of course VPEP runs inside a KVM virtual machine just fine... :).

-ELG

Doing an installer right: Microsoft Office 2010

2010-06-02T21:38:00.000-07:00

So out of curiousity I downloaded and installed Microsoft Office 2010 today (don't freak out about piracy, folks -- I'm a Microsoft TechNet subscriber and this copy is a quite legit eval copy). I haven't had a chance to use the software yet, but one thing I have to say is about the installer: Microsoft did it right.

A good installer must do the following things:

It must be SIMPLE. People don't want to select lots of stuff, they just want to click one button and have it happen. With the Office 2010 installer you click the 'Install' button (or 'Upgrade' button if Office 2007 is installed on your system), it prompts you for the license key, validates it right then and there, you click 'Next', accept the license, and then it just does it. It's basically four clicks (assuming you can cut-and-paste the license key from the TechNet site of course, if you have to type it in then there's a few keystrokes too). If your geeks or marketroids insist on all sorts of additional functionality, hide it behind a little "+" sign or something where people won't get freaked out about it, users just want it to Just Work, they don't care about all that stuff.
It must handle both upgrades and fresh installs in a clean manner. So if a prior version is already installed, it should give you the option of upgrading it and keeping your configuration settings as much as possible.
If possible, it should offer to import settings from a prior program, or from a competing program, much as the latest IE will import settings from an install of Firefox or Safari.
It must handle aborted installs gracefully. The installer should be idempotent -- you should be able to run it regardless of what state the system got left in, and it will just Do The Right Thing. If the process fails halfway through removing the old version of the software due to something out of your control -- like the moron behind the keyboard accidentally hitting the shutdown button when he was trying for another button -- you should be able to run the installer again and have it Do The Right Thing, knowing what part of the process was last successfully finished and continuing from there, or unwinding back to the original conditions and starting from scratch again, but either way it should Just Work.
Once it starts actually installing, it should just do it, not bother you anymore, until the end of the process where, if a reboot is required, it can prompt you for that.

Microsoft has accomplished all of these things with the Office 2010 installer. And you should do the same when you write yours.

So let's state that principle one more time: End users want it to Just Work. Geeking out with oodles of settings and such might make marketroids drool with all the checkboxes they can fill in on the inevitable "competitive comparison checklist", and might make geeks drool over all the cool widgets they can play with, but for 99% of the people out there all you're doing is a) confusing them, and b) making your technical support people pull their hair out trying to deal with end users who want it to Just Work rather than have all these options to select. Especially now, with 2 terabyte hard drives selling for $130 at Fry's and most computers shipping with a minimum of 4 gigabytes of RAM, it doesn't make sense to do anything other than install the whole tamale in the default place. For 99.9% of your users, that's going to be all they want. For the other 0.1% of your users, put that little "+" if you want... just put it somewhere out of the way so someone has to *want* to click on it. And realize that in reality, nobody cares other than a few fellow geeks.

Thinking like an end user. That's what it takes to make a program that Just Works. That's something I've had to pound into my team's head over and over and over again over the years, think like an end user, not like a geek... and Microsoft, at least, appears to have finally learned that lesson in at least this one instance. At which point I must congratulate them, because it's *hard* for geeks to think like end users, but in this one instance, at least, they managed it.

--ELG

Configuring Compiz to emulate Spaces and Expose

2010-05-27T10:33:00.000-07:00

If you have an OpenGL-capable video card and driver for Linux, like the GeForce 7900 GS in my big box, you can run Compiz on Ubuntu 10.04 and emulate Spaces and Expose'. So here's how to do it in Gnome (I do not recommend KDE on Ubuntu 10.04 due to some serious bugs I found):

Install the latest proprietary driver via System->Administration->Hardware Drivers. Without this installed, my GeForce 7900 simply would not do 3D, and Compiz wouldn't run.
Install the Compiz settings manager:
- # apt-get install compizconfig-settings-manager simple-ccsm
Identify the X11 mouse buttons you wish to use. Sorry, I identified those via trial and error. On my Logitech Anywhere MX mouse, here is the map from mouse action to X11 mouse button:
- Button 1: Left click
- Button 2: Right click
- Button 3: The 'menu' button
- Button 4: scroll wheel forward
- Button 5: scroll wheel back
- Button 6: scroll wheel left
- Button 7: scroll wheel right
- Button 8: backmost-arrow button (on left side of mouse)
- Button 9: forwardmost-arrow button (on left side of mouse)
Select Preferences->Appearances and select your theme that you want, then click the Visual Effects tab. Select 'Normal' unless you want the wobble-on-window-move effect (which I hate because it makes it hard to accurately place the window). It should do some work, then ask you if you want to keep it. Say yes :). Then exit out of that.
Select Preferences->CompizConfig Settings Manager
Under Desktop, choose "Expo". This is half of the Spaces look-alike, though it doesn't *quite* work like Spaces. Under Expo Key, set it to whatever key you wish to use to enable Expo, either Windows-E (the usual setting, note that the Windows key is called 'Super' in this UI because the Compiz folks apparently hate Windows ;), or a function key of your choice. Note that I recommend using the 'Super' prefix for that function key, because otherwise you end up conflicting with applications, since normal keyboards don't have a 'fn' key like Mac keyboards that can be used to access regular function key codes of function keys assigned functions in the GUI.
Expo won't work with four workspaces all in a row, so right-click on the four workspaces in a row at the bottom right of your screen, select 'Preferences' from the resulting pop-up menu, and set it to a 2x2 grid.
Now you need to set your arrow keys left/right/up/down to move you between the workspaces. Click 'Back' in the Compiz Settings UI, and select 'Desktop Wall'. Click the 'Bindings' tab. Expand the 'Move within wall' collection, and set the Move Left/Right/Up/Down keyboard shortcuts. I suggest Super-left, Super-right, Super-up, Super-down.
Okay, click Back, and now let's set our Expose'-lookalike. This is under 'Windows' and is called 'Scale'
The question is, which one of these do you want to use? "Initiate Window Picker" allows "Expose" on all windows on the current workspace. Unfortunately, if you have a multi-monitor system, it puts all those windows onto the monitor where your mouse is currently residing. This gets very cluttered if you have two large monitors (I'm running a pair of 1050p monitors). In that case, I suggest using 'Initiate Window Picker for Windows on Current Output', which does it just on the monitor your mouse pointer is hovering over. I selected Mouse Button 9, the forward-arrow button on the left side of my mouse, to do this.

The end result: Something that approximates what Expose' and Spaces would have looked like if implemented by someone in the Communist-era USSR who'd approximately heard descriptions of how they worked, but had never actually seen them. You can practically hear the clunks of heavy metal and whirring of primitive gyroscopes as you activate them. The Expose-clone doesn't work well with multiple displays because of its bad habit of trying to collect all the windows onto the current display, thus requiring the 'on Current Output' kludge to avoid getting window overload. The Spaces clone requires more mouse clicks to move windows between "Spaces" (it doesn't simply exit to the workspace you just moved the window to, it stays activated until you double-click on a workspace), and doesn't have a handy icon to get to it quickly in case you're using a laptop that lacks a 9-button mouse (!). And like virtually all things dealing with Linux user interfaces, it takes a jillion-step process to get it configured and set up, with oodles of trial and error to figure out which X11 "mouse button" corresponds to which actual button on a mouse.

In short, Linux programmers still haven't figured out that users just want things to work. I've had to whack my own teams on the hands a few times when they brought back a design prototype that had oodles of screens, buttons and widgets to tweak -- "No, I want one input box here, one submit button there, that's all the user cares about, he just wants to do the job, he doesn't want to adjust all the internal stuff you're exposing here." Complexity is the enemy of user interface usability and consistency -- something which geeks seem to not understand, I've had to do this (shoot down too-complex user interfaces) repeatedly over the past ten years. Sadly, there is nobody to do this for Linux (well, except what Nokia did for Maemo, which works really well at giving a consistent user interface to all Maemo apps, but Maemo is pretty specific to its particular environment) -- and thus Linux continues being an incoherent mess for the average end-user.

Still, for my purposes, it works fine at keeping my workflow working while I run a bunch of KVM virtual machines with VNC viewers into them. So I'm a bit less grumpy today. But I sure wish there was a benevolent dictator for the Linux user interface the way there is for the Linux kernel itself... it's frustrating, the technology is there, but nobody has actually turned it into a coherent whole, and the distribution vendors seem either overwhelmed by the situation or just don't care. Oh well, back to work...

-ELG

Attack of the Linux Penguins

2010-05-26T15:54:00.000-07:00

Recently I've been using Ubuntu 10.04 Lucid Lynx as my desktop development system. This is the first time in a long time that I've used Linux on the desktop -- for the past three years I've been using MacOS on the desktop, sharing the source code tree via NFS to a Linux VMware client to do compiles and testing. Since what I've been principally doing during this time is appliance and distribution design, the fact that VMware is not the speediest of environments did not make much difference.

However, I needed more virtual machines to look at potential client configurations than I could easily fit on a Macbook Pro, even the very well endowed Macbook Pro that I own (which has 8 gigabytes of memory), so clearly I needed to throw more hardware at the problem (heh! Typical software engineer answer!). So I installed 10.04 on a well-endowed desktop/server system complete with RAID5 array and 8GB of RAM, and set out to configure clients using the KVM (Kernel Virtual Machine) system that comes with Ubuntu 10.04.

So, the Good:

KVM rocks. The performance I get out of KVM is far superior to anything I've ever experienced with VMware even on faster hardware. I am currently running a load on my system that would have both cores maxed out under VMware due to VMware's per-VM overhead. Under KVM, I'm at around 30% on both cores.
The Virtual Machine Manager that ships with Ubuntu for managing KVM is far easier to use than the latest editions of VMware Server's management UI, but not as good as the VMware Fusion management UI on MacOS in that it doesn't have any equivalent, as far as I can tell, of the VMware Tools module that allows transparent window pointer moves between the KVM virtual screen window and the host OS screen.
Gibber is a really cool Twitter/Facebook/etc. client, easily as good as anything on MacOS. I especially like its multi-pane capabilities for viewing multiple streams in parallel.
Performance in general rocks. It feels snappy and things happen quickly, even though it's running software RAID5 so you'd expect significant overhead from the disk driver. But there isn't -- Linux's cache-block elevator does very well at optimizing RAID access.

Now for the bad:

The user interface in general is reminiscent of mid-1990's Windows. It looks and feels dated and obsolete, and lacks many of the modern navigational aids of Windows 7 or MacOS 10.5.
While I got multi-screen support working for two displays attached to my nVidia 7900 video card (which has two outputs), the dated UI gives no easy way to navigate between the various applications open on the two screens. The window boxes clustered at the bottom of the leftmost screen are not easily accessible from the rightmost screen, and there's no equivalent of Apple's Expose' or Windows 7's similar application-picker function that can be bound to a mouse button so selecting an application is a mouse button away.
Neither the KDE nor Gnome filesystem browser allowed providing a user name/password to a CIFS server whose shares you wished to list. As a result, I could not use the CIFS browser to browse to shares on my Mac or on our Windows servers here on the office, neither of which will provide you with a share list unless you authenticate first. On the Mac, this Just Works -- the initial attempt to get the share list fails, but then you click the "Connect As" button, put in the user name and password, voila.
In general, integration with Apple and Windows networks was pathetic. I was reduced to doing manual mounts via the CLI mount.cifs command, which I should never have to do on a supposedly modern desktop operating system.

And finally, the ugly:

KDE on Ubuntu 10.04 is atrocious. There is an important directory service under KDE that sucks up gigabytes of RAM. I had to switch back to the unsatisfactory, but at least functional and efficient, Gnome UI to get work done. KDE used to be fast and clean, but it has become as bloated and dysfunctional as Windows.
Multi-screen support on 10.04 has taken several steps backwards. Window managers crashed when I used the standard X11 Xinerama extension. So I used the proprietary TwinView nVidia driver for my nVidia 7900 video card (a high end card from two generations back, somewhat obsolete today but I use it for Linux because its support is mature) and now multi-screen support is back... but only for the displays attached to this single video card.

My conclusion: Ubuntu 10.04 is thus far the closest that Linux has gotten to a usable desktop operating system. It is a state of the art desktop environment -- if you have been locked in a server room with pizza slid under the door for the past 15 years and still think Windows 95 is the be-all and end-all of user interface design. Both Windows 7 and MacOS 10.5/10.6 put it to shame on all measures of appearance and usability.

I can appreciate the effort that has been put into Ubuntu 10.04, and the difficulties involved in trying to turn a mass of random software from miscellaneous strangers into a coherent operating environment. But that does not change the fact that Linux on the desktop remains stuck in a time warp, fighting the battles of two decades previous in an era where time has moved on. It is especially sad that KDE today is no more usable than KDE 2.0 was ten years ago -- indeed, is *less* usable because of the bloat that has been put on top of what was a clean fast simple and well-integrated user interface. KDE has become a 1959 Cadillac, with fins the size of a aircraft tailfin and a thousand pounds of chrome weighting it down as it staggers down the highway like a bloated whale. Windows 7 suffers from some similar user interface bloat, but it has an excuse -- it's Windows. KDE has no such excuse.

So for the meantime, here's what I say: If you want a coherent, usable user environment, buy a Mac. If you want a fast server environment, use Linux. And if you want something that's neither as coherent as a Mac or as good as a server as Linux, then run Windows 7, which gives you a mediocre implementation of both worlds. As for me, I'm going to stick with my Mac on the desktop, and continue using Linux on my server. That gives me the best user environment *AND* the best server environment... but not, alas, in the same box. So it goes.

-ELG

SaaS and the Dot-com Set

2010-04-05T10:58:00.000-07:00

One of the hilarious things that has come about with cloud computing and the advent of large scale SaaS is that I'm seeing the same kinds of arguments I saw back in the dot-com days, that this is a fundamentally different business model that doesn't obey the same rules as traditional software development. Which, of course, is utter nonsense. The method of delivery to customers has changed, but customers remain customers.

My response to all this:

1. Our primary requirement is to meet the needs of the customer. Some customers have legal requirements which preclude SaaS in the sense of SaaS in the cloud, but still wish to have the advantages of SaaS. Think doctors, schools, etc. -- it is actually illegal for them to host patient / student data outside of their own facilities on shared servers. But if we can give them the benefits of SaaS inside their facility using the same software that we have deployed into the cloud -- i.e., *not* a separate version of the software -- then we've fulfilled their needs without any (zero) additional development overhead. And BTW, just to counter one of Marko's points, anybody who structures their sales commissions structure to reward selling private rather than SaaS in the cloud is an idiot and deserves to fail, cloud is generally *much* less support on our part, it just doesn't meet the needs of certain customers.

2. I have not encountered any customers who want rapid updates of critical applications, ever. My boss once ordered me to deploy a new update to a school scheduling program in the middle of a school year. I pointed out that a) school secretaries and counsellors were currently doing mid-term schedule changes, b) school secretaries and counsellors had not been trained on the changes, which were significant (I had re-written the entire scheduling system from scratch going back to first principles, because the old system was incapable of handling some of the new scheduling paradigms that had come out, such as multi-shift scheduling and quarter-system scheduling), and thus c) it would be a fiasco. He said the old system was broken, so deploy the new one anyhow. I did. And got to say "I told you so" to my boss when it turned into the fiasco I'd predicted. My point: Users are fond of *saying* they want the latest, greatest features, but what they actually want is to get their job done. Paying attention to what users say, as vs. what they actually want, can be a huge mistake costing you a lot of money in additional support costs and losing a lot of customer goodwill. Not only were our support lines clogged solid for a week, my boss had to eat a lot of crow at the next user group meeting to get some of that lost goodwill back.

3. I am on Twitter. Yes, customers will Tweet stuff. 140 characters doesn't exactly get you in-depth commentary though. If you let Twitter guide your product development, what you get is a product designed by tweets, which is indistinguishable from a product designed by twits. I have only met one customer in my entire life who actually knew what he wanted (a school discipline coordinator who said, "I want a computerized version of these three state-mandated forms, and reports that fulfill the requirements of these four legally-required forms that I must submit at the end of the school year"). The rest have some vague idea, but you must get with them and engage them in a lengthy discussion complete with design proposals that include sample screen displays of what the application might look like. For one clustered storage product I actually spent more time talking to potential customers, writing proposals, and getting feedback than I spent implementing the actual product. Needless to say, that is *not* a process that occurs in 140 characters.

4. Anybody who goes into a business where there is an entrenched incumbent expecting to compete on features is an idiot in the first place. The incumbent has basically infinite resources at his disposal compared to you and is capable of implementing far more features than any newcomer. He will simply steal any features you innovate in order to stay out ahead. In the old days incumbents like IBM weren't capable of innovating rapidly. But this is the Internet era, and the successful giants have become much more nimble. If there's a feature you have that the incumbent doesn't have, expect him to have it soon. The way to win today is to change the game -- to do something so novel, so different in a fundamental way, that the incumbent could not match you without re-writing his entire product from scratch and ditching his entire current customer base. In short, competing based on features is a fool's game in today's era unless you're the incumbent. The way to win is to change the paradigm, not attempt to compete on features within an existing one.

5. Yes, selling "private SaaS" means we basically end up having to support multiple versions. But that is true regardless, unless we want to force customers into a death march to new versions. Some customers are comfortable with that, the majority, however, arrive at a version they like and just want to stick with that, much as the majority of Windows users are still using Windows XP, or the majority of Linux users are using Red Hat Enterprise Linux 5 (basically a three-year-old version of Linux) rather than the latest and greatest Fedora or Ubuntu. They'll accept security fixes, but that's it -- if you attempt to death march them, they'll go to a competitor who won't.

I've been dealing with satisfying customer requirements for around 15 years now, and my actual experience of those 15 years is that customers are ornery folks where what they say and what they actually want are two different things, and your job as an architect and designer is to try to suss out what they actually want, which is often *not* the same as what they say. Young engineers tend to take customers at face value, then not understand why the customer rejects the result as not meeting his needs. I get called conservative sometimes because I call for slowing down release cycles, maintaining backward compatibility wherever feasible, extensive trials with customers prior to product release, etc., but my experience is that this is what customers want -- as vs. what they say they want, which is something else entirely.

For the record - my phone is an iPhone, and the only paper maps in my car are detailed 1:24000 USGS topographical maps not available on current GPS units with reasonable screen sizes. Just sayin' ;).

-EG

Cryptography engineering

2010-03-29T11:52:00.001-07:00

A new post at the VPEP blog.

I'm currently reading Bruce Schneier's new book Cryptography Engineering (actually the second edition of Applied Cryptography), and the above was just me riffing on some thoughts I had while reading the first chapter.

-EG

About work...

2010-03-22T10:00:00.000-07:00

One thing you'll notice, reading this blog, is that I haven't blogged about anything happening at work. There's a reason for that: It is, in general, a bad idea. If an employer believes a post puts the company in a bad light or simply decides that you have leaked proprietary information without permission, it's a great way to get fired -- dozens of bloggers have been fired over the past decade for posting about things that happened at work.

So anyhow, who I work for is no secret -- you can click on my LinkedIn profile and see -- but now I will be blogging about things I'm doing at work on my employer's own group blog. My first posts are up. You might recognize one of them as a revised version of one of the posts on this blog, except now I can say what I could only hint at then :).

-ELG

Standards and rent seeking behavior

2010-02-23T08:31:00.000-08:00

Rent-seeking behavior is defined by economists as behavior intending to gain competitive advantage by manipulating the environment to your benefit, rather than through profiting from production of goods and services. An example of this would be if a company X managed to get a law passed specifying that all goods purchased by the government must comply with ISO standard 3.052345.32431, where company X happens to hold a critical patent on the technology in that ISO standard. In this case company X is profiting not because they produced goods and services, but, rather, because they manipulated the environment (got a law passed) which says that everybody wanting to do business with the government must pay rent (patent fees) to company X.

William Vambenepe complains that cloud standards are being created in a secretive manner. He complains that this means that those of us actually implementing cloud computing software are being locked out of the process. And this is true. Yet this is not unusual. Why? Well, because there are certain large corporations who, for some reason, still believe that rent-seeking behavior is useful when it comes to the standards process -- i.e., that, as with creating a law dictating that everybody pay rent to them, that they can set a standard that dictates that everybody pays rent to them.

Let me explain: The more complex the standard (and the more BigCorp-patented technologies included in it of course!), the more resources it will take to fully implement it. The goal is to make the resources and patent licenses needed to fully implement the standard so onerously huge that only large organizations will have the resources to do so, meaning they are the only ones who are “standards-compliant” and they can slam any potential upstart competitors as not being “standards-compliant”. Not going to name names here, but I’ll just point out that simpler standards tend to drive out the more complex standards, thereby leaving the big companies high and dry with a product that nobody wants to buy. Has anybody here used the complex X.25 protocol lately? What, you’re using the simpler TCP/IP protocol instead? Exactly.

Which points out why rent-seeking behavior is invariably self-defeating when it comes to standards. Unlike compliance with the law, compliance with standards is generally voluntary. If a standard is too complex or too expensive to implement, people simply won't use it, and a “standard” that nobody uses — or that only customers of a few large corporations use — is hardly a real standard. And keeping the standards discussions secretive is hardly in the best interests of anybody also, it means that real problems with “standards” will be overlooked until the “standard” is actually published, at which point all the effort used to produce the “standard” is useless because nobody will create products that implement the “standard” (thus rendering it *not* a standard). Yet we still see this sort of rent-seeking behavior on the part of certain large corporations that seem convinced that it actually works. Inexplicable…

-ELG

Is there such a thing as "open source management"?

2010-02-19T17:12:00.000-08:00

The Open Source advocates have been talking about how we could apply "open source management" to things other than open source projects. But the question is, does such a thing exist?And my answer is... no. Open Source projects which do not have strong leadership fail. Commercial projects which do not have strong leadership fail. There is no "open source management" in the end, because people are people and software is software. I've been in both situations -- Open Source and commercial -- and software development is software development, in the end.

Any successful software development project of any scale other than a one-off one-person utility has some sort of leadership hierarchy where various people are in charge of various parts of the project and where there's a mechanism to insure that only high-quality code that complies with the general architectural vision of the project makes it into the project. Projects which do not develop this sort of leadership hierarchy fail -- they devolve into squabbles, or their architecture degenerates into such a mess that the project can't be successfully completed without a total re-write from scratch and a reboot. And if the quality of the people who make it into positions of being in charge is low, the project fails too, because the code base turns into a mess of buffer overflows, memory leaks, and unreadable/unfixable spaghetti code and the Object Hierarchy From Heck (the one that has 20 different levels of inheritance to do the simplest tasks, each of which reaches into the internals of its parent class to tweak something or another that it shouldn't be tweaking).

Whether you call these people "managers", "gatekeepers", "leaders", whatever, software development is software development and leadership is leadership. If you have good leadership, your project succeeds. If you don't, it fails, or is so late to market and such a low-performing mess that you might as well don't bother. That's how it's always been, whether Open Source or commercial is irrelevant. The only real difference is that Open Source contributors won't put up with pure BS as is typical in huge corporations. But that sort of BS is not typical in the small startup environment either, which shares a lot in common with Open Source.

-ELG

And now for a photo of the Linux Penguin Command and Control Center...

Where have all the legends gone?

2010-02-16T21:59:00.001-08:00

Back in the early days of the Linux industry, it was pretty easy to know everybody who was everybody. I remember going to the Linux Expo in North Carolina in May 1998, the last year it was run by Red Hat. It was in a small upstairs part of the Student Union of one of the local universities, if I recall correctly, and there were a small handful of vendors selling software and science fiction books. Ted Tso and Linus and Alan Cox and a few others were sitting on a hassock in the lobby outside the conference rooms talking about the ext3 filesystem and how to improve the block cache for the Linux 2.2 kernel, which was scheduled for release shortly, and Richard Stallman... ah, Richard Stallman. He was... RMS. Complete with his saint outfit -- sandals, robe, and halo. Everybody avoided him as if he had fleas in his beard or something. For all I know, he may very well have. Then there was the final keynote, in an auditorium-style classroom (or is that a classroom-style auditorium?), where Linus walks out onto the stage and announced, "I am Linus Torvalds, and I am your god." The audience applauded wildly. Because we were all geeks there, and we got the joke. Today... today, I suspect Linus would get boos from humorless boobs if he tried something like that. Things change, and sometimes not for the better.

Of the vendors who were there -- The Linux Mall (I have one of the very first plush penguins!), Linux Hardware Solutions, Enhanced Software Technologies, VA Linux, DEC, etc. -- very few are still around. That fall Linux was discovered by the big guys -- IBM, Oracle, and so forth -- and everything changed forever. The Atlanta Linux Showcase in October 1998 was a zoo. Comdex in November 1998 was even more of a zoo. The big guys were moving in, and the small cottage industry that was the Linux industry was about to change forever. A few of the little guys survived, but most didn't -- they didn't have the mentality to do what it took to go big, and going big -- going for venture capital, going for IPO, going for a big business model that would have competed with the big guys -- was the only way they could survive without running into a cashflow crunch or into a hard ceiling on what they could do. It simply was not in the skill set of the small cottage industry guys, that wasn't what they did, they ran a conservatively managed business out of a small office-warehouse somewhere, they didn't try to build a huge empire. Unfortunately, cottage industry could not compete. By the end of 2001, most of them were gone, memories the only thing left.

I know what happened to a lot of the people, they have largely moved into other industries or are serving as consultants, marketing managers, or similar for other businesses -- but of the technical people, it's interesting that most of us are still around somewhere hacking on Linux. As for me, I talked to the owner of Linux Hardware Solutions there at Linux Expo and decided to move to North Carolina to work for them. It was a gamble, but it was a gamble that produced enough connections that when LHS folded I could move on elsewhere in the Linux industry into a software engineering role that eventually resulted in my first team leadership role. I spent some time as a nomadic Linux penguin. Now I'm not so nomadic -- I've lived in the same apartment for close to six years, for cryin' out loud -- but sometimes I think back to those early days of the Linux industry, and wonder if we haven't lost something since then, some excitement, some energy, some sheer exuberance that made it a joy to get it up in the morning to create something new and unheard of that had never before existed in the world. Today we're all stuffy professionals, spending our time writing design documents and doing design reviews and managing engineering teams. Then... then we were changing the world. And we did.

-ELG

iPad insanity

2010-02-14T19:41:00.000-08:00

I suppose that, like every other geek on the planet, I have to talk about the iPad. Okay. I'm underwhelmed. Good enough?

The iPad has two problems: 1) It's too big, I can't fit it into the side pouch of my laptop case or of my carry-on bag. 2) The 10" LCD uses too much power. My little Sony PRS-300 uses e-Ink, which uses power only when you're flipping from one page to the next and allows a 2-week nominal battery life (1 week in actual heavy use). The iPad nominally has a 10 hour battery life, but my experience with my Macbook Pro, which nominally has a 7 hour battery life, says that the iPad will actually have a 7 hour battery life in real-life use -- the nominal battery life is if you have the backlighting turned down to pretty much unreadable levels, and I'm not as young as I was 14 years ago when I was releasing my first Linux product, I can't read things that are so dim anymore.

So the iPad, to me, is like the Apple TV -- a so-so product that might at some point in the future become useful, but right now is a "so what?". I normally carry my Macbook Pro and my iPhone with me when I travel, and maybe the Sony Reader tucked in the side pocket of my laptop case if it's going to be a long trip just so that I don't run out of books to read (hauling that many paper books can get problematic). I just can't see the point of the iPad, at least for me. Maybe if it were an iTampon, a smaller handier more convenient device similar in size to the Kindle, I could "get it", but for now? Too big, too power hungry, too inconvenient.

--ELG

Convicted monopolist wants RealID for Internet

2010-02-08T14:38:00.000-08:00

Yes indeed, Craig Mundie, head of Microsoft Research, wants a "driver's license" for the Internet.

This is the craziest thing I ever heard of. It's estimated that 20% of drivers here in the state of California don't have a driver's license. Despite that, they still drive, even though if they're caught it means their car would be impounded and they'd face possible jail time. But people who are here illegally, have had driving rights removed for legal reasons such as DWI, or whatever, a lot of them choose to drive anyhow despite those risks.

There are no traffic cops on the Internet to stop you and seize your computer if you don't have the proper Internet driver's license, so how the heck would any such thing be enforced? I mean, it's as if Craig Mundie never heard of the fact that Social Security cards, birth certificates, and driver's licenses are regularly forged.

There are a lot of things that can be done to improve Internet security, such as cutting off any ISP that refuses to deal with spam or DOS attacks emanating from its IP addresses. But this? This is a non-starter. Apparently Mr. Mundie still hasn't figured out that not everybody is a law-abiding Microsoftie like he is, and that people will violate the law if they feel that it is in their best interests to do so. I mean, all he has to do is look at his own employer, for cryin' out loud! Where there is a will, there is a way -- even if it requires forgery. If Mr. Mundie's proposal were adopted, the only people who would comply would be law-abiding people, which sort of renders the whole thing useless, don't you think?

-ELG

The problem with eBooks

2010-01-01T16:57:00.000-08:00

I've been looking at ebook readers lately -- the Sony Reader, Amazon Kindle, and Barnes & Noble Nook being the three that I looked at. All three do a good job from a hardware perspective. E-paper is quite acceptable for reading text novels and you can store enough novels on one of these things for a whole year, saving a whole lot of dead trees in the process. And the one-to-two-week battery life on these things is adequate for all purposes short of hiking the Appalachian Trail, and even there it would work as long as you brought along a USB solar charger. These e-paper devices with their relatively large high-contrast screens put reading eBooks on the small screen of an iPhone to shame, and even the lowest-power-use laptops won't get much further than five hours before they go dark, much less one to two weeks. Most of this can be attributed to the e-paper screen, whose large size, high contrast, and lack of power usage (the only time they use power is when you flip the page, static displays use no power) allows a much more pleasant experience than any prior electronic reading device.

Yet... yet. E-book uptake is ridiculously small. Even the Kindle has probably not sold more than 400,000 copies despite the fact that every time you go to the Amazon.com site the thing is hyped to you. So what, exactly, is the problem?

Well, the problem is simple: A lack of books, compounded by a) DRM, and b) the outrageous costs for the ebooks, costs that in many cases are higher than the costs for equivalent paperback books from your local bookstore. For example, Sony's ebook chief claims it's impossible to make money selling ebooks for $9.99. WTF? Bookstores can sell paperbacks for $7.99, despite the huge overhead of having a physical plant and employees and shipping charges to get the books there and etc., yet E-vendors can't make money at $9.99?!

The big publishers, to me, seem to have their heads stuck up their a$$'es in much the same way that the big music studios had their heads stuck up their a$$'es about digital music until Steve Jobs finally brow-beat them into submission to get them to sell songs for 99 cents without DRM. The various ebook vendors all have incompatible ebook formats, sell only a small smattering of publishers' catalogs in e-format, and sell them for prices higher than the paper version of the publication. And, they have them DRM-protected. Meaning that five years from now, they're worthless, because the device you installed them on will have broken and you won't be able to read them anymore.

I have files on my computer that are 20 years old. That's because I never used proprietary formats (or at least used proprietary formats that were so common that they were easily translated into non-proprietary formats), and transferred the files from older computer to newer computer every time I upgraded using open protocols such as xModem over RS-232 in the early days, and FTP and SCP over TCP/IP networks nowadays. Similarly, I have paper books in my storage room that are 30 years old that I can read just fine today. I did not buy any songs from the iTunes store until they became DRM-free because having songs in a proprietary protocol tied to a specific computer simply was incompatible with my experience in technology, which said that the songs would become unlistenable within five years if I did that. It's like having a 9 track tape from 1982. How are you going to read the data off of it? Where are you going to find the device? If it's in some proprietary format, how are you going to convert it into some modern format? Digging through my storage I came across a couple of 9-track tapes from the mid 1980's, I think they had early copies of GNU Emacs on them to be read into our VAX minicomputer at college. At least they're (probably) in tar or cpio format, but how in the world could I possibly read them when the hardware to read them is long gone and obsolete?

Yet that is what the eBook vendors want to lock us into -- having to re-buy our entire library every three to five years when proprietary eBook formats become obsoleted and the devices they were installed upon die the normal death of consumer electronic devices (i.e., 3 to 5 years, then bam, one day they just don't wake up again, maybe a battery wore out and you can't get a replacement battery because the vendor doesn't make them anymore or because the battery is more expensive than a new device, maybe a capacitor in a critical path reached its end-of-life, but it's dead, Jim). It is laughable, and it's no wonder that only a few early adopters are buying these things right now despite the great hardware that's come out recently, even though the DRM put on these early e-books is itself laughable (all -- ALL -- major e-book DRM formats have been cracked, even the Kindle one, copy protection works no better today than it worked in 1985 when we were cracking the protection on Commodore 64 games, all it does is annoy people while not impairing the pirates at all). Because, as with computers, hardware is only half the equation. If you don't have content -- if you can't get books for it for an affordable price, and more importantly, can't get books that will last longer than the device they are installed upon, what's the point?

-E

ACL management on MacOS Snow Leopard

2009-11-07T21:10:00.000-08:00

So I was going to follow the directions at this hint site to prevent Time Machine from doing a full backup again once I updated my MacBook Pro to a bigger drive. After all, I don't want to re-backup the stuff I just restored from my backup! But my attempt slammed to a halt after I type 'fsaclctl' and... uhm... WTF? It isn't in Snow Leopard! And by the time you get to userland the permission to override a "Deny All to All" ACL is dropped even if you su to root... you just can't get there from here unless you can somehow turn off ACL support for the whole filesystem!

Ah, but never fear, the Leopard version of fsaclctl works just fine on Snow Leopard. The question is, which of my half dozen backup drives up in the storage closet or offsite is old enough to have Leopard on it? I was about to get up and go grab one, when I glanced down and... there was the Mac OS Leopard 10.5.2 install DVD, right there, in the pile of disks I'd used to re-image the Mac.

So first thing to do was drill down and find the package. The packages live in '/Volumes/Mac OS X Install DVD/System/Installation/Packages' and the easiest thing to do is 'go to folder' from the Finder 'Go' menu to go there. Then by dragging dropping packages onto the /Developer/Applications/Utilities/PackageMaker utility, I discovered that fsaclctl lives in package "BSD.pkg" in directory /usr/sbin.

The next question is, how do we get the file out of the package? I couldn't drag it out of PackageMaker, PackageMaker simply refused to do so. So I grabbed a utility called 'Pacifist'. I won't claim it's the best utility for doing this because it's simply the first one that came up when I googled, but it allowed me to drop the BSD.pkg onto it, drill down to the file, then drag the file out to a folder on my desktop, from whence I could then put it into ~/bin and use it.

Now, this isn't about the Time Machine hack (BTW, it didn't work -- apparently Time Machine's implementation has changed since Leopard), but, rather, about security. Some folks wonder why MacOS is more secure than Windows. This experience gives you one clue why. There are things you cannot override even if you have full administrative access, once permissions are dropped during the boot process. I suspect that in future releases of Snow Leopard will remove the low-level ioctl that fsaclctl relies on, further securing the system. But it's clear that while Apple doesn't make splashy announcements about security and doesn't have some of the bells and whistles like address space randomization, they're doing some things quite right in the background to keep things secure.

-EG

Parallels 5 vs. VMware Fusion 3

2009-11-06T00:07:00.000-08:00

So I have tried both of these virtualization solutions for MacOS Snow Leopard and the winner is... VMware by a landslide. Not because of performance. VMware's performance is acceptable for my purposes but I can definitely tell that I'm running in a virtualized environment. But, rather, because VMware WORKS, and Parallels doesn't. That's the bottom line. I can go into more detail, but I'm just too frustrated with Parallels right now and would use language not appropriate for polite conversation. Having Parallels crash my computer *TWICE*, and lock up three different times, simply does not make me happy.

I am saddened to say this, because I've owned Parallels since version 2.0, but this is it. This is the end. They are not getting any more money from me. Each new release of Parallels they promise that they got it right this time. Each time, they break things badly -- for example, in Parallels 4, one of my mapping programs ended up going BLAMMO unless I turned off mouse pointer acceleration in the Windows control panel, and then the Parallels device driver simply refused to display any mouse pointer at all. Meanwhile VMware Fusion 3 is a rock solid product. It might be slightly slower than Parallels on some benchmarks (hard to tell, I could never keep Parallels running long enough to run the benchmarks I was wanting to run), but it *works*, and the integration between Windows and MacOS Snow Leopard is quite good, no problems with cut-and-paste or sharing files between Windows and MacOS or anything like that. The competition between VMware and Parallels is over, and Parallels is done. Finished. Kaput. They had first mover advantage, and like Netscape with web browsers, simply failed to execute.

Which reminds me of the time that my manager was the guy who ran Netscape's development process into the dirt. Needless to say the common Linux fanboy notion that Microsoft ran Netscape out of business is utter nonsense -- Netscape's browser technology disintegrated without any help from Microsoft at all. Their technology simply disintegrated under the weight of too many idiotic false deadlines and hacks, and the manager who did that then did the same thing for my then-employer's development process. But that's another ugly tale that tends to evoke unwise language so I'll do something a bit more abstract about deadlines and why they're both useful and, in some cases, toxic.

-EG

Numbers from Windows Experience quickie benchmark:

VMware 3:

Processor: 5.9
Memory: 3.9
Graphics: 2.9
Gaming graphics: 3.4
Primary hard disk: 6.3

Parallels 5:

Processor: 4.5
Memory: 3.9
Graphics: 2.9
Gaming graphics: 4.1
Primary hard disk: 5.9

Parallels has somewhat better 3D performance, somewhat poorer performance on processor and hard drive tests, same as VMware elsewhere. Parallels is probably better if you want to play games, but that's why Boot Camp was invented...

The Windows 7 'reg' command

2009-11-02T19:11:00.000-08:00

So I had a problem. I had a Topo 8 install on my old XP hard drive and wanted to transfer it to my new Windows 7 machine. No problem, just re-install, right? Well, that would be a problem alright, because my activation key for Delorme Netlink would not work in the new install -- Delorme links it to a single installation of Topo USA. Note that the licensing for Netlink allows me to run it on any computer that I own as long as it's just one computer at a time (i.e. I can't have it installed on more than two computers and can only use it on one computer at a time), but the actual implementation is similar to the lamentable Windows Activation in that it often disallows things that are allowed under your licensing agreement, requiring you to call in to a support center and have a database entry adjusted at the other end to allow activation.

So now let's talk about what actually happens on modern versions of Windows when you install a program. Things get placed into basically four areas:

Start Menu folder -- usually a folder is created here with a new Shortcut to the application plus utilities. The location of the Start Menu folder differs wildly between Windows XP and Windows 7, but it's easy to find.
Program Files -- usually a folder with the program and all its data is created here.
Windows -- Any driver bundles are plopped into the appropriate folders here, as is installer/uninstaller info.
Registry -- Configuration data and component registration.

Of these, the first three are easy to copy from one computer to another. But the registry entries... ah yes, now that is a problem!

The fundamental problem is that the registry is a database, and thus you can't simply use drag and drop to move entries from point A to point B, unlike with MacOS where you could just copy the appropriate directory from the old /System/Library/xxx to the new /System/Library/xxx and/or the old ~/Library/xxx to the new ~/Library/xxx to move the configuration data, or Linux where you could just copy the appropriate directory from the old /etc/xxx to the new /etc/xxx to move the configuration data. You have to use database tools, and the Windows database tools for accessing the registry are crude and primitive compared to the tools available for accessing file data. This is especially true for the 'regedit' GUI command which is utterly incapable of copying registry from place A in the registry tree to place B in the registry tree. But never fear: This is a capability that the command line 'reg' command has, and we're going to use it.

The first thing to do is to mount the old hard drive as your "D:" drive. Make sure you've added the 'Run' menu option to your Start Menu with the appropriate control panel entry (sorry, you've already seen my opinion of the Windows control panel, it's in there *somewhere* but you'll have to do like me and just dig until you find it!). Select 'Run' from your start menu, and go into 'Regedit'.

The next thing you'll need to do after that is import the HKLM hive into your registry. Click on the HKEY_LOCAL_MACHINE entry and select File->Load Hive. Browse over to D:\Windows\System32\config and select 'software' as the hive to import. Then give it a name, like OLD_SOFTWARE. Once you finish doing this, you'll find that OLD_SOFTWARE is now in your registry tree. You can now exit regedit, because regedit has no (zero) ability to copy subtrees from one place to another in your registry tree.

Now you'll need an administrative mode command prompt in order to operate. Now, I'm going to assume you have some basic Unix-compatible command line tools available using Cygwin or by copying files to MacOS or Linux via a network file share then executing Unix commands there, simply because there are no native Windows tools which will do the same command line parsing in as easy a manner, but it COULD be done with VBscript. It'd just be a lot more coding to make it work.

So: now that we have a command line,let's query out all the Delorme keys:

reg query HKLM\OLD_SOFTWARE /s /f delorme >\Delorme_keys

Then copy Delorme_keys someplace where you can run Unix commands on it:

grep "^HKEY" Delorme_keys >Delorme_keys2
vi Delorme_keys2

Take a look at those keys, and at the original file too, to see which ones you want. In general you will not want to completely replace the contents of all keys that have some data item related to your application, you'll want the Classes and any software-specific key. So I edited Delorme_keys2 to have the keys I wanted to copy from the old install, then:

awk ' { t=$0; sub("OLD_","",$0) ; printf("reg copy \"%s\" \"%s\" /s\n", t,$0); } ' Delorme_keys2 >DelormeRegCopy.bat

This gives me a file that has lines in it that look like this:

reg copy "HKEY_LOCAL_MACHINE\OLD_SOFTWARE\Classes\CLSID\{20016EDD-4CB6-11D3-A3FA-0000C0506658}" "HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{20016EDD-4CB6-11D3-A3FA-0000C0506658}" /s

The 'reg copy' command will copy both the key (and its data items) and any subkeys to the new location, assuming you provide the /s flag. Then I copy this .bat file back to the Windows 7 system, and from the Command prompt type "delormeregcopy.bat" and voila! Now my copy of the Delorme application works, Netlink works, and I can then re-install Topo 8 on top of this install to "repair" it (i.e., put the installer and driver bundles in the right place and verify that everything is registered properly) and the installation keys will still be there to keep my Netlink operational.

Note: As always, back up your registry before mucking around in it. And be *very* careful with any keys you copy in, I examined the contents of each key using regedit before I allowed it to stay in my final Delorme_keys2 file. The above is NOT the full directions for how to do this specific task, simply because I do not wish to enable software piracy, but, rather, an example of how to use the 'reg' command to copy critical registry entries from an old installation into a new installation. And the usual disclaimer "this might destroy your system!" applies.

Now: I could go off on a rant about how stupid the Windows registry is, how the tools to manipulate it are primitive and far inferior to the tools available to manipulate text files, blah blah blah, but we already all know about that. The "reg" command at least gives us some of the missing functionality that regedit doesn't have, even if it requires typing obscure commands at a command prompt. But then, "obscure" and "Windows Registry" do go together like "macaroni" and "cheese", eh?

-- EG

The Windows 7 hoopla

2009-11-01T07:56:00.000-08:00

So is Windows 7 a Mac killer? Or is Windows 7 lipstick on a pig? The answer is "No."

Let's look at the first one first. Windows 95 in many ways introduced "the" Windows user experience. It was a clean, reasonably logical user interface that was surprisingly good from a user interface perspective considering the limitations of the underlying platform, limitations which were necessitated by the limitations of the underlying hardware and the need for DOS compatibility until Windows-specific software arrived. It was Windows 95 that I evaluated, then went to my boss and said, "This is going to be big. We need to figure out some way to make money with it." That was a few months before a customer brought Linux to our attention (and my reaction to that later -- it was not favorable, initially), but certainly I wasn't wrong when I said that to my boss.

It's been all downhill since from a user interface standpoint, with each new release of Windows having yet more useless folderol to waste resources and confuse customers but no fundamental change in the UI. Windows 7 continues that tradition, adding lipstick to the pig that has become Microsoft's overly complex user interface by re-naming some things, changing text to icons on the menu bar, and somehow managing to make the Control Panel even more complex than it already was. People who claim Windows 7 could somehow be a "Mac Killer" are being ridiculous. Changing the text on the menu bar to icons does not make it a dock, and Windows 7 is even more confusing to set up and configure than its predecessors were if you're trying to integrate it into an already-existing network. I clicked away in the control panel for quite some time before finally typing "change workgroup" into the search bar. That took me to a place where I could change the workgroup (so it matched my home and office workgroup name so my systems would appear in the network browser), but where is that located in the morass that is the Windows 7 control panel? I have absolutely no idea, I clicked into the logical place and it changed my workgroup to "WORKGROUP", which isn't what I wanted at all.

Meantime, click on the open-apple icon and select 'System Preferences'. There's two possible places where you could set the workgroup -- 'Sharing', or 'Network'. I clicked on 'Sharing' and didn't find it, so I clicked on 'Network', there's a button 'Advanced', I clicked on 'Advanced', saw the word 'WINS', and yep, there's my NetBIOS name and workgroup name. Three clicks once I got the Mac "control panel" up - Network, Advanced, WINS -- to get me where I needed to be.

So from a user interface perspective, Windows 7 definitely is lipstick on a pig. It's just a bunch of lipstick on top of the original Windows 95 user interface, and like a toddler messing with mommy's lipsticks, the results are not all that great from a usability perspective. Frankly, I prefer the original, which was fast, clean, useful. However, that's not the important changes that have been made to Windows 7. The important changes are under the hood. Windows 7, in my test, used approximately 3GB more disk space than Windows XP -- i.e., around 8GB rather than 5GB. Its memory usage for snappy performance is approximately 256MB more than Windows XP (around 756M vs. 512M) if you disable Aero by switching to a 'Basic' theme, and since Aero is just lipstick, that's no big deal. In exchange you get a more secure operating system that has built-in functionality that Windows XP lacks, such as the ability to record a DVD. I have not tested Windows 7 on a netbook yet, but I'm not seeing any reason why it wouldn't work -- even with Microsoft Office installed and various third-party Internet software (Firefox, Safari, Flash, etc.) I'm using only 14GB of disk space for my Windows 7 system, and even low-end netbooks come with 32GB SSD drives and 1GB of memory today.

So from that perspective, Windows 7 accomplishes what Microsoft wanted it to do -- it allows them to discontinue support for Windows XP because it will run pretty much everywhere that XP is currently required due to the resource usage of Vista. It also accomplishes what most IT people want -- a more secure operating system that won't require them to spend half their time cleaning up after virus outbreaks, and which allows them to standardize on *one* operating system, rather than having a mismash of various versions of Windows. On the other hand, it's pretty clear that Microsoft needs more than lipstick on a pig to clean up their user interface. They need a few iFools to lead the charge against useless UI complexity, including at least one iFool who has the status in the corporation to push back against the marketing droids and geeks who always want one...more...feature... to never be used by actual customers, but look good on a marketing flyer or looks, like, really rad, dude. I wish them luck, because after fifteen years of putting lipstick on a pig, there's almost more lipstick than pig insofar as the Windows UI is concerned.

-- EG

The smartphone maze

2009-10-28T20:39:00.000-07:00

Much has been made of recent improvements in Google Android phone sales. Android phones are now available (or will be available by November 1) on all major U.S. carriers except AT&T, and many carriers will have multiple Android phones. There are some who say that this will doom the Palm Pre, which along with the iPhone has the slickest user interface of all the various smartphones out there. But my own analysis is that this isn't so: The smartphone OS that Android is supplanting is not RIM's or Palm's, but, rather, Windows Mobile.

It is little secret that the development of the next generation of Windows Mobile is a disaster. Windows Mobile 6.5 has been announced for the end of this year to collective yawns -- nobody thinks anybody is going to actually ship a phone based on it. Windows Mobile 7 has been announced for next year to barely concealed guffaws. Nobody who is serious expects a viable Windows Mobile 7 to come out anytime before the end of next year. What has happened, during this era of stagnation in Windows Mobile, is that WM vendors are now migrating to Android for their new smartphones. Android supports the new features of the new smartphone hardware, while Windows Mobile doesn't. And while Android is a user interface disaster, so is Windows Mobile -- both systems embody pre-iPhone paradigms of how to do things where each application has its own unique user interface, as vs. the new multi-touch common-user-interface paradigm where all user interface coding must go through a library that enforces a common look and feel. In short, where geeks used to go to WM because it was a (relatively) open platform with a lot of capabilities such as multi-tasking that the competition did not have, now they're going to Android instead because it has those same attributes but supports newer hardware.

So what seems to be falling out of all this is that Windows Mobile is going to go the way of old-school PalmOS shortly. The current vendors of WM phones such as HTC appear to be engaged in a mass migration to Android. But this does not mean that sales of the iPhone and Palm Pre will be hurt by Android. They are simply different markets -- Android, due to its fundamental design and development processes, will simply never be able to match Apple or Palm on ease of use or consistency of user interface between applications. Like Windows Mobile, Android is a geek product. Plenty of geeks will likely end up migrating to Android, but there is a huge market for smartphones as people max out the capabilities of standard candybar/flip phones between Twittering and everything else they want to do with phones, and most of these people are not geeks. Vendors like Apple and Palm are well positioned to go after that market... but Android simply doesn't play there, any more than RIM does with their crackberries.

-E

People are NOT fungible

2009-10-26T21:47:00.000-07:00

One of the things that happened during the transformation from being "employees" to being "human resources" is that large corporations apparently decided that employees are fungible. That is, if you have two employees, employee A and employee B, and employee A is making a lot more money than employee B, it's fine to just drive off employee A and replace his position with employee B then hire a contractor for even less to fill employee B's position. Hey, an employee is an employee, right? Interchangeable, just like cogs, eh?

Much has been said about Microsoft's T-Mobile Sidekick disaster and what that says about the notion of "cloud computing" (hint: As I said earlier, cloud computing does not eliminate normal IT tasks other than actual hardware maintenance). But it says even more about the whole concept of "human resources". The infrastructure that Microsoft purchased with Danger included Oracle databases, Sun servers, and a set of non-Microsft NAS or SAN systems. None of these are things that Microsoft has experience with. The current hypothesis is that Microsoft contracted a contractor to do an Oracle database upgrade, and the contractor did exactly that, and Oracle -- as it often does -- ate its database during the upgrade. This was compounded by, apparently, the database backups being unreadable by the new version of the database. All of this is remediable if you have sufficient Oracle expertise on staff, but apparently neither Microsoft nor their contractor had such expertise -- they'd all left Danger after the acquisition after being shifted to positions dealing with other technologies that they didn't like or did not have the skills to do successfully.

Lesson for managers: Identify the critical skills that you need in order to continue to have a viable business, and retain those people. It's a lot easier to retain the people you need, than to find new people with those same skills to replace them once they do leave and you discover that suddenly you no longer have a viable enterprise because critical tasks are no longer being done for lack of expertise to do them. Employees are not fungible. You simply cannot replace an Oracle database expert with a contractor hired off the streets or with an expert in Microsoft databases, Oracle databases are black magic and the people who can successfully maintain them are worth every penny you pay them.

Of course, it's easy to throw stones at Microsoft here, but it's not a Microsoft problem. It's an industry-wide problem. Managers industry-wide are failing at the task of properly identifying the skills they need in order to perform the tasks needed to have a viable business, and when the people with critical skills leave are blind-sided. Thus you get disasters like Sprint's Nextel disaster, or this Sidekick disaster, where critical infrastructure people left and the infrastructure fell apart and rendered the enterprise non-viable. Employees are not fungible, and if you fail in your job of identifying the skills needed to keep your business operating and retaining the people with those skills, you may not get the press of the Sidekick disaster, but your business will operate slower, less efficiently, and have difficulty getting product out the door. And especially look at IT and operations people. That's not sexy stuff, but both Sprint/Nextel and Microsoft/Danger show that you simply cannot fire all the operations people you just acquired and replace them with your experienced employees who are experienced with a totally incompatible technology. It doesn't work. It just doesn't. And remedying the disaster that arises after you do this will be far, far more expensive than just retaining those critical infrastructure people in the first place.

-E

SSD in a low-end netbook

2009-10-12T22:28:00.000-07:00

Netbooks tend to live a hard life. They're used in moving cars, they spend a lot of time banging around in backpacks, and so forth. Early netbooks like the Asus eee that practically defined the category used Linux and a small flash memory chip. This dealt quite well with the problem of durability -- flash memory chips don't care about vibration (at least, not about levels of vibration that wouldn't utterly disintegrate the whole computer) and The problem is that people want to use their netbooks to view multimedia content, and Linux is woefully inadequate in that area due to the fact that Linux users today are either utter geeks (parodied in this Xkcd comic) or are using it for servers where multimedia is not an issue (other than serving it via a web server). So netbooks have moved to using Windows XP rather than Linux.

The problem is that Windows XP does not run well off of the slow flash memory chips included with first-generation netbooks, thus netbooks have moved towards the cheapest hard drives available. Unfortunately this brings two problems: 1) those hard drives are still painfully slow compared to current state-of-the-art hard drives, and 2) those hard drives have the same vibration sensitivity and G sensitivity of all hard drives, making them a poor fit for netbooks.

The solution would be a high-speed SSD drive. They perform much better than low-end hard drives, and the only vibration or G forces that could destroy them would turn the entire netbook into a pile of shards. The problem is that SSD's were typically expensive. Until now: 64GB SSD for $150, in this case a Kingston SSDNow V-Series.

64GB doesn't sound like a lot of storage, but I examined the hard drive on my Acer Aspire One netbook and discovered that I was using a whole 20GB of hard drive space. I think my usage of the netbook is probably typical of most people's usage of a netbook -- Internet browsing and light word processing. These aren't computers that you buy to do video processing or music recording, they don't have the CPU horsepower for that, but they're perfectly acceptable for Internet browsing. When I'm bouncing around in my Jeep on field expeditions I don't want to haul around my expensive Macbook Pro, I want something small and durable for doing quick email checks whenever I get near civilization, and the Aspire One suffices for that. Except for the hard drive issue.

Thus I purchased the above SSD and installed it in my Aspire One. I had previously purchased the disk imaging CD/DVD set from Acer to allow re-imaging my netbook when the hard drive failed (note the "when", not "if" -- netbooks live hard lives), and it installed fine onto the SSD. The results have been gratifying. Performance is much better than with the low-end hard drive, and the durability is excellent. The second-generation SSD's have now conquered the stuttering problems that plagued the first-generation SSD's, at least for applications such as netbooks where large writes are rare -- I have never encountered stuttering problems.

What does this mean for the future? It means yet more low-power energy-efficient netbooks, perhaps higher in price than current netbooks but with better durability and performance. Netbooks will be relegated to the long-battery-life small-storage-capacity category rather than being marketed based on low performance and low price. You will start seeing some netbooks in the $700 range, around the same as a "real" notebook, assuming that sufficient performance can be obtained to justify that price. The question is whether Intel will deliberately cripple their Pineview follow-on to the current Atom processors the way they currently cripple it by forcing netbook makers to use the antiquated high-power-use 945 chipset, which has atrocious graphics performance (i.e., cannot even play HD videos from YouTube without stuttering, which is a major problem given that many people buy these things to browse Internet multimedia content). If they do, expect rival chips from AMD and VIA to gain popularity, albeit not with major vendors due to Intel's anti-competitive behavior of charging vendors more for chips if computer vendors use a rival's chips for more than 5% of their shipping computers. Given that there are major markets where Intel's chips are the only available chips, this clearly is going to limit how many jump ship to AMD and VIA. But if Intel can't deliver the performance that people want, somebody will jump ship to AMD or VIA, even if it isn't Dell or HP...

-E

In the Cloud

2009-10-05T21:41:00.000-07:00

Cloud computing. Ah, how the buzzwords love to flock. This is no different from the 1970's when it appeared that the future was going to be large timesharing services. You could deploy your applications in that "cloud" and have redundancy, automatic backups, and so forth without the time and trouble of maintaining your own infrastructure. If you needed more storage an additional DASD in your VM virtual machine could be easily deployed from the "cloud" of storage devices available at the timesharing service, if you needed more CPU your application could be deployed on a VM given access to more of a CPU or even to a whole CPU, and so on and so forth. Large time sharing services with IBM 370's and follow-ons were doing cloud computing before the word existed. There is no (zero) functional difference between an IBM 370 running VM in 1973 and a Dell server running VMware ESXi today, other than the fact that the Dell is much faster and has much larger hard drives of course. But both do the exact same task, and arguably the IBM 370 did it better since the IBM 370 would even let you migrate all processes off of a CPU and take that CPU offline and remove it entirely for service, *with no disruption to running jobs*. Something which, I might add, its descendant IBM mainframes are still capable of doing, and which VMware wishes it could do.

So what happened, you ask? Why did large corporations move to networked microcomputers and their own insourced mainframes, and why did smaller businesses move to microcomputers and servers? Part of the reason was one of data security -- having your data reside with a third party entity that might go out of business at any time was a business risk that was not acceptable. But also, they ran into the same problem that cloud computing runs into when you try to deploy large enterprise databases into the cloud: a lack of I/O bandwidth. We are talking about an era where 300 baud acoustic couplers were high tech, remember, and where the backbones of the primitive data networks ran at 56kbit and operated in block mode. As a result, user interfaces were crude and based around block transfers of screen data, since real-time updates of screen contents based on immediate response to keystrokes was simply impossible. When microcomputers were invented with their megahertz-speed connections to their video displays and keyboards, that made possible entire classes of applications that were simply impossible on the prior timesharing systems, such as spreadsheets and real WSIWYG text editing and word processing.

Pipes to cloud computing facilities today are similarly constrained compared to local pipes. 10 gigabit network backbones are now not unusual at major corporations, yet most ISP connections are going to be DS3's that are operating at 45 megabits per second. It is clear that cloud computing runs into the same communications problem that prior time-sharing operations ran into, except in reverse -- where the problem with the time sharing systems was inadequate bandwidth to the user interface, the problem with cloud computing is inadequate bandwidth to the database. Most major corporations generate gigabytes of data every day. One major producer of graphics cards, for example, has so many NetApp appliances filled with simulation data for their cards that they had to expand their data center twice in the past five years. This is not a problem for a 10 gigabit backbone, but you are not going to move that data into the cloud, you're hard pressed to save it to local servers.

So what makes sense to deploy to the cloud? Well, primarily applications that are Internet-centric and operate upon a limited set of data. A web site for a book store that occasionally makes a query to a back end database server to get current inventory works fine for cloud computing. Presumably the cloud servers are colocated at critical points in the Internet infrastructure so that buyers from around the world can reach your book store and order at any given time, and the data requirements to the back end are modest and, because much of the content is static (War and Peace is always going to have the same ISBN and description for example), much of the data can be cached in those same data centers to reduce bandwidth to the central inventory system. I can imagine that this bookstore might even decide to sell access to their own internally developed system for managing this "cloud" of web application servers to third parties (hmm, I wonder who this bookstore could be? :-). Another possible application is for "bursting" -- where you need to serve a significant number of web pages for only a small part of the year. The Elections Results web site, for example, only gets hammered maybe six times per year, and gets *really* hammered only once every four years (when the Presidential race hits town). It serves a limited amount of data to the general public that is easy to push to data centers and serve from local caches there, and maintaining huge infrastructure that will be used only once every four years makes no sense from a financial point of view. Cloud computing makes a lot of sense there.

But one thing to remember about cloud computing: Even for those applications where it does make sense, it is no panacea. Yes, it removes the necessity to purchase actual hardware servers and find a location for them either in your own data center or in a colo, and provide plumbing for them. But you still have all the OS and software management problems that you have if the servers were local. You still need to deploy an OS and manage it, you still need to deploy software and manage it, you have simplified your life only in that you no longer need to worry about hardware.

At the recent Cloudworld trade show, one of the roundtables made the observation that "the next stage in cloud computing is going to be simplifying deployment of applications into the cloud." I.e., previously it had been about actually creating the cloud infrastructure. Now we have to figure out how to get applications out there and manage them. And past that point I cannot go :-).

-EG

The product cycle does not end at the doors of QA

2009-09-26T22:42:00.000-07:00

Most engineers, I've found, have a very limited view of the product cycle. They get a spec from product marketing. They implement this spec. They hand the product off to QA. The product gets shipped to customers after bugs are fixed. They're done.

The problem is that this limited view of the product cycle utterly ignores the most important question of all for customers, the question that causes the most pain and headache for customers: How will this product be distributed and deployed in the enterprise?

Product marketing's spec doesn't answer this question, usually. For that matter, the customer often has no idea. The end result is that you end up with atrocities like the Windows install cycle, where deploying Windows and Windows applications across your enterprise requires massive manpower and huge amounts of time and trouble.

When you're working on your product's detailed functional specification and architecture, you must be also thinking about how to automate its deployment. You're the one who knows how the technology will work internally. So let's look at the evolution of deployment...

My first contract job after leaving teaching was a school discipline tracking program, necessary in order to meet new federal requirements for tracking disciplinary offenses. Once I had the actual program working, the next question I had was, "how will this be deployed?" I knew this company had over a dozen clients scattered all over the state, each of which had at least five schools. There was no way that we were going to deploy it by hand to all of these places. The program also required a new database table inserted into the database, so you couldn't just place the program into some location and have it work. And the program also required a modification of the main menu file to add the new program to the list of programs. So there was at least three files involved here. My solution, given we had to get this thing deployed rapidly, was to write a shell script (this was on Xenix back in the day), tar it all up, and then give short directions:

Place distribution floppy in drive
Go to Unix shell prompt by logging in as user 'secret' then selecting menu option 2/3/4 .
Type:
- cd /
- tar xvf /dev/floppy
- sh /tmp/install-program.sh
Exit all logged-in screens, and re-login.

This worked, but still required us to swiftly ship floppy disks to the central office technology coordinators of all these school districts, and required them to physically go to each school and perform these operations. So the next thing I did was look at the technology available for modem communications on Unix, and decide, "you know, we could do all this with uux!" The modern equivalent would be 'ssh', but this was years before ssh existed.

By the time I left that company, several years later, I could sit at my desk and press one key and send a software update to every school district in our state, which the school technology coordinator could then review and trial at the central office, then, once it was approved, himself (or herself) push one key and send that software to every school in his district. This was all being done via modems and UUCP, since this predated Internet access for schools, but because this was also the era of green screen dumb terminals where 64K-byte programs were large programs, 2400 baud modems were plenty to do the job. We had arrived at a system of deployment that used the minimum manpower possible to deploy this software across a geographically dispersed enterprise. Because of the swiftly changing requirements of state and federal regulators (who would often require updates several times during the course of the year as they decided to collect new chunks of data), this system gave us a considerable cost advantage in the marketplace compared to our competitors, who still required technicians to physically go to each school and install software updates by hand.

Now, this was a specific environment with specific needs. But you should still go into any project aimed at the enterprise with a specific goal to make it as easy as possible to deploy into the enterprise. The customer should have to enter as little data as possible into the program to make it function. It should Just Work, for the most part. And lest you say, "but my program requires complex configurations in order to make it work!", you should investigate and make sure that's true. It was thought to be true about enterprise tape backup software, for example -- that setting up enterprise tape backup software required huge amounts of technical expertise in order to configure SCSI tape jukeboxes. It was the insight of my VP of Engineering at that time, however, that all the information we needed in order to configure the entire SCSI tape subsystem was already exported either by Linux or by the SCSI devices themselves. We told customers to place one tape into each jukebox, then press the ENTER key. They pressed the ENTER key and my scanning routine went out and did the work for them. What was a half-day affair with competitors became a single button press.

The point is that a) you must think about how to distribute software updates to multiple enterprises and across enterprises with the minimum necessory human intevention, and b) even the initial deployment of seemingly complex products can become easy to deploy once you look at what technology is involved and figure out ways to automate the configuration. But you need to be thinking about deployment -- how is this going to be deployed into the enterprise -- or it's not going to happen. Instead what happens is the typical product on the market today which is expensive to deploy across the enterprise and thus it doesn't happen, or happens only haphazardly. And being typical, in today's day and age, is hardly a way towards success...

-E

iFool

2009-09-21T22:53:00.001-07:00

My main computing platform, the one I use for all my software development (via VMware which lets me develop for a half dozen variants of Linux), is a top-of-the-line Apple MacBook Pro 13.3". My phone is an iPhone 3G 16GB. Am I an iFool? Have I drank the kool-aid? Shouldn't a hardcore Linux penguin be using an Android phone and running Linux on his laptop?

Well, folks, that would be fine and dandy except for one thing: I want to get things done, I don't want to spend all my time trying to get Linux drivers working on my work laptop. And for my phone, I want it to seamlessly sync music with my laptop, I want it to be able to be plugged into the iPod input of my car stereo and just work, I want it to seamlessly sync the address book and bookmarks and notes from my laptop without having to do anything special. In short, I want it to Just Work.

So for my laptop I bought a MacBook several years ago. It Just Worked. And was Unix to boot, so all my normal tools were available from the command line, and GUI versions of them (like Emacs, The Gimp, etc.) just worked. The entire developer toolkit came for free with the computer, so I could even type './configure ; make' in my project and compile it under MacOS if I wanted to do so. I chose the 13.3" form factor because it fits on an airline tray, where bigger laptops won't. I've upgraded to the latest and greatest where it gives me a real advantage -- most recently to the new aluminum MacBook Pro with the 7-hour battery and Firewire 800 and nVidia graphics chipset -- and whenever I upgrade, the new MacBook sucks in my accumulated years of files, notes, photos, and videos without a problem. It Just Works, meaning I can do my job, instead of fiddle with the technology all day long.

For my phone, I used a Palm Treo running the Palm OS for many years after it was obsolete. Yes, the email and web clients sucked. But it synced my addresses, notes, ToDo lists, and calendars without a problem. The problem came when I was stuck in an airport waiting room needing to check on what was up with my itinerary. The hoary old Palm web browser just completely choked on the airline's web site. So what to do?

Now, I have an HTC Wizard that I used on T-Mobile some years ago. It runs Windows Mobile 5. WM5 will not sync with my Mac without extra-cost software, and my experience with WM5 was that it was technically astute, but had the user interface from hell. The Treo's user interface was simple, plain, and easy to use one-handed, the WM5 user interface really wanted three hands -- one to hold the phone sideways, and two to thumb on the thumb-board.

Then I looked at Android. And Android, alas, reminded me sorely of Windows Mobile 5. It is technically astute, but its user interface was similarly designed by geeks for geeks, rather than designed for simplicity and ease of use. As with WM5, getting anything done requires two hands, and the user interface is complex and, well, ugly.

So I arrived at the iPhone by default. It syncs seamlessly with my MacBook, and the user interface, while more complex than that of my old Treo, is fairly simple to use to do ordinary things. When I plunk it into its WindowSeat in my Jeep I can do the most common operations with one finger of one hand (mostly selecting an iTunes playlist since I'm using it for music while hooked into my car stereo at that point). No fumbling with a stylus, no poking at tiny indecipherable little pictures with the point of said stylus, everything is big and bold and easy to reach out and touch.

So what's the lessons here? First of all, if you're designing a user interface, complexity is the enemy. The iPhone was blasted for its simple -- some say simplistic -- user interface, much as PalmOS was blasted for its user interface. Yet both manage to make a virtue of simplicity to make their device much easier to use. Geeks love adding complexity to products, as do product managers who are looking to satisfy marketing checklists. It is your job as a software development manager to push back on the continual drive for user interface complexity. I once had one of my engineers give me a design proposal that was five web pages worth of highly technical stuff, all of which was useful to geeks but which would simply be gibberish for our user base. I sent it back to him with all five pages X'ed out and, on the launch page which led into that long series, I drew a selection box and a button beside it. The user selected the previously-downloaded configuration to upload, then the program just did what it was supposed to do. We probably missed out on some marketing checkboxes somewhere, but the end product was much more usable, because most users simply do not want, need, or care about all the technical details of what exactly is supposed to happen behind the scenes. They just want the computer to do the right thing. They want it to Just Work.

The second lesson is that lack of a marketing checkbox often means nothing in real life. The iPhone lacked cut-and-paste for the first two years of its life. This was a missing marketing checkbox, but it didn't hurt the iPhone's sales any. The iPhone became the best-selling smartphone in the USA despite not having a feature that everybody claimed was "necessary". The simplicity of use that Apple got from not having that feature was far more attractive to customers than the additional feature would have been, and when Apple finally developed a way to add cut-and-paste that would not impact the simplicity of the product, it was just icing on an already tasty cake as far as most iPhone customers were concerned.

The final lesson is that people just want to get work done. They want to get work done without having to fight the technology all the time, and without having to look at the internals of the technology to figure out what's wrong and how to fix it. I'm very good at debugging our products. When there's a defect that nobody else can figure out, I'm the guy who looks at it, goes and puts a few printf statements in the right place (source code debuggers are for wimps, heh!), says "Ah, I see," and then tells the appropriate programmer what went wrong and why and how to fix it. But while I'm doing that, I don't want to have to be debugging my laptop too. Both Windows and Linux force me to fix OS stuff all the time rather than actually do my work. MacOS just works. And that should be your product too -- the customer should install it, maybe type in a few setup options, and then it Just Works.

So am I an iFool? Well, yes. And you should be too. By which I do not mean that you should go out and buy an iPhone and a Mac (especially with the recent release of the Palm Pre, which appears to have learned some of the lessons of its predecessors), but, rather, that you should embrace the lessons that these products teach -- simplicity as a virtue, simplicity as not being a barrier to sales, and a product that just works, without a constant need to tweak or maintain it in order to keep it working. Do that, and your product has a chance to become the next iFool's purchase. Make it a typical overly complex difficult to manage product, on the other hand... well, then you're just another mess in a large marketplace full of messes, and will stand out about as well as a bowling pin at a bowling alley, just one more indistinguishable product in a marketplace full of indistinguishable products. Which isn't what you were setting out to do, right?

-E

So what about China?

2009-09-16T18:20:00.000-07:00

Yet more paranoia about China and outsourcing engineering to China.

Pretty much every laptop computer you buy today is already made in China. That Macbook Pro that the paranoid executive won't take to China? Made in China.

There are some reasons to suspect corporate espionage, but the nationality of the perpetrator is irrelevant. I'm unclear about the exact motivations for the rampant China bashing that seems to happen in our media today, but the Chinese as a people have every motivation to not engage in dishonest dealing -- they need the West's money in order to continue modernizing their economy. China was a third world nation only 20 years ago, with an industrial base similar to that of most Western nations in the 1950's but 10 times more people to support with that industrial base. They've come a long ways in the past twenty years, but still have a lot further to go and they know it. Engaging in organized skulduggery (as vs. the ordinary disorganized industrial espionage that happens between business competitors) is not in their best interests and the least of our worries.

I've managed Chinese programmers working on security products. They're smart, but green. They still have a lot to learn about what it takes to get products through the whole product cycle from concept to final delivered product in the customer's hands, and they know it. Perhaps at some time in the future we'll need to worry about Chinese programmers inserting time bombs into security products, but today? Again, they have too much to lose, especially if their code is being regularly reviewed by senior American engineers, as was true in our case.

In short, you should definitely follow your normal procedures for detecting and closing security vulnerabilities, but singling out one nation -- China -- for special scrutiny is just plain silly. Yes, follow good practices -- don't leave your cell phone out and about, same deal with your laptop computer, make sure your firewall software is running, don't stick foreign media into your computer's ports or hard drives or install unsigned programs, if you've outsourced development have regular design and code reviews to catch security issues early, etc. But I have to think that all this emphasis upon one nation as a "threat" has more to do with politics than with technology, and distracts us from the real problems of securing our computers against real threats -- which are more likely to come from Eastern European virus writers than anything coming out of China.

-E

A new gadget

2009-09-14T23:42:00.000-07:00

This is an HP OfficeJet Pro 8500. It is a fax machine, copy machine, scanner, photo printer, and just plain printer. It uses a water-resistant pigment-based ink so while it's not a laser printer, it fills the same basic need. It's fast, and the inks are priced reasonably on a per-page basis. So far, so good.

It replaces a laser printer, an inkjet printer, a scanner, and a fax machine, and frees up a huge amount of space in my home office even if it does sort of overlap the sides of my filing cabinet. And because it's network-connected to my Airport Extreme (via a network cable, not via WiFi), I can print wirelessly from my MacBook Pro while I'm at the dining room table or in the bedroom.

It just showed up in my Bonjour window when I went into Printer Preferences to add it as a printer. The scanner just showed up there too. Both work, and the document feeder works too for both copying and scanning, albeit I wish it had a duplexer. But for under $200, it's hard to be disappointed that the document feeder "only" gets one side of pages you feed it...

-Eric

Language wars!

2009-09-13T20:55:00.000-07:00

This one gets all the flames when development teams meet to decide on what language to use for the next project. The crusty old Unix guy in the corner says "C, it was good enough for Dennis and Ken, it's good enough for us." The Microsofty says "C++ of course. C is for Neanderthals." The J2E guy says, "Why does anybody want to use those antiquated languages full of stack smash attacks and buffer overflows anyhow? Write once, run everywhere!" And finally, the Python/Ruby guy says, "look, I can write a program all by my lonesome in two weeks that would take months for a whole team to write in Java, why is there even any question?"

And all of them are wrong, and here's why: They're talking about technologies, when they should be talking about the problem that needs solving and the business realities behind that.

I'll put myself pretty firmly in the Python/Ruby camp. My first Python product hit the market in 2000, and is still being sold today. My most recent project was also written primarily in Python. I also had a released product written largely in Ruby, albeit that was supposed to be only a proof-of-concept for the Java-based final version (but the proof of concept shipped as the real product, huh!). Still, none of these products are in Python or Ruby because of language wars, and indeed these products also included major portions written in "C" that did things like encryption, fast network file transfer, fast clustered network database transactions, and so forth. Rather, the language chosen was chosen because it was the right tool for the job. The first product was basically a dozen smaller projects written in "C" with a Python glue layer around them to provide a good user interface and database tracking of items. The second product, the one prototyped in Ruby, was prototyped in Ruby because Ruby and Java are quite similar in many conceptual ways (both allow only single inheritance, both have a similar network object execution model, etc.) and it made sense to prototype a quicky proof-of-concept in a language similar to what the final product would be written in. The last project was written in Python because Python provided easy-to-use XML and XML-RPC libraries that made the project much quicker to get to market, but also included major "C" components written as Unix programs.

So, how do you choose what language to use? Here's how NOT to choose:

The latest fad is XYZ, so we will use XYZ.
ABC programmers are cheap on the market, so we'll use ABC.
DEF is the fastest language and is the native language of our OS, so we'll use DEF.
I just read about this cool language GHI in an airline magazine...

Rather, common sense business logic needs to be used:

We need to get a product out the door as quickly as possible to beat the competition to market.
We need tools that support rapid development of object and component-oriented programs that are easy to modify according to what user feedback says future versions will need .
Performance must be adequate on the most common platforms our customers will be using
Whatever tools we use must allow a reasonable user experience.

The reality is that if you're trying to get a product to market as quickly as possible, you want to use the highest level language that'll support what you're trying to do. Don't write Python code when you can describe something in XML. Don't write Java code where Python will do the job. Don't write C++ code where Java will do the job. Don't write "C" code where C++ will do the job. Don't write assembly code where "C" will do the job. In short, try to push things to the highest level language possible, and don't be ashamed to mix code. I once worked on a product that had Ruby, Java, and "C" components according to what was needed for a particular part of the product set. There were places where Ruby lacked functionality and its performance was too poor to handle the job, for example, but Java would do the job just fine. And there was places where absolute performance was needed or where we were interfacing to low-level features of the Linux operating system where we went straight to "C", either as a network component accepting connections and data in a specified format, or as a JNI or Ruby equivalent.

The whole point is to get product out the door in a timely manner. If you decide, "I will write everything in 'C' because it is the native language of Unix and my product will be smaller and faster", you can get a product out the door... in three years, long after your competition's product hits the market and gains traction. At that point you'll be just an also-ran. What you have to do is get product out the door, get it out the door as quickly as possible with the necessary functionality and features and performance (not theoretical best, but "good enough"), and then work on getting traction in the marketplace. Perfection is the enemy of good enough, and seeking perfection often doesn't even produce a product that's any closer to perfection than the product originally written to be "good enough". That program that took three years to write? That company went bankrupt, and one reason was because the constant search for perfection ended up with a product that was inflexible, difficult to modify to work for different problem sets, and, frankly, poorly architected. Seeking the smallest memory footprint and theoretical best performance resulted in a product that failed in the marketplace because they missed the main reason we're writing programs in the first place: To meet customer needs. A program whose architecture is about memory footprint and performance at all costs is unlikely to have an architecture capable of being changed to meet changing customer needs, so not only were they late to market -- their product sucked. And the hilarious thing is, they didn't even manage to achieve the good performance their head architect claimed would happen with their architecture! So much for the three year "write everything in C" plan... late to market, poor performance, hard to modify, and they claim that those of us who advocate "good enough" rapid application development in the highest-level language feasible are "sloppy" and "advocating non-optimal practices"? Heh!

Next up... I talk about tools, and the odd story of how I got into the Linux industry in the first place. Note - I was the guy against using Linux in our shop. But that's a tale for another post :).

Is the end of rotational storage near?

2009-09-13T00:49:00.000-07:00

Okay, so this one's been predicted for over 20 years now. I remember reading Jerry Pournelle in Byte Magazine in the early 80's making this prediction. But I just went over to NewEgg.com and found a decent 64GB SSD for $150. While that's not going to replace the 2 terabyte RAID array in my Linux server, for the average consumer that's more than enough storage for anything they're ever going to do with a computer, and the 128GB SSD's are going to be just as cheap within six months, which is more storage than the average consumer will ever use.

Okay, now I hear you laughing. "64 gigabytes? Heck, my collection of unboxing videos is bigger than that!" But you're not the average person. The average person looks supiciously like me when I'm using my Acer Aspire One netbook on my Jeep expeditions rather than my top-of-the-line Macbook for development. My Aspire One gets used pretty much the same way the typical person uses their computer -- it does Internet browsing and email, I suck photos out of my camera into the Acer and resize them and post them on the Internet, and it handles a single major application -- in my case, loading detailed topographical maps and imagery into my DeLorme PN-40 GPS. It came with a 120gb hard drive. I'm using about 35GB of that hard drive right now, because I copy all my photos off onto an external drive when I get home (because I don't trust the integrity of a hard drive that's been bouncing around in a Jeep on an expedition into the Mojave Desert). Why wouldn't I put a 64GB "hard drive" into the thing that won't ever fail (at least, not because I bounce it around in a Jeep, anyhow)? After all, if I need more space, I can always plug in an external drive and copy stuff off. Frankly, 64GB is plenty of space for everything that the average person ever does with a personal computer, and 128GB is going to be $150 by the end of this year and most decidedly is enough space for the average consumer.

So maybe the end of rotational storage isn't here. But unless some new consumer application comes up that requires huge amounts of storage, it may be that we're seeing the last gasp of rotational storage in consumer hardware. After all, who needs a 500gb hard drive that might crash and fail, if a 128gb SSD costs the same amount of money and is plenty big for everything the average person wants to do with a computer?

-E

Leadership vs. arrogance

2009-09-10T21:54:00.000-07:00

At a recent meeting, we were talking about Carly Fiorina, what she did to HP that almost destroyed the company, and the sad state of leadership in corporate America today. "They're arrogant," someone said. "They had it good for so many years, and thought it was all their own doing." I disagreed. "Steve Jobs is arrogant. The problem is leadership, not arrogance."

To put it bluntly: Steve Jobs is an arrogant jerk. He has decided opinions about what his products should look like, and if you're one of his employees and going to go up against him, you better have a darn good reason for why what you're going to do is going to make the product easier to use for the majority of customers or add functionality that the majority of customers (not just a few geeks) need. You better have data, reasons, pretty pictures showing how it makes the product more consistent and easier to use, and so forth. If you can't do that, he'll tell you to go away and do it his way. When Steve came back to Apple he came back to a company in chaos, where multiple fiefdoms were feuding, where all attempts to replace the failing MacOS 9 were shot down as "too risky", where Apple's products were starting to look just like everybody else's beige boxes except running an OS far inferior to Windows NT or even Windows 95 on any technical basis. There was no shortage of talent. What there was, was a shortage of leadership -- someone who would take that talent, listen to what they had to say, process it, then say "This is what we're going to do, bang bang bang" and people walk out with their marching orders knowing exactly what they're going to do, why they're going to do it, and that failing to do it is not an option. It took, in effect, a real jerk being in charge who could rein in all these prima donnas and get them all rowing in the same direction.

Unfortunately, it's as if all of modern-day corporate America thinks being a jerk is how to be a leader. It's not. Rather, having decided opinions about what makes a product line the best in the world, opinions informed by listening to dozens of people then applying your own judgement, is what makes a leader. It's that whole vision thing, once again. That takes a certain degree of arrogance to push that vision onto a company because, let's face it, most of us in engineering are pretty arrogant ourselves. If we've been in the business for a number of years, we've seen companies come and go, we've seen products come and go, we've developed our own opinions about what makes a good product and what makes a bad one. But what a lot of managers and far too many CEO's get confused about is that they think being arrogant is leadership. It's not. It's a common product of leadership, but leadership is something else entirely -- call it vision. And given that many CEO's got their job by sucking up to the Board of Directors rather than having any vision of their own, they're followers by nature, not leaders. They're like the former CEO of DEC whose idea of "leadership" was to survey current owners of DEC VAX minicomputers asking them what they wanted as future products from DEC. Their answer was, of course, bigger and faster DEC VAX minicomputers. You'll notice that DEC is no longer in business -- because following is not leading whether you're following customers, following industry trends, or following conventional wisdom. And you can follow for only so long before you fall so far behind that you end up going down the tubes.

Yet these followers believe that, by being arrogant and expressing uninformed opinions that they refuse to change despite all data to the contrary, that this turns them into leaders. That is the problem with far too many companies today -- they are led by scared followers who are afraid that they're going to be found out as frauds, as not really being leaders, and who thus try to bluster their way into being seen as leaders via arrogance and inflexibility. But for a real manager, whether we're talking about a team lead or a CEO, all that manages to do is create an unmotivated, disillusioned work force that in the end is not going to create the innovations needed to move the company forward.

And that's the end of today's discussion of leadership. I'm going to return to this more in the future, because leadership is one of those things that is hard to quantify, but you know it when you see it. I'll further attempt to quantify it anyhow because, well, I'm just arrogant that way (heh), I always want to understand things and the best way to do that is to quantify it. And hopefully that effort might be useful both to myself, and to future team leads. We'll see, eh?

- Eric

Real life vs. movie

2009-09-08T21:55:00.000-07:00

So there is apparently a EMP bomb doomsday conference going on today, presumably to generate some leads for hardening financial institution computers. Must be a shortage of ways to spend taxpayer money, presumably.

In reality, terrorists do not need EMP devices to disable internet-connected devices. All they need to do is rent a botnet to do a DDOS. Furthermore, the biggest risk to the integrity of financial data and computing systems is not the explosion of a nuclear device, which is what would be required to create an EMP pulse capable of wiping out data (and I might add that if you just got vaporized in a nuclear explosion you're unlikely to care anymore anyhow). Rather, the biggest risk is insider fraud and technological meltdowns caused by inadequate internal controls. Remember, Countrywide and its fellow sub-prime lenders and securitizers such as Goldman Sachs did more damage to the nation's financial system than any nuke has ever done.

In short, as usual, people are focusing on flash, not meat. Computer security and financial integrity in general is not, in reality, a glamour product. It's a multi-level system of firewalls, security software, policies, procedures, regulatory activities, and checks and balances intended to detect and remedy any intrusion, extrusion, or fraud before it can culminate in actual data loss or financial loss. Or as Bruce Schneier is fond of saying, security is a process, not a product. It's a bunch of hard work, and too often not done correctly, but I suppose talking about the hypothetical effects of fictional nuclear devices is more exciting for our mindless press to blather about...

-Eric

An introduction

2009-09-08T21:22:00.000-07:00

Welcome to my new blog. It may seem odd that, just as other forms of social media are taking off, I'd take up blogging, which is now widely regarded as "old school". But in a sense, I was blogging at my old site long before blogging existed as such, I just quit doing it because it was not fun blogging with /bin/vi as your main blogging tool and I never had time to write any automated blogging tools worth the name between architecting and releasing multiple products for multiple companies.

This blog is primarily for my technology ramblings. I'm not going to talk about politics, what I did over the Labor Day weekend, or anything like that. I will point out interesting new technologies, talk about development methodologies, and talk about failed projects and un-failed projects and what's the difference between good management and poor management. I released my first product in 1988, over twenty years ago, so I have a little bit of experience in the area. I am proud to say that I have never been part of a failed project (i.e., one that failed to ship), though in more than one case that was despite incompetent management that resulted in a development process much more protracted than necessary. I've had good managers, I've had bad managers, and I've managed multiple teams of my own. Sharing some of this expertise with younger engineers and freshly minted managers might seem a bit arrogant, but I have the battle scars -- and delivered product -- to justify it.

How often will posts appear here? Good question. I'm going to aim for at least twice a week. Like most good engineers I have no shortage of opinions about what makes a good piece of software or what is the difference between a well-managed company and a poorly-managed one, and as a serial startup guy and general geek I love talking about virtualization vs. containerization and cloud computing and things of that sort, even if it just to remind folks that only the terminology is new, there are no fundamentally new concepts at work here. So I don't have a shortage of material. What I do have is a shortage of time -- that whole startup thing, remember? So we'll see. In the meantime... Virtualization is successful because operating systems are weak. Read. Think. Discuss. Enjoy :).

- Eric