SaaS and the Dot-com Set

One of the hilarious things that has come about with cloud computing and the advent of large scale SaaS is that I'm seeing the same kinds of arguments I saw back in the dot-com days, that this is a fundamentally different business model that doesn't obey the same rules as traditional software development. Which, of course, is utter nonsense. The method of delivery to customers has changed, but customers remain customers.

My response to all this:

1. Our primary requirement is to meet the needs of the customer. Some customers have legal requirements which preclude SaaS in the sense of SaaS in the cloud, but still wish to have the advantages of SaaS. Think doctors, schools, etc. -- it is actually illegal for them to host patient / student data outside of their own facilities on shared servers. But if we can give them the benefits of SaaS inside their facility using the same software that we have deployed into the cloud -- i.e., *not* a separate version of the software -- then we've fulfilled their needs without any (zero) additional development overhead. And BTW, just to counter one of Marko's points, anybody who structures their sales commissions structure to reward selling private rather than SaaS in the cloud is an idiot and deserves to fail, cloud is generally *much* less support on our part, it just doesn't meet the needs of certain customers.

2. I have not encountered any customers who want rapid updates of critical applications, ever. My boss once ordered me to deploy a new update to a school scheduling program in the middle of a school year. I pointed out that a) school secretaries and counsellors were currently doing mid-term schedule changes, b) school secretaries and counsellors had not been trained on the changes, which were significant (I had re-written the entire scheduling system from scratch going back to first principles, because the old system was incapable of handling some of the new scheduling paradigms that had come out, such as multi-shift scheduling and quarter-system scheduling), and thus c) it would be a fiasco. He said the old system was broken, so deploy the new one anyhow. I did. And got to say "I told you so" to my boss when it turned into the fiasco I'd predicted. My point: Users are fond of *saying* they want the latest, greatest features, but what they actually want is to get their job done. Paying attention to what users say, as vs. what they actually want, can be a huge mistake costing you a lot of money in additional support costs and losing a lot of customer goodwill. Not only were our support lines clogged solid for a week, my boss had to eat a lot of crow at the next user group meeting to get some of that lost goodwill back.

3. I am on Twitter. Yes, customers will Tweet stuff. 140 characters doesn't exactly get you in-depth commentary though. If you let Twitter guide your product development, what you get is a product designed by tweets, which is indistinguishable from a product designed by twits. I have only met one customer in my entire life who actually knew what he wanted (a school discipline coordinator who said, "I want a computerized version of these three state-mandated forms, and reports that fulfill the requirements of these four legally-required forms that I must submit at the end of the school year"). The rest have some vague idea, but you must get with them and engage them in a lengthy discussion complete with design proposals that include sample screen displays of what the application might look like. For one clustered storage product I actually spent more time talking to potential customers, writing proposals, and getting feedback than I spent implementing the actual product. Needless to say, that is *not* a process that occurs in 140 characters.

4. Anybody who goes into a business where there is an entrenched incumbent expecting to compete on features is an idiot in the first place. The incumbent has basically infinite resources at his disposal compared to you and is capable of implementing far more features than any newcomer. He will simply steal any features you innovate in order to stay out ahead. In the old days incumbents like IBM weren't capable of innovating rapidly. But this is the Internet era, and the successful giants have become much more nimble. If there's a feature you have that the incumbent doesn't have, expect him to have it soon. The way to win today is to change the game -- to do something so novel, so different in a fundamental way, that the incumbent could not match you without re-writing his entire product from scratch and ditching his entire current customer base. In short, competing based on features is a fool's game in today's era unless you're the incumbent. The way to win is to change the paradigm, not attempt to compete on features within an existing one.

5. Yes, selling "private SaaS" means we basically end up having to support multiple versions. But that is true regardless, unless we want to force customers into a death march to new versions. Some customers are comfortable with that, the majority, however, arrive at a version they like and just want to stick with that, much as the majority of Windows users are still using Windows XP, or the majority of Linux users are using Red Hat Enterprise Linux 5 (basically a three-year-old version of Linux) rather than the latest and greatest Fedora or Ubuntu. They'll accept security fixes, but that's it -- if you attempt to death march them, they'll go to a competitor who won't.

I've been dealing with satisfying customer requirements for around 15 years now, and my actual experience of those 15 years is that customers are ornery folks where what they say and what they actually want are two different things, and your job as an architect and designer is to try to suss out what they actually want, which is often *not* the same as what they say. Young engineers tend to take customers at face value, then not understand why the customer rejects the result as not meeting his needs. I get called conservative sometimes because I call for slowing down release cycles, maintaining backward compatibility wherever feasible, extensive trials with customers prior to product release, etc., but my experience is that this is what customers want -- as vs. what they say they want, which is something else entirely.

For the record - my phone is an iPhone, and the only paper maps in my car are detailed 1:24000 USGS topographical maps not available on current GPS units with reasonable screen sizes. Just sayin' ;).

-EG

Cryptography engineering

A new post at the VPEP blog.

I'm currently reading Bruce Schneier's new book Cryptography Engineering (actually the second edition of Applied Cryptography), and the above was just me riffing on some thoughts I had while reading the first chapter.

-EG

About work...

One thing you'll notice, reading this blog, is that I haven't blogged about anything happening at work. There's a reason for that: It is, in general, a bad idea. If an employer believes a post puts the company in a bad light or simply decides that you have leaked proprietary information without permission, it's a great way to get fired -- dozens of bloggers have been fired over the past decade for posting about things that happened at work.

So anyhow, who I work for is no secret -- you can click on my LinkedIn profile and see -- but now I will be blogging about things I'm doing at work on my employer's own group blog. My first posts are up. You might recognize one of them as a revised version of one of the posts on this blog, except now I can say what I could only hint at then :).

-ELG

Standards and rent seeking behavior

Rent-seeking behavior is defined by economists as behavior intending to gain competitive advantage by manipulating the environment to your benefit, rather than through profiting from production of goods and services. An example of this would be if a company X managed to get a law passed specifying that all goods purchased by the government must comply with ISO standard 3.052345.32431, where company X happens to hold a critical patent on the technology in that ISO standard. In this case company X is profiting not because they produced goods and services, but, rather, because they manipulated the environment (got a law passed) which says that everybody wanting to do business with the government must pay rent (patent fees) to company X.

William Vambenepe complains that cloud standards are being created in a secretive manner. He complains that this means that those of us actually implementing cloud computing software are being locked out of the process. And this is true. Yet this is not unusual. Why? Well, because there are certain large corporations who, for some reason, still believe that rent-seeking behavior is useful when it comes to the standards process -- i.e., that, as with creating a law dictating that everybody pay rent to them, that they can set a standard that dictates that everybody pays rent to them.

Let me explain: The more complex the standard (and the more BigCorp-patented technologies included in it of course!), the more resources it will take to fully implement it. The goal is to make the resources and patent licenses needed to fully implement the standard so onerously huge that only large organizations will have the resources to do so, meaning they are the only ones who are “standards-compliant” and they can slam any potential upstart competitors as not being “standards-compliant”. Not going to name names here, but I’ll just point out that simpler standards tend to drive out the more complex standards, thereby leaving the big companies high and dry with a product that nobody wants to buy. Has anybody here used the complex X.25 protocol lately? What, you’re using the simpler TCP/IP protocol instead? Exactly.

Which points out why rent-seeking behavior is invariably self-defeating when it comes to standards. Unlike compliance with the law, compliance with standards is generally voluntary. If a standard is too complex or too expensive to implement, people simply won't use it, and a “standard” that nobody uses — or that only customers of a few large corporations use — is hardly a real standard. And keeping the standards discussions secretive is hardly in the best interests of anybody also, it means that real problems with “standards” will be overlooked until the “standard” is actually published, at which point all the effort used to produce the “standard” is useless because nobody will create products that implement the “standard” (thus rendering it *not* a standard). Yet we still see this sort of rent-seeking behavior on the part of certain large corporations that seem convinced that it actually works. Inexplicable…

-ELG

Is there such a thing as "open source management"?

The Open Source advocates have been talking about how we could apply "open source management" to things other than open source projects. But the question is, does such a thing exist?And my answer is... no. Open Source projects which do not have strong leadership fail. Commercial projects which do not have strong leadership fail. There is no "open source management" in the end, because people are people and software is software. I've been in both situations -- Open Source and commercial -- and software development is software development, in the end.

Any successful software development project of any scale other than a one-off one-person utility has some sort of leadership hierarchy where various people are in charge of various parts of the project and where there's a mechanism to insure that only high-quality code that complies with the general architectural vision of the project makes it into the project. Projects which do not develop this sort of leadership hierarchy fail -- they devolve into squabbles, or their architecture degenerates into such a mess that the project can't be successfully completed without a total re-write from scratch and a reboot. And if the quality of the people who make it into positions of being in charge is low, the project fails too, because the code base turns into a mess of buffer overflows, memory leaks, and unreadable/unfixable spaghetti code and the Object Hierarchy From Heck (the one that has 20 different levels of inheritance to do the simplest tasks, each of which reaches into the internals of its parent class to tweak something or another that it shouldn't be tweaking).

Whether you call these people "managers", "gatekeepers", "leaders", whatever, software development is software development and leadership is leadership. If you have good leadership, your project succeeds. If you don't, it fails, or is so late to market and such a low-performing mess that you might as well don't bother. That's how it's always been, whether Open Source or commercial is irrelevant. The only real difference is that Open Source contributors won't put up with pure BS as is typical in huge corporations. But that sort of BS is not typical in the small startup environment either, which shares a lot in common with Open Source.

-ELG

And now for a photo of the Linux Penguin Command and Control Center...

Where have all the legends gone?

Back in the early days of the Linux industry, it was pretty easy to know everybody who was everybody. I remember going to the Linux Expo in North Carolina in May 1998, the last year it was run by Red Hat. It was in a small upstairs part of the Student Union of one of the local universities, if I recall correctly, and there were a small handful of vendors selling software and science fiction books. Ted Tso and Linus and Alan Cox and a few others were sitting on a hassock in the lobby outside the conference rooms talking about the ext3 filesystem and how to improve the block cache for the Linux 2.2 kernel, which was scheduled for release shortly, and Richard Stallman... ah, Richard Stallman. He was... RMS. Complete with his saint outfit -- sandals, robe, and halo. Everybody avoided him as if he had fleas in his beard or something. For all I know, he may very well have. Then there was the final keynote, in an auditorium-style classroom (or is that a classroom-style auditorium?), where Linus walks out onto the stage and announced, "I am Linus Torvalds, and I am your god." The audience applauded wildly. Because we were all geeks there, and we got the joke. Today... today, I suspect Linus would get boos from humorless boobs if he tried something like that. Things change, and sometimes not for the better.

Of the vendors who were there -- The Linux Mall (I have one of the very first plush penguins!), Linux Hardware Solutions, Enhanced Software Technologies, VA Linux, DEC, etc. -- very few are still around. That fall Linux was discovered by the big guys -- IBM, Oracle, and so forth -- and everything changed forever. The Atlanta Linux Showcase in October 1998 was a zoo. Comdex in November 1998 was even more of a zoo. The big guys were moving in, and the small cottage industry that was the Linux industry was about to change forever. A few of the little guys survived, but most didn't -- they didn't have the mentality to do what it took to go big, and going big -- going for venture capital, going for IPO, going for a big business model that would have competed with the big guys -- was the only way they could survive without running into a cashflow crunch or into a hard ceiling on what they could do. It simply was not in the skill set of the small cottage industry guys, that wasn't what they did, they ran a conservatively managed business out of a small office-warehouse somewhere, they didn't try to build a huge empire. Unfortunately, cottage industry could not compete. By the end of 2001, most of them were gone, memories the only thing left.

I know what happened to a lot of the people, they have largely moved into other industries or are serving as consultants, marketing managers, or similar for other businesses -- but of the technical people, it's interesting that most of us are still around somewhere hacking on Linux. As for me, I talked to the owner of Linux Hardware Solutions there at Linux Expo and decided to move to North Carolina to work for them. It was a gamble, but it was a gamble that produced enough connections that when LHS folded I could move on elsewhere in the Linux industry into a software engineering role that eventually resulted in my first team leadership role. I spent some time as a nomadic Linux penguin. Now I'm not so nomadic -- I've lived in the same apartment for close to six years, for cryin' out loud -- but sometimes I think back to those early days of the Linux industry, and wonder if we haven't lost something since then, some excitement, some energy, some sheer exuberance that made it a joy to get it up in the morning to create something new and unheard of that had never before existed in the world. Today we're all stuffy professionals, spending our time writing design documents and doing design reviews and managing engineering teams. Then... then we were changing the world. And we did.

-ELG

iPad insanity

I suppose that, like every other geek on the planet, I have to talk about the iPad. Okay. I'm underwhelmed. Good enough?

The iPad has two problems: 1) It's too big, I can't fit it into the side pouch of my laptop case or of my carry-on bag. 2) The 10" LCD uses too much power. My little Sony PRS-300 uses e-Ink, which uses power only when you're flipping from one page to the next and allows a 2-week nominal battery life (1 week in actual heavy use). The iPad nominally has a 10 hour battery life, but my experience with my Macbook Pro, which nominally has a 7 hour battery life, says that the iPad will actually have a 7 hour battery life in real-life use -- the nominal battery life is if you have the backlighting turned down to pretty much unreadable levels, and I'm not as young as I was 14 years ago when I was releasing my first Linux product, I can't read things that are so dim anymore.

So the iPad, to me, is like the Apple TV -- a so-so product that might at some point in the future become useful, but right now is a "so what?". I normally carry my Macbook Pro and my iPhone with me when I travel, and maybe the Sony Reader tucked in the side pocket of my laptop case if it's going to be a long trip just so that I don't run out of books to read (hauling that many paper books can get problematic). I just can't see the point of the iPad, at least for me. Maybe if it were an iTampon, a smaller handier more convenient device similar in size to the Kindle, I could "get it", but for now? Too big, too power hungry, too inconvenient.

--ELG

Convicted monopolist wants RealID for Internet

Yes indeed, Craig Mundie, head of Microsoft Research, wants a "driver's license" for the Internet.

This is the craziest thing I ever heard of. It's estimated that 20% of drivers here in the state of California don't have a driver's license. Despite that, they still drive, even though if they're caught it means their car would be impounded and they'd face possible jail time. But people who are here illegally, have had driving rights removed for legal reasons such as DWI, or whatever, a lot of them choose to drive anyhow despite those risks.

There are no traffic cops on the Internet to stop you and seize your computer if you don't have the proper Internet driver's license, so how the heck would any such thing be enforced? I mean, it's as if Craig Mundie never heard of the fact that Social Security cards, birth certificates, and driver's licenses are regularly forged.

There are a lot of things that can be done to improve Internet security, such as cutting off any ISP that refuses to deal with spam or DOS attacks emanating from its IP addresses. But this? This is a non-starter. Apparently Mr. Mundie still hasn't figured out that not everybody is a law-abiding Microsoftie like he is, and that people will violate the law if they feel that it is in their best interests to do so. I mean, all he has to do is look at his own employer, for cryin' out loud! Where there is a will, there is a way -- even if it requires forgery. If Mr. Mundie's proposal were adopted, the only people who would comply would be law-abiding people, which sort of renders the whole thing useless, don't you think?

-ELG

The problem with eBooks

I've been looking at ebook readers lately -- the Sony Reader, Amazon Kindle, and Barnes & Noble Nook being the three that I looked at. All three do a good job from a hardware perspective. E-paper is quite acceptable for reading text novels and you can store enough novels on one of these things for a whole year, saving a whole lot of dead trees in the process. And the one-to-two-week battery life on these things is adequate for all purposes short of hiking the Appalachian Trail, and even there it would work as long as you brought along a USB solar charger. These e-paper devices with their relatively large high-contrast screens put reading eBooks on the small screen of an iPhone to shame, and even the lowest-power-use laptops won't get much further than five hours before they go dark, much less one to two weeks. Most of this can be attributed to the e-paper screen, whose large size, high contrast, and lack of power usage (the only time they use power is when you flip the page, static displays use no power) allows a much more pleasant experience than any prior electronic reading device.

Yet... yet. E-book uptake is ridiculously small. Even the Kindle has probably not sold more than 400,000 copies despite the fact that every time you go to the Amazon.com site the thing is hyped to you. So what, exactly, is the problem?

Well, the problem is simple: A lack of books, compounded by a) DRM, and b) the outrageous costs for the ebooks, costs that in many cases are higher than the costs for equivalent paperback books from your local bookstore. For example, Sony's ebook chief claims it's impossible to make money selling ebooks for $9.99. WTF? Bookstores can sell paperbacks for $7.99, despite the huge overhead of having a physical plant and employees and shipping charges to get the books there and etc., yet E-vendors can't make money at $9.99?!

The big publishers, to me, seem to have their heads stuck up their a$$'es in much the same way that the big music studios had their heads stuck up their a$$'es about digital music until Steve Jobs finally brow-beat them into submission to get them to sell songs for 99 cents without DRM. The various ebook vendors all have incompatible ebook formats, sell only a small smattering of publishers' catalogs in e-format, and sell them for prices higher than the paper version of the publication. And, they have them DRM-protected. Meaning that five years from now, they're worthless, because the device you installed them on will have broken and you won't be able to read them anymore.

I have files on my computer that are 20 years old. That's because I never used proprietary formats (or at least used proprietary formats that were so common that they were easily translated into non-proprietary formats), and transferred the files from older computer to newer computer every time I upgraded using open protocols such as xModem over RS-232 in the early days, and FTP and SCP over TCP/IP networks nowadays. Similarly, I have paper books in my storage room that are 30 years old that I can read just fine today. I did not buy any songs from the iTunes store until they became DRM-free because having songs in a proprietary protocol tied to a specific computer simply was incompatible with my experience in technology, which said that the songs would become unlistenable within five years if I did that. It's like having a 9 track tape from 1982. How are you going to read the data off of it? Where are you going to find the device? If it's in some proprietary format, how are you going to convert it into some modern format? Digging through my storage I came across a couple of 9-track tapes from the mid 1980's, I think they had early copies of GNU Emacs on them to be read into our VAX minicomputer at college. At least they're (probably) in tar or cpio format, but how in the world could I possibly read them when the hardware to read them is long gone and obsolete?

Yet that is what the eBook vendors want to lock us into -- having to re-buy our entire library every three to five years when proprietary eBook formats become obsoleted and the devices they were installed upon die the normal death of consumer electronic devices (i.e., 3 to 5 years, then bam, one day they just don't wake up again, maybe a battery wore out and you can't get a replacement battery because the vendor doesn't make them anymore or because the battery is more expensive than a new device, maybe a capacitor in a critical path reached its end-of-life, but it's dead, Jim). It is laughable, and it's no wonder that only a few early adopters are buying these things right now despite the great hardware that's come out recently, even though the DRM put on these early e-books is itself laughable (all -- ALL -- major e-book DRM formats have been cracked, even the Kindle one, copy protection works no better today than it worked in 1985 when we were cracking the protection on Commodore 64 games, all it does is annoy people while not impairing the pirates at all). Because, as with computers, hardware is only half the equation. If you don't have content -- if you can't get books for it for an affordable price, and more importantly, can't get books that will last longer than the device they are installed upon, what's the point?

-E

ACL management on MacOS Snow Leopard

So I was going to follow the directions at this hint site to prevent Time Machine from doing a full backup again once I updated my MacBook Pro to a bigger drive. After all, I don't want to re-backup the stuff I just restored from my backup! But my attempt slammed to a halt after I type 'fsaclctl' and... uhm... WTF? It isn't in Snow Leopard! And by the time you get to userland the permission to override a "Deny All to All" ACL is dropped even if you su to root... you just can't get there from here unless you can somehow turn off ACL support for the whole filesystem!

Ah, but never fear, the Leopard version of fsaclctl works just fine on Snow Leopard. The question is, which of my half dozen backup drives up in the storage closet or offsite is old enough to have Leopard on it? I was about to get up and go grab one, when I glanced down and... there was the Mac OS Leopard 10.5.2 install DVD, right there, in the pile of disks I'd used to re-image the Mac.

So first thing to do was drill down and find the package. The packages live in '/Volumes/Mac OS X Install DVD/System/Installation/Packages' and the easiest thing to do is 'go to folder' from the Finder 'Go' menu to go there. Then by dragging dropping packages onto the /Developer/Applications/Utilities/PackageMaker utility, I discovered that fsaclctl lives in package "BSD.pkg" in directory /usr/sbin.

The next question is, how do we get the file out of the package? I couldn't drag it out of PackageMaker, PackageMaker simply refused to do so. So I grabbed a utility called 'Pacifist'. I won't claim it's the best utility for doing this because it's simply the first one that came up when I googled, but it allowed me to drop the BSD.pkg onto it, drill down to the file, then drag the file out to a folder on my desktop, from whence I could then put it into ~/bin and use it.

Now, this isn't about the Time Machine hack (BTW, it didn't work -- apparently Time Machine's implementation has changed since Leopard), but, rather, about security. Some folks wonder why MacOS is more secure than Windows. This experience gives you one clue why. There are things you cannot override even if you have full administrative access, once permissions are dropped during the boot process. I suspect that in future releases of Snow Leopard will remove the low-level ioctl that fsaclctl relies on, further securing the system. But it's clear that while Apple doesn't make splashy announcements about security and doesn't have some of the bells and whistles like address space randomization, they're doing some things quite right in the background to keep things secure.

-EG