The end of Google Reader

So Google Reader is going away, and people has a sad, me included, because it is by far the best RSS reader out there. On the other hand, as someone who has worked in the industry, I can pretty much tell you *why* it is going away (this is speculation on my part, but speculation that matches the actual known facts): It is going away because within months, Google is planning on making some internal infrastructure changes which will completely break Google Reader beyond any hope of repair.

The core problem is that Google Reader is old code. It originated back in Google's early days (well, 2005 is sorta early), when they didn't have any well-defined internal API's. So Google Reader depends on deep dark secrets of Google's actual infrastructure implementation, rather than using a well-defined internal API that will keep working when the infrastructure changes. The result has been that Google Reader has continually experienced outages for the past five years of its life as the infrastructure changes. It's costing Google money to keep hacking at it to keep it running, and they're not making money on it. And fixing that would require a re-write to a stable API in common with other products that wouldn't break anytime that the infrastructure changes -- something they're not going to do on something that's not making them any money.

I guess the takeaway from this is simple: Turning everything you do into an API is hard, but the alternative is that bit rot will eventually kill your code. This is, for example, why I created a complete virtualization infrastructure for the Intransa virtualized cluster product that hid all details of virtualization behind an API. The reality is that we only needed to touch ESXi in a handful of places -- but by hiding the details of that behind an API, I guaranteed that when we moved to a different virtualization system in the future (such as Xen or KVM) then all that would need changing would be the virtualization API, not any of the internal workings of Storstac. This is also one of the things that Linus has been fairly successful at doing over the past ten years or so with the Linux kernel. He may have broken the block subsystem by applying the big-kernel-lock removal patches, but at least he didn't break the API. Some additional changes have been added to the block API so that filesystems like BTRFS can work better, but the core API still remains the same as it's been for quite a few years now.

But clearly this wasn't done for Google products back in the early part of the '00s, and now it's just too difficult to maintain the less-used code. Google has to upgrade their infrastructure -- and how their infrastructure works -- in order to continue to scale. What that means is that products that weren't written to a consistent internal shared API are going to continue getting sloughed off, unless there is enough interest (and possible money to be made) to justify a re-write against a stable API. That's just how reality works. Oh well.

-ELG

Architectural decisions

Let's look at two products. The first product is a small 1U rackmount firewall device with a low-power Celeron processor and 256 megabytes of memory. It can be optionally clustered into a high availability cluster so that if one module fails, the other module takes over. Hard drive capacity is provided by a 120gb hard drive or a 64GB SSD. The second is a large NAS file server with a minimum configuration of 4 gigabytes of memory and with a minimum hard drive configuration of 3.8 terabytes. The file system on this file server is inherently capable of propagating transactions due to its underlying design.

So: How are we going to handle failover on these two devices? That's where your architectural decisions come into play, and your architectural decisions are going to in large part influence how things are going to be done.

The first thing to influence our decisions is going to be how much memory and CPU we have to play with. This directly influences our language choices, because the smaller and more limited the device, the lower level we have to go in order to a) fit the software into the device, and b) get acceptable performance. So for the firewall, we chose "C". The architect of the NAS system also chose "C". As an exercise for the reader, why do you think I believe the architect of the NAS system was wrong here? In order to get acceptable performance with the small module, we chose a multi-threaded architecture where monitor threads were associated with XML entries of what to monitor, and faults and alerts were passed through a central event queue handler which used that same XML policy database to determine which handler module (mechanism) to execute for a given fault or alert event, nothing was hard-wired, everything could be reconfigured simply by changing the XML. The architect of the NAS system had an external process sending faults and alerts to the main system manager process via a socket interface using a proprietary interface, and the main system manager process then spawned off agent threads to perform whatever tasks were necessary -- but the main system manager process had no XML database or any other configurable way to associate mechanism with policy. Rather, policy for handling faults and alerts was hard-wired. Is hard-wiring policy into software wise or necessary if there is an alternative?

The next question is, what problem are we going to solve? For the firewall system, it's simple -- we monitor various aspects of the system, and execute the appropriate mechanism specified by XML-configured policies when various events happen with the goal of maintaining service as much as possible. One possible mechanism could be to ask the slave module to take over. Tweaking policy so that this only happens when there's no possibility of recovery on the active module is decidedly a goal because there is a brief blink of service outage as the upstream and downstream switches get GARP'ed to redirect gateway traffic to a different network port, and service outages are bad. We don't have to worry about resyncing when we come back up -- we just resync from the other system at that point, if we had any unsynced firewall rules or configuration items that weren't on the other system at the point we went down, well, so what. It's no big deal to manually re-enter those rules again. And in the unlikely event that we manage to dual-head (not very likely because we have a hardwired interconnect and differential backoffs where the current master wins and does a remote power-down of the slave before the slave can do a remote power-down of the master), no data gets lost because we're a firewall. We're just passing data, we're not serving it ourselves. All that happens if we dual-head is that service is going to be problematic (to say the least!) until one of the modules gets shut down manually.

For the NAS system, it's quite a bit harder. Data integrity is a must. Dual-heading -- both systems believing they are the master -- requires either advanced transaction merge semantics when partitioning is resolved (transaction merge semantics which are wicked hard to prove do not lead to data corruption), or must be avoided at all costs by having all systems associated with a filesystem immediately cease providing services if they've not received an "I'm going down" from the missing peer(s), have no ability to force the missing peer to shut down (via IPMI or other controllable power), and no way of assuring (via voting, or other mechanisms) that the missing peers are going down. Still, we're talking about the same basic principle, with one caveat -- dual-heading is a disaster and it is better to serve nothing at all than risk dual-heading.

For the NAS system, the architectural team chose not to incorporate programmable power (such as IPMI) to allow differential backoffs to assure that dual-heading couldn't happen. Rather, they chose to require a caucus device. If you could not reach the caucus device, you failed. If you reached the caucus device but there were no update ticks on the caucus device from your peer(s), you provided services. This approach is workable, but a) requires another device, and b) provides a single point of failure. If you provide *multiple* caucus devices, then you still have the potential for a single point of failure in the event of a network partition. That is because when partition happens (i.e. you start missing ticks from your peers), if you cannot reach *all* caucus devices, you cannot guarantee that the missing peers are not themselves updating the missing caucus device and thinking *you* are the down system. How did the NAS system architectural team handle that problem? Well, they didn't. They just had a single caucus device, and if anybody couldn't talk to the caucus device, they simply quit serving data in order to prevent dual-heading, and lived with the single point of failure. I have a solution that would allow multiple caucus devices while guaranteeing no dual-heading, based on voting (possibly weighted in case of a tie), but I'll leave that as an exercise to the reader.

So... architectural decisions: 1) Remember your goals. 2) Make things flexible. 3) Use as high-level an architecture as possible on your given hardware to ensure that #2 isn't a fib, i.e., if what you're doing is doable in a higher-level language like Java or Python, for heaven's sake don't do it in "C"!. 4) Separate policy from mechanism -- okay, so this is same as #2, but worth repeating. 5) Document, document, document! I don't care whether it's UML, or freehand sketches, or whatever, but your use cases and data flows through the system *must* be clear to everybody in your team at the time you do the actual design or else you'll get garbage, 6) Have good taste.

Have good taste? What does that mean?! Well, I can't explain. It's like art. I know it when I see it. And that, unfortunately, is the rarest thing of all. I recently looked at some code that I had written when I was in college, that implemented one of the early forum boards. I was surprised and a bit astonished that even this many years later, the code was clearly well structured and showed a clean and well-conceived architecture to it. It wasn't because I had a lot of skill and experience, because, look, I was a college kid. I guess I just had good taste, a clear idea of what a well-conceived system is supposed to look like, and I don't know how that can be taught.

At which point I'm rambling, so I'm going to head off to read a book. BTW, note that the above NAS and firewall systems are, to a certain extent, hypothetical. Some details match systems I've actually worked on, some do not. If you worked with me at one of those companies, you know which is which. If you didn't, well, try not to read exact details as gospel of how a certain system works, because you'll be wrong :).

-ELG

SaaS and the Dot-com Set

One of the hilarious things that has come about with cloud computing and the advent of large scale SaaS is that I'm seeing the same kinds of arguments I saw back in the dot-com days, that this is a fundamentally different business model that doesn't obey the same rules as traditional software development. Which, of course, is utter nonsense. The method of delivery to customers has changed, but customers remain customers.

My response to all this:

1. Our primary requirement is to meet the needs of the customer. Some customers have legal requirements which preclude SaaS in the sense of SaaS in the cloud, but still wish to have the advantages of SaaS. Think doctors, schools, etc. -- it is actually illegal for them to host patient / student data outside of their own facilities on shared servers. But if we can give them the benefits of SaaS inside their facility using the same software that we have deployed into the cloud -- i.e., *not* a separate version of the software -- then we've fulfilled their needs without any (zero) additional development overhead. And BTW, just to counter one of Marko's points, anybody who structures their sales commissions structure to reward selling private rather than SaaS in the cloud is an idiot and deserves to fail, cloud is generally *much* less support on our part, it just doesn't meet the needs of certain customers.

2. I have not encountered any customers who want rapid updates of critical applications, ever. My boss once ordered me to deploy a new update to a school scheduling program in the middle of a school year. I pointed out that a) school secretaries and counsellors were currently doing mid-term schedule changes, b) school secretaries and counsellors had not been trained on the changes, which were significant (I had re-written the entire scheduling system from scratch going back to first principles, because the old system was incapable of handling some of the new scheduling paradigms that had come out, such as multi-shift scheduling and quarter-system scheduling), and thus c) it would be a fiasco. He said the old system was broken, so deploy the new one anyhow. I did. And got to say "I told you so" to my boss when it turned into the fiasco I'd predicted. My point: Users are fond of *saying* they want the latest, greatest features, but what they actually want is to get their job done. Paying attention to what users say, as vs. what they actually want, can be a huge mistake costing you a lot of money in additional support costs and losing a lot of customer goodwill. Not only were our support lines clogged solid for a week, my boss had to eat a lot of crow at the next user group meeting to get some of that lost goodwill back.

3. I am on Twitter. Yes, customers will Tweet stuff. 140 characters doesn't exactly get you in-depth commentary though. If you let Twitter guide your product development, what you get is a product designed by tweets, which is indistinguishable from a product designed by twits. I have only met one customer in my entire life who actually knew what he wanted (a school discipline coordinator who said, "I want a computerized version of these three state-mandated forms, and reports that fulfill the requirements of these four legally-required forms that I must submit at the end of the school year"). The rest have some vague idea, but you must get with them and engage them in a lengthy discussion complete with design proposals that include sample screen displays of what the application might look like. For one clustered storage product I actually spent more time talking to potential customers, writing proposals, and getting feedback than I spent implementing the actual product. Needless to say, that is *not* a process that occurs in 140 characters.

4. Anybody who goes into a business where there is an entrenched incumbent expecting to compete on features is an idiot in the first place. The incumbent has basically infinite resources at his disposal compared to you and is capable of implementing far more features than any newcomer. He will simply steal any features you innovate in order to stay out ahead. In the old days incumbents like IBM weren't capable of innovating rapidly. But this is the Internet era, and the successful giants have become much more nimble. If there's a feature you have that the incumbent doesn't have, expect him to have it soon. The way to win today is to change the game -- to do something so novel, so different in a fundamental way, that the incumbent could not match you without re-writing his entire product from scratch and ditching his entire current customer base. In short, competing based on features is a fool's game in today's era unless you're the incumbent. The way to win is to change the paradigm, not attempt to compete on features within an existing one.

5. Yes, selling "private SaaS" means we basically end up having to support multiple versions. But that is true regardless, unless we want to force customers into a death march to new versions. Some customers are comfortable with that, the majority, however, arrive at a version they like and just want to stick with that, much as the majority of Windows users are still using Windows XP, or the majority of Linux users are using Red Hat Enterprise Linux 5 (basically a three-year-old version of Linux) rather than the latest and greatest Fedora or Ubuntu. They'll accept security fixes, but that's it -- if you attempt to death march them, they'll go to a competitor who won't.

I've been dealing with satisfying customer requirements for around 15 years now, and my actual experience of those 15 years is that customers are ornery folks where what they say and what they actually want are two different things, and your job as an architect and designer is to try to suss out what they actually want, which is often *not* the same as what they say. Young engineers tend to take customers at face value, then not understand why the customer rejects the result as not meeting his needs. I get called conservative sometimes because I call for slowing down release cycles, maintaining backward compatibility wherever feasible, extensive trials with customers prior to product release, etc., but my experience is that this is what customers want -- as vs. what they say they want, which is something else entirely.

For the record - my phone is an iPhone, and the only paper maps in my car are detailed 1:24000 USGS topographical maps not available on current GPS units with reasonable screen sizes. Just sayin' ;).

-EG

People are NOT fungible

One of the things that happened during the transformation from being "employees" to being "human resources" is that large corporations apparently decided that employees are fungible. That is, if you have two employees, employee A and employee B, and employee A is making a lot more money than employee B, it's fine to just drive off employee A and replace his position with employee B then hire a contractor for even less to fill employee B's position. Hey, an employee is an employee, right? Interchangeable, just like cogs, eh?

Much has been said about Microsoft's T-Mobile Sidekick disaster and what that says about the notion of "cloud computing" (hint: As I said earlier, cloud computing does not eliminate normal IT tasks other than actual hardware maintenance). But it says even more about the whole concept of "human resources". The infrastructure that Microsoft purchased with Danger included Oracle databases, Sun servers, and a set of non-Microsft NAS or SAN systems. None of these are things that Microsoft has experience with. The current hypothesis is that Microsoft contracted a contractor to do an Oracle database upgrade, and the contractor did exactly that, and Oracle -- as it often does -- ate its database during the upgrade. This was compounded by, apparently, the database backups being unreadable by the new version of the database. All of this is remediable if you have sufficient Oracle expertise on staff, but apparently neither Microsoft nor their contractor had such expertise -- they'd all left Danger after the acquisition after being shifted to positions dealing with other technologies that they didn't like or did not have the skills to do successfully.

Lesson for managers: Identify the critical skills that you need in order to continue to have a viable business, and retain those people. It's a lot easier to retain the people you need, than to find new people with those same skills to replace them once they do leave and you discover that suddenly you no longer have a viable enterprise because critical tasks are no longer being done for lack of expertise to do them. Employees are not fungible. You simply cannot replace an Oracle database expert with a contractor hired off the streets or with an expert in Microsoft databases, Oracle databases are black magic and the people who can successfully maintain them are worth every penny you pay them.

Of course, it's easy to throw stones at Microsoft here, but it's not a Microsoft problem. It's an industry-wide problem. Managers industry-wide are failing at the task of properly identifying the skills they need in order to perform the tasks needed to have a viable business, and when the people with critical skills leave are blind-sided. Thus you get disasters like Sprint's Nextel disaster, or this Sidekick disaster, where critical infrastructure people left and the infrastructure fell apart and rendered the enterprise non-viable. Employees are not fungible, and if you fail in your job of identifying the skills needed to keep your business operating and retaining the people with those skills, you may not get the press of the Sidekick disaster, but your business will operate slower, less efficiently, and have difficulty getting product out the door. And especially look at IT and operations people. That's not sexy stuff, but both Sprint/Nextel and Microsoft/Danger show that you simply cannot fire all the operations people you just acquired and replace them with your experienced employees who are experienced with a totally incompatible technology. It doesn't work. It just doesn't. And remedying the disaster that arises after you do this will be far, far more expensive than just retaining those critical infrastructure people in the first place.

-E

The product cycle does not end at the doors of QA

Most engineers, I've found, have a very limited view of the product cycle. They get a spec from product marketing. They implement this spec. They hand the product off to QA. The product gets shipped to customers after bugs are fixed. They're done.

The problem is that this limited view of the product cycle utterly ignores the most important question of all for customers, the question that causes the most pain and headache for customers: How will this product be distributed and deployed in the enterprise?

Product marketing's spec doesn't answer this question, usually. For that matter, the customer often has no idea. The end result is that you end up with atrocities like the Windows install cycle, where deploying Windows and Windows applications across your enterprise requires massive manpower and huge amounts of time and trouble.

When you're working on your product's detailed functional specification and architecture, you must be also thinking about how to automate its deployment. You're the one who knows how the technology will work internally. So let's look at the evolution of deployment...

My first contract job after leaving teaching was a school discipline tracking program, necessary in order to meet new federal requirements for tracking disciplinary offenses. Once I had the actual program working, the next question I had was, "how will this be deployed?" I knew this company had over a dozen clients scattered all over the state, each of which had at least five schools. There was no way that we were going to deploy it by hand to all of these places. The program also required a new database table inserted into the database, so you couldn't just place the program into some location and have it work. And the program also required a modification of the main menu file to add the new program to the list of programs. So there was at least three files involved here. My solution, given we had to get this thing deployed rapidly, was to write a shell script (this was on Xenix back in the day), tar it all up, and then give short directions:

• Place distribution floppy in drive
• Go to Unix shell prompt by logging in as user 'secret' then selecting menu option 2/3/4 .
• Type:
  • cd /
  • tar xvf /dev/floppy
  • sh /tmp/install-program.sh
• Exit all logged-in screens, and re-login.

This worked, but still required us to swiftly ship floppy disks to the central office technology coordinators of all these school districts, and required them to physically go to each school and perform these operations. So the next thing I did was look at the technology available for modem communications on Unix, and decide, "you know, we could do all this with uux!" The modern equivalent would be 'ssh', but this was years before ssh existed.

By the time I left that company, several years later, I could sit at my desk and press one key and send a software update to every school district in our state, which the school technology coordinator could then review and trial at the central office, then, once it was approved, himself (or herself) push one key and send that software to every school in his district. This was all being done via modems and UUCP, since this predated Internet access for schools, but because this was also the era of green screen dumb terminals where 64K-byte programs were large programs, 2400 baud modems were plenty to do the job. We had arrived at a system of deployment that used the minimum manpower possible to deploy this software across a geographically dispersed enterprise. Because of the swiftly changing requirements of state and federal regulators (who would often require updates several times during the course of the year as they decided to collect new chunks of data), this system gave us a considerable cost advantage in the marketplace compared to our competitors, who still required technicians to physically go to each school and install software updates by hand.

Now, this was a specific environment with specific needs. But you should still go into any project aimed at the enterprise with a specific goal to make it as easy as possible to deploy into the enterprise. The customer should have to enter as little data as possible into the program to make it function. It should Just Work, for the most part. And lest you say, "but my program requires complex configurations in order to make it work!", you should investigate and make sure that's true. It was thought to be true about enterprise tape backup software, for example -- that setting up enterprise tape backup software required huge amounts of technical expertise in order to configure SCSI tape jukeboxes. It was the insight of my VP of Engineering at that time, however, that all the information we needed in order to configure the entire SCSI tape subsystem was already exported either by Linux or by the SCSI devices themselves. We told customers to place one tape into each jukebox, then press the ENTER key. They pressed the ENTER key and my scanning routine went out and did the work for them. What was a half-day affair with competitors became a single button press.

The point is that a) you must think about how to distribute software updates to multiple enterprises and across enterprises with the minimum necessory human intevention, and b) even the initial deployment of seemingly complex products can become easy to deploy once you look at what technology is involved and figure out ways to automate the configuration. But you need to be thinking about deployment -- how is this going to be deployed into the enterprise -- or it's not going to happen. Instead what happens is the typical product on the market today which is expensive to deploy across the enterprise and thus it doesn't happen, or happens only haphazardly. And being typical, in today's day and age, is hardly a way towards success...

-E