Today something happened which epitomized this suck. I opened up an email in Microsoft Hotmail. At the top of the email, in red, was the following message: "This message looks very suspicious to our SmartScreen filters, so we've blocked attachments, pictures, and links for your safety."
The title of the email: "TechNet Subscriber News for November".
The sender of the email: technote@microsoft.com
Siiiiigh... even their own spam filter thinks they suck.
-ELG
So, here we go:
/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BRIDGE=br0
ONBOOT=yes
/etc/sysconfig/network-scripts/ifcfg-eth1:
DEVICE=eth1
BRIDGE=br0
ONBOOT=yes
/etc/sysconfig/network-scripts/ifcfg-eth1.4000:
VLAN=yes
DEVICE=eth1.4000
BRIDGE=br1
ONBOOT=yes
/etc/sysconfig/network-scripts/ifcfg-br0:
DEVICE=br0
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=dhcp
/etc/sysconfig/network-scripts/ifcfg-br1:
DEVICE=br1
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.22.2
NETMASK=255.255.255.0
And there you are. Two bridges, one of which has a single port (eth1.4000) for virtual machines to attach to, one of which bridges eth0 and eth1 so that the machine plugged to eth1 on this cluster can also make it out to the outside world (via that bridge) as well as having the VM's on both systems communicate with each other (via the bridge attached to eth1.4000 on the 192.168.22.x network). Internal VM cluster communications stay internal, either within br1 or on VLAN 4000 that never gets routed to the outside world (we'd need an eth0.4000 to bridge it to the outside world -- but we're not going to do that). This does introduce a single point of failure for the cluster -- the cluster controller -- but it's one that's managable, if we need to talk to the outside world after the cluster controller dies we can simply plug in the red interconnect cable to a smart switch that blocks VLAN 4000 rather than into the cluster controller if the cluster controller goes down.
Now, there's other things that can be done here. Perhaps br1 could be given an IP alias that migrates to another cluster controller if the current cluster controller goes down. Perhaps we have multiple gigabit Ethernet ports that we want to bond together into a high speed bond device. There's all sorts of possibilities allowed by Red Hat's "old school" /etc/sysconfig/network-scripts system, and it won't stop you. The same, alas, cannot be said of the new "better" NetManager system, which would simply throw up its hands in disgust if you asked it to do anything more complicated than a single network port attached to a single network.
-ELG
I just finished rebuilding my system with the latest goodies to do virtualization -- I now have a quad-core processor with 12 gigs of memory and VT-d support. As part of that I just upgraded to Ubuntu 10.10, their latest and greatest. I've been using Fedora 13, Red Hat's latest and greatest, at work for the past month. My overall conclusion is... erm. Well. Fedora has its issues, but Ubuntu is getting to the point of utter opaqueness. For example, Ubuntu has a grand new grub system that generates elaborate boot menus. The only problem: Said elaborate boot menus are *wrong* for my system, they all say (hd2) where they should say (hd0). And the system that generates these elaborate boot menus is entirely opaque.... though at least I can go into the very elaborate grub.cnf file and manually edit it. Well, unless a software update has happened, at which point the elaborate boot menu generator subsystem runs again and whacks all your hd0's back to hd2s... even in entries that are old. Red Hat's grubby might be old and creaky, but at least it's never done *that* kind of silliness.
Now, granted, I am running rather unusual hardware in a far different configuration from what a desktop Linux user would want. If you want a desktop Linux system, I still believe Ubuntu is the best Linux you can put onto your system, I've run Ubuntu on the desktop for years and it serves well there. I especially like Ubuntu 10.10's ability to transparently encrypt your home directory similar to the way MacOS can, this will resolve a lot of issues with lost laptops and stolen data, this is an opaque subsystem but necessarily so. You can also put the proprietary Nvidia video drivers onto your system with a simple menu item, while with Fedora 13 you have to fight to put the proprietary driver onto the system (the GPL driver is loaded *in the initrd*, which makes it a PITA to get rid of). In short, if I were running Linux on the desktop, 10.10 is hard to beat. But for my purposes, doing virtualization research with KVM and VT-d, I'm wiping out Ubuntu in the morning and installing Fedora 13.
-ELG
Stuck in Red Hat Enterprise Linux server-land for so many years, I didn't have to deal with NetworkManager. But now I'm doing some work with kvm/qemu which requires me to have the latest and greatest kvm/qemu, which requires me to get bleeding edge on my Linux distribution because mismatches between your kernel version and the userland tools can cause some weird issues, like seemingly random kernel panics where old tools don't fill out new fields and system calls end up randomly crashing (this should never happen, BTW, the kernel should check all inputs and reject any calls that would result in a kernel panic, but I have first-hand proof that this is happening). And bleeding edge either Ubuntu or Fedora means you get to deal with NetworkManager.
By and large folks who have a single network card plus a single WiFi adapter in their Linux box and don't intend to do anything unusual will have no problems with NetworkManager. But I wanted to set up a configuration that was unusual. My Linux server has two gigabit NIC's in it. One goes out to the corporate network. I wanted the other to be a direct connection to the Ethernet port on my Macbook Pro, and then have the two be bridged so it also appears I'm directly on the corporate network. I also wanted to set up a private non-routed VLAN tied to the specific network port between my Macbook Pro and the Linux server so I could set up a private network for file sharing between the two systems -- it's still far more pleasant to do editing, email, word processing, etc. on the Mac and use the Linux box as just a big bucket of bytes. Netatalk is not perfect but works "good enough" for this particular application.
All of this is functionality that Red Hat Enterprise Linux had implemented correctly by RHEL4 days (I know that because I backported most of the RHEL4 stuff to RHEL3 to get all of this functionality working in RHEL3 back in the old days for the Resilience firewalls). That is, we're talking about things that have worked correctly for over six years. Unless -- unless you're using a system that has its network cards managed by NetworkManager, at which point you're SOL unless you disable NetworkManager, because the system absolutely refuses to bring up bridges and vlans if you have NetworkManager enabled.
So my first inclination was to simply take an axe to NetworkManager and go back to the old way of doing things, which as I point out has been working for years. That is, "rpm -e NetworkManager". At which point I find out that half of bloody Gnome seems to have a dependency upon NetworkManager, albeit two dependencies removed, and while Gnome is evil it's the lesser of two evils (hmm, sounds like politics there, eh?). So, I settled for the simple way of disabling it:
-ELG
The Droid X is an awesome piece of hardware. But my experiments with the various Android-based phones said to me that Android is still a work in progress. Each of them had odd bits of user interface that seemed unfinished or clunky or just plain badly thought out. After my brief experience with the Google hiring process it's pretty clear why that's true -- Google's hiring process, other than for a few superstars, has a built-in bias towards young mathematical types who recently took an algorithms course in college and also has a built-in filter to get rid of those of us who've been around long enough to know what we like to do and are good at doing. Youth has its advantages, but also its disadvantages -- young arrogant mathematical types rarely give much thought to user experience.
Which reminds me of an incident at a prior job. Me and my office-mate combined had maybe 15 years experience in the industry at the time, had actually used our product in production environments before joining the company that made it, and were now working on taking it to the next level with a new GUI and a new management infrastructure around our core data engine to make it easier to use in modern network environments. Our boss had about 20 years experience in the industry. So my boss assigns one of these young arrogant math PhD types to mock up a user interface for our project, we gave him the basic architecture and workflow and told him "make it easy to use." So he produces this mock up and calls me in to take a look at it and I scratch my head because I can't make heads or tails of what he's done, it isn't oriented around the workflow of any site admin that I've ever encountered. I call in my office-mate. He can't make heads or tails of it either. We ask this young brilliant mathematical type just out of college questions about how to do various site-admin-ish kinds of things, and he takes us on this long complicated set of procedures through a number of incomprehensible dialogs. My office-mate and I say "This doesn't seem like an easy to use interface for site admins." He goes, "but it's obvious! It's simple!"
So we look at each other, think, "hmm, he seems really sure about this, maybe it's just us," and call in our boss, just telling him "You have to look at this" but not why we want him to look at it. By this time we have 35 years of experience in the room, people who've actually used the technology in question in production environments. He runs through the same thing as we did, and comes to the same conclusion. By this time the arrogant young mathematical type is in pure snit mode. How *dare* we question his impeccable user interface! I explain to him that there's 35 years of experience in this room who've actually used the technology in production environments, so if we can't make heads or tails of it our customers will be utterly lost. "Then your customers are idiots!" he shouts.
Indeed. Indeed. But they are *paying* idiots. Which is what Apple understands. The customer may not always be right, but the customer is what keeps you in business, and customers want something that doesn't require a math PhD to understand or use. And in that regard, the iPhone despite its occasional glitches is still the one to beat, and Android still has a lot of growing up to do. As for that math PhD guy? Eventually after another design disagreement months later (on internals, not GUI -- we knew not to put him anywhere near the GUI by then) he stomped off and turned in his resignation because we didn't properly respect his brilliance, and my manager's manager talked him into working in another area of the company on another product rather than resigning outright, and eventually he turned in his resignation from *that* position after his tastes in user interface proved to be equally daunting for them for the same basic reason. Oddly enough, after a few years of seasoning elsewhere in the industry to brush the edges off his arrogance to turn it into less-obnoxious self confidence, he turned out to be a decent engineer... just don't put him anywhere near a user interface, for cryin' out loud. Which Google would have done immediately, if the Android user interface is any indicator.
-ELG
On the right side, weighing in at 3.8 inches, the Apple iPhone 4. This sleek little beauty is running an 800mhz Samsung A4 with a 5 megapixel camera whose sensor is the same size as the Droid's (i.e., fewer, but more sensitive, pixels).
So who is the winner? The iPhone 4's camera, despite fewer pixels, is definitely better than the Droid X camera. As in, ridiculously good. On the other hand, the iPhone 4 is more tightly-walled than any previous iPhone. As in, jailbreaking it to run "unauthorized" software is ridiculously difficult. Given the way AT&T and Apple cripple the thing, that's a major problem. The iPhone's screen is too teensy for my not-as-young-as-they-used-to-be eyes. On the other hand, it also has the iPod ecosphere with it, and tight integration with my Macbook Pro, and all my current software will continue working with it.
The Droid X, on the other hand, is easily "rooted". Motorola has created their own ecosphere of sorts, with a car dock and charger, home docks, etc. so that you can do pretty much the same things as with the Apple ecosphere. It doesn't by default have any integration with my Macbook Pro, but a third-party program called The Missing Sync will do most of that. The big screen is nice for older eyes. But Android itself is ugly and clunky, though serviceable.
So who's the winner? Call it a draw -- for now. Which presents a problem, because my aging iPhone 3G really is not liking iOS 4.0, everything runs really slow and clunky. Maybe I ought to flip a coin... or maybe if I wait a few more months, the horse will sing. Hmm...
-ELG
-- ELG
Person 1: "Well, we failed because we weren't using software engineering system X" (where X is some software engineering scheme that was popular at the time).
"Okay, so we'll use software engineering system X, I have no objection to using any particular system, as long as we use one. What's the first thing we need to do, in that system?"
Person 2: "We need to figure out what we want the product to do."
"Okay, let's do that. What is the product supposed to do?"
We discussed it for a while, then from there the meeting devolved into a list of action items, and eventually broke up with another meeting scheduled to work on the detailed functional requirements. But on the whiteboard before we left, I had already sketched out the basics of "what we want it to do", and eventually that turned into an architecture and then a product that is still being sold today, many years later.
So what's my point? Simple: Meetings must be constructive. One of the things my teacher supervisors told me, when I first entered the classroom, was to always ask myself, what do I want the students to be doing? And then communicate it. A classroom where every student knows what he's supposed to be doing at any given time is a happy classroom. Idle hands being the devil's workshop and all that. The same applies to meetings. Unless it's intended to be an informational meeting, meetings should always be about, "what do we want to do". And meetings should never be about blame-casting, finger-pointing, or any of the other negative things that waste time at meetings. No product ever got shipped because people pointed fingers at each other.
Everybody should have a takeaway from a development meeting -- "this is what I am supposed to be doing." Otherwise you're simply wasting time. So now you know why one of my favorite questions, when a meeting has gone on and on and on and is now drawing to a close but without any firm conclusion, is "what do we need to be doing? What are our action items?" We all need to know that we're on the same page and that we all know what we're supposed to be doing. That way there are no surprises, there are no excuses like "but I thought Doug was supposed to do that task!" when the meeting minutes show quite well that Doug was *not* assigned that action item, and things simply get done. Which is the point, after all: Get the product done, and out the door.
--ELG
* Usual disclaimer: The above is at least slightly fictionalized to protect the innocent. If you were there, you know what really happened. If you weren't... well, you got my takeaway, anyhow.
So: How are we going to handle failover on these two devices? That's where your architectural decisions come into play, and your architectural decisions are going to in large part influence how things are going to be done.
The first thing to influence our decisions is going to be how much memory and CPU we have to play with. This directly influences our language choices, because the smaller and more limited the device, the lower level we have to go in order to a) fit the software into the device, and b) get acceptable performance. So for the firewall, we chose "C". The architect of the NAS system also chose "C". As an exercise for the reader, why do you think I believe the architect of the NAS system was wrong here? In order to get acceptable performance with the small module, we chose a multi-threaded architecture where monitor threads were associated with XML entries of what to monitor, and faults and alerts were passed through a central event queue handler which used that same XML policy database to determine which handler module (mechanism) to execute for a given fault or alert event, nothing was hard-wired, everything could be reconfigured simply by changing the XML. The architect of the NAS system had an external process sending faults and alerts to the main system manager process via a socket interface using a proprietary interface, and the main system manager process then spawned off agent threads to perform whatever tasks were necessary -- but the main system manager process had no XML database or any other configurable way to associate mechanism with policy. Rather, policy for handling faults and alerts was hard-wired. Is hard-wiring policy into software wise or necessary if there is an alternative?
The next question is, what problem are we going to solve? For the firewall system, it's simple -- we monitor various aspects of the system, and execute the appropriate mechanism specified by XML-configured policies when various events happen with the goal of maintaining service as much as possible. One possible mechanism could be to ask the slave module to take over. Tweaking policy so that this only happens when there's no possibility of recovery on the active module is decidedly a goal because there is a brief blink of service outage as the upstream and downstream switches get GARP'ed to redirect gateway traffic to a different network port, and service outages are bad. We don't have to worry about resyncing when we come back up -- we just resync from the other system at that point, if we had any unsynced firewall rules or configuration items that weren't on the other system at the point we went down, well, so what. It's no big deal to manually re-enter those rules again. And in the unlikely event that we manage to dual-head (not very likely because we have a hardwired interconnect and differential backoffs where the current master wins and does a remote power-down of the slave before the slave can do a remote power-down of the master), no data gets lost because we're a firewall. We're just passing data, we're not serving it ourselves. All that happens if we dual-head is that service is going to be problematic (to say the least!) until one of the modules gets shut down manually.
For the NAS system, it's quite a bit harder. Data integrity is a must. Dual-heading -- both systems believing they are the master -- requires either advanced transaction merge semantics when partitioning is resolved (transaction merge semantics which are wicked hard to prove do not lead to data corruption), or must be avoided at all costs by having all systems associated with a filesystem immediately cease providing services if they've not received an "I'm going down" from the missing peer(s), have no ability to force the missing peer to shut down (via IPMI or other controllable power), and no way of assuring (via voting, or other mechanisms) that the missing peers are going down. Still, we're talking about the same basic principle, with one caveat -- dual-heading is a disaster and it is better to serve nothing at all than risk dual-heading.
For the NAS system, the architectural team chose not to incorporate programmable power (such as IPMI) to allow differential backoffs to assure that dual-heading couldn't happen. Rather, they chose to require a caucus device. If you could not reach the caucus device, you failed. If you reached the caucus device but there were no update ticks on the caucus device from your peer(s), you provided services. This approach is workable, but a) requires another device, and b) provides a single point of failure. If you provide *multiple* caucus devices, then you still have the potential for a single point of failure in the event of a network partition. That is because when partition happens (i.e. you start missing ticks from your peers), if you cannot reach *all* caucus devices, you cannot guarantee that the missing peers are not themselves updating the missing caucus device and thinking *you* are the down system. How did the NAS system architectural team handle that problem? Well, they didn't. They just had a single caucus device, and if anybody couldn't talk to the caucus device, they simply quit serving data in order to prevent dual-heading, and lived with the single point of failure. I have a solution that would allow multiple caucus devices while guaranteeing no dual-heading, based on voting (possibly weighted in case of a tie), but I'll leave that as an exercise to the reader.
So... architectural decisions: 1) Remember your goals. 2) Make things flexible. 3) Use as high-level an architecture as possible on your given hardware to ensure that #2 isn't a fib, i.e., if what you're doing is doable in a higher-level language like Java or Python, for heaven's sake don't do it in "C"!. 4) Separate policy from mechanism -- okay, so this is same as #2, but worth repeating. 5) Document, document, document! I don't care whether it's UML, or freehand sketches, or whatever, but your use cases and data flows through the system *must* be clear to everybody in your team at the time you do the actual design or else you'll get garbage, 6) Have good taste.
Have good taste? What does that mean?! Well, I can't explain. It's like art. I know it when I see it. And that, unfortunately, is the rarest thing of all. I recently looked at some code that I had written when I was in college, that implemented one of the early forum boards. I was surprised and a bit astonished that even this many years later, the code was clearly well structured and showed a clean and well-conceived architecture to it. It wasn't because I had a lot of skill and experience, because, look, I was a college kid. I guess I just had good taste, a clear idea of what a well-conceived system is supposed to look like, and I don't know how that can be taught.
At which point I'm rambling, so I'm going to head off to read a book. BTW, note that the above NAS and firewall systems are, to a certain extent, hypothetical. Some details match systems I've actually worked on, some do not. If you worked with me at one of those companies, you know which is which. If you didn't, well, try not to read exact details as gospel of how a certain system works, because you'll be wrong :).
-ELG
-ELG
