Network monitoring is not for amateurs

So, another blogger recently posted that network monitoring was easy. All you needed to do was deploy off the shelf network monitoring tools X, Y, and Z, and your network would be monitored and you would be able to sleep well at night knowing that your network was working well.

At which point I have to stop and say... what... the.... bleep?

I've been doing this a *long* time. In this or prior jobs (where I was a staff engineer at a firewall company in particular) I've deployed Nagios, MRTG, Zenoss, OpenNMS, Big Brother, and a variety of proprietary network monitoring tools such as PRTG and combined firewall/IDS tools such as Checkpoint, as well as specialty tools such as smartd, arpwatch and snort. And what I have to say is that monitoring networks is *hard*. Pretty much any of the big network monitoring tools (other than Nagios, whose configuration is utterly non-automated) will take you 80% of the way to effective monitoring of your network with a few mouse-clicks. But to go to that extra 20% needed to make sure that you're immediately notified if anything your client relies on is not providing services, you end up having to install plugins, write sensors, and combine multiple tools together. It's a lengthy task that can require days of research to solve just 1% of that last 20%.

Here's an example. I was having issues where one of my iSCSI servers was dropping off the network for thirty seconds or so. I'd learn about it because my users would start howling that their Linux virtual machines were no longer functioning properly. I investigated, and found that when Linux cannot write to a volume for a certain amount of time, it then marks the volume read-only. Which, if the volume is the root volume on a Linux virtual machine, means things go to bleep in a handbasket.

And Nagios reported not a problem during all this time. All the normal Nagios sensors for system load, disk space free, memory free, and so forth showed that the virtual machine was operating completely normally. The virtual machine was pingable, it was responding to SNMP queries, it looked completely healthy to any network monitoring tool in the universe. So I ended up writing a Nagios sensor that 'touched' a file in /, and if it couldn't because the file system had been marked read-only, reported an error. I then deployed it to my virtual machines and added it to my list of sensors to monitor, and when my iSCSI server dropped offline and the filesystems went read-only, I'd get a Nagios alert. (Said iSCSI server, BTW, is not an Intransa iSCSI server, it's a generic SuperMicro box running Linux with the LIO iSCSI stack, but that's irrelevant to the conversation). After some time of getting alerted when things went AWOL I managed to zone in on exactly the set of log file entries on the iSCSI server that indicated what was going on at the time things went to bleep, and discovered that I had a failing disk in the iSCSI system that was not triggering the smartmond monitoring tool. Unfortunately figuring out which of those 24 disks were causing the SAS bus reset loop was non-trivial given that smartctl was showing no problems when I looked at them, but once I tracked it down and replaced the disk, the problem was solved.

So let's recap. I had a serious problem with my storage network. smartmond, the host-based disk monitoring tool on the iSCSI box, couldn't detect it, all the disks looked fine when you looked at them with the tool it uses, smartctl. Nagios and all its prepackaged sensors couldn't detect it until I wrote a custom sensor. And this other blogger says that monitoring complex networks including clients, storage boxes, video cameras, and so forth is *easy*? Just a matter of installing packages X, Y, and Z and sitting back and relaxing?

Crack. That's my only explanation for anybody making such a statement. Either that, or utter inexperience at monitoring complex heterogeneous networks with multiple points of failure, of which only 80% or so are covered by off-the-shelf network monitoring tools, and the remainder requiring writing custom sensors to detect domain-specific points of failure. Either way, anybody who listens to that blogger and believes that monitoring a network effectively is easy is a fool. And that's all I will say on that subject.

-ELG

Why no cloud?

So I promised I'd explain why I was setting up normal Linux-based storage and normal KVM/ESXi compute servers for our new small business's network rather than an OpenStack private cloud, so I'll do so.

1. One risky technology per deployment. It's about risk management -- the ability to manage risks in a reasonable manner. If you have multiple risky technologies, the interactions between risks rise exponentially and cause risks to be unmanageable. Normal Linux-based storage is a mature technology with over a decade of active deployment in production environments with the exception of the LIO iSCSI target. I concluded that the LIO iSCSI target was a necessity in our environment because the TGTD target provided with stable Linux distributions has multiple serious deficiencies (see earlier postings) that render it nothing more than a toy, and our legacy infrastructure was based around iSCSI talking to that pile of ancient Intransa blue boxes. So I've reached my limit on new technologies. Meanwhile OpenStack is multiple immature technologies under active development. Add that to LIO and the existing VMware ESX/ESXi servers' need for block storage and I'd require multiple storage networks to mitigate the risks. Which brings up...
2. Power and space budget. My power and space budget allows for one storage network with a total of 8U of space and 1000 watts of power consumption. I don't have power and space for two storage networks, one for OpenStack and one for ESX/ESXi.
3. Performance. The majority of what my network provides to end users is file storage via NFS and CIFS. In an OpenStack deployment file servers run as virtual machines talking to back end storage via iSCSI. This scales very well in large installations, but I don't have the power and space budget for a large installation so that's irrelevant. Running the NAS storage stack directly on the storage boxes results in much better responsiveness and real-world performance than running the NAS storage stack on a virtual machine talking to the storage boxes via iSCSI, even if the theoretical performance should be the same. The biggest issue is that this limits the size and performance of any particular data store to one storage box, but the reality is that this isn't a particularly big limitation for our environment, since we have far more iops and storage on a single storage box than any single data store in our environment will use for quite some time. (My rule of thumb is that no ext4 data store will ever be over 1Tb and no xfs data store will ever be over 2Tb, due to various limitations of those filesystems in a NAS environment... any other filesystem runs into issue #1, one risky technology per deployment, and I already hit that with LIO)
4. Deep understanding of the underlying technologies. The Linux storage stack has been mature for many years now, with the exception of LIO. I know its source code at the kernel level fairly well. If there is an issue, I know how to resolve it, even to the point of poking bytes into headers on disk drive blocks to make things work. Recovery from failure thus is low risk (see #1). OpenStack is a new and immature technology. If there is an issue, we could be down for days while I chase around in the source code trying to figure out what went wrong and how to fix it.

Note that this is *not* a slam on OpenStack as a technology, or saying that you should not use one of the OpenStack cloud providers such as RackSpace or HP. They have massive redundancies in their OpenStack deployment and people on staff who have the expertise to manage it, and do not have to deal with legacy infrastructure requirements such as our ESXi servers with their associated Windows payloads. Plus they are based around a totally different workload. Our in-house workload is primarily a NAS workload for workstations, and our compute workload is primarily a small number of virtualized test servers or build servers for our software in a variety of environments as well as a handful of infrastructure servers to e.g. handle DNS. What OpenStack mostly gives you is the ability to manage massive numbers of storage servers and massive numbers of compute servers and massive numbers of virtual machines on those compute servers, none of which is our local workload.

The workload that RackSpace etc. are supporting is mostly about Big Data and Big Compute in the cloud or about web server farms in the cloud. All of that has far larger space and power requirements than our little two-rack data center can ever provide, and the reality is that we simply use their infrastructure when we have those requirements rather than attempt to replicate their infrastructure in-house. It simply isn't reasonable for a small business to try to replicate RackSpace or Amazon AWS in-house. We don't have the space and power for the massive amount of infrastructure they use to achieve redundancy and reliability, we don't have the requirement for our local workload, and we don't have the in-house expertise. In the end, it's a case of using the appropriate technology for the appropriate task -- and for what I'm attempting to achieve for the local infrastructure of a small business, using NAS-based Linux storage was more appropriate than attempting to shoe-horn our workload into an infrastructure that would give us no more capability for our needs but would cost us in terms of power, space, performance, and maintainability.

-ELG

Making auto-proxy configuration work

Okay, so I finally got auto-proxy browser configuration to work with ClearOS. It required a couple of different things.

First, you'll need to install the web server plugin in the ClearOS marketplace. Yes, I know you don't want a web server running on your router. But there's not much choice, wpad.dat is served via http on port 80. Just don't add a firewall rule allowing connecting to it from outside your network (note that in ClearOS you have to explicitly allow external access to services) and you'll be fine.

Next, in your DNS configuration on your master DNS server (whether that's on the ClearOS server or elsewhere), set up wpad.yourdomain.com pointing at your ClearOS server. If the ClearOS server is providing DNS that's pretty easy, just use the web interface.

Okay, now we're at the end of what the web interface can do for you. We'll need to do some things via ssh now. Make sure ssh access is turned on in your firewall rules (in the GUI), and ssh in as root/yourpassword.

In /var/www/html create a file 'wpad.dat' with these contents:

function FindProxyForURL(url,host)
{
return “PROXY 192.168.0.1:3128″;
}

Replace the 192.168.0.1 with the actual address of your ClearOS server, and if you're running the content filter, replace the 3128 with 8080.

Now access "http://wpad.your.domain" with your web browser. You should see that file appear. But that's not going to get your auto-proxy working on Chrome, at least, because Chrome wants a MIME type of application/x-ns-proxy-autoconfig rather than  text/plain . So let's go set up the MIME type. In the directory /etc/httpd/conf.d create a file 'wpad.conf' with these contents:

<Files "wpad.dat">
   ForceType application/x-ns-proxy-autoconfig

</Files>

And one 'apachectl restart' later, there you are. Your browsers on the network will now auto-configure their proxy settings to go through the ClearOS proxy.

-ELG

A tale of two routers

One of the things I want to set up at the office to preserve precious Internet bandwidth is a general proxy/firewall box. This would sit between our current infrastructure and the Internet and do firewall-y type things plus provide VPN connectivity. We have an ancient Cisco that is providing VPN connectivity but it requires a proprietary client that is hard to come by unless you have a Cisco support contract, which isn't available for this antique. Given that I have plenty of fairly recent vintage surplus server equipment hanging around plus a few quad-port Ethernet cards it seemed to me to be a no-brainer to slap the cards into a spare server and toss Linux on it and run with that as the router.

The problem, of course, is time and complexity. I know how to use iptables. I know how to set up things like squid and openvpn and dhcp and so forth. But I really don't want to. I really, REALLY don't want to. I have better things to do with my life. So I went out to find general-purpose router distributions that would do all that hard work for me. Given the equipment available to me (mostly Nehalems with 2.4ghz quad-core processors and 6gb of RAM, modest by the standards of modern compute engines, but way more system than our border router needs), I didn't need to settle for one of the tiny little distributions that are intended to fit on flash memory chips on tiny embedded systems. I could put a full-fledged Linux on there. After some research, I settled on two distributions that are based on a stable core distribution: ClearOS, which is based on Centos 6, and Zentyal, which is based on Ubuntu 12.04 LTS. I know how to manage both Centos and Ubuntu since I've used them in production for years, so if all else failed I could take the starting point that the GUI configurator gave me and fix it to work.

Before doing this at the office I had to of course do a proof of concept. And the perfect proof of concept was my home network, which has five PC's and five devices on it as well as the wireless access point and the managed switch both of which have their own addresses. So I grabbed a decommissioned box that had some pretty hefty stats (Core I7-950 with 12gb of RAM) but no drives (since I'd moved the drives to the big file server box), found a pair of 2.5" drives to fit in its front-loading slot to swap out ClearOS and Zentyal, and set to installing.

I'd been playing with Zentyal for a while at work, seeing if I could make sense of whether it would replace our ancient Windows 2003 domain controllers, and so I started with ClearOS. It slid onto my server just fine, locating my expensive 4 port Intel NIC card and the on-board Intel NIC. I configured it to provide separate networks for my wireless and wired networks (so I could monitor what was happening on the wireless network specifically), and route all traffic out the cable modem connection. All was well. I played with the proxy server settings. That worked pretty well, with the bizarre exception that I can't figure out how to make the automatic proxy settings work, I enabled the Apache server and created the correct wpad file and I see Chrome using it in the Apache server logs but Chrome isn't doing applying the settings for some reason. Okay, something to check out on Zentyal when I do it. I then configured OpenVPN and installed OpenVPN clients on my Android and iPad (I already know OpenVPN works on Windows, Linux, and Mac, duh). My first couple of attempts to connect from my Android didn't work and I was baffled. Finally I clicked on the firewall module and noticed no rule had been created to allow OpenVPN connections when I configured OpenVPN. Point, click, allow, and all works well. iPad worked fine too once I got the certificates on there, which required using iTunes (bleh!) but at that point the iPad OpenVPN software was up and going. And finally I got the mail relay up and going, which forwards all outbound smtp traffic to my mail server in the proto-cloud which then forwards it onwards. Again there were some interesting limitations -- I see no place to set the name and password to authenticate with the remote smartmail server, for example -- but that's easy enough to fix by hand.

Okay, so there's a couple of small glitches but things pretty much were going smoothly with ClearOS. The main issues I ran into with ClearOS were between my ears, i.e., I didn't RTFM and forgot to set up things that needed setting up or set them up incorrectly. So next I shut down the ClearOS system, slid out its drive, slid the new drive in, and installed Zentyal. That, on the other hand... that was pretty much a disaster. It crashed halfway through the setup wizard. It crashed after it updated. It crashed trying to set up the mail relay. The OpenVPN functionality worked but the user interface left a lot to be desired. I noticed that it'd set my domain to a comcast.net domain and set it back to my own domain, and that pretty much was all she wrote -- it wiped out my VPN, it wiped out the user LDAP directory, and put the system into a completely unusable state.

Which is a shame, because I really wanted to like Zentyal. It is based on a newer and arguably better Linux distribution than ClearOS, and it has some really nice features. But I just can't deal with software that crashes when we're talking about a mission-critical server. It just isn't going to work. There's some places elsewhere in my infrastructure that Zentyal can live, but the border router? Nope. Not happening. So it goes.

UDP or TCP for lossy networks?

So I was discussing video streams today and how typical video streaming protocols like RTP react to loss on the network -- i.e, by dropping frames -- and someone made the statement, "TCP would be better for lossy networks." He was wrong, of course. But I hear you saying, "but... but... TCP recovers from data loss!" Well, yes... if you don't care about timing or throughput, only about reliability. But if you're talking video streaming, TCP could result in loss of not only a few frames, but could result in loss of seconds worth of frames, or even cause video cameras, switches, or servers to crash.

At which point you're saying, "say wha?!" But here's the deal. If you lose a video packet with UDP, you lose a frame. RTP handles reassembling the packets into frames, if the next frame has come in and there's no packet to fill in slot X in the previous frame, RTP just throws the frame away. If you're getting 28 frames per second, you lose one frame out of that, big deal. Just replay the previous frame on replay, nobody will notice.

Now let's say you're using TCP. A missing packet occurs. TCP is expecting packet N... and doesn't get it. The TCP window goes SLAM at that point -- 4 megabytes of data backs up behind the missing packet as TCP tries to build its window and has a missing piece that it is waiting for, then TCP flow control assures that no more data is accepted until after the 1.5 second timeout causes a retransmit on the sender side. What this does is add 1.5 seconds worth of delay. You have a 1.5 seconds of delay, 4 megabyte window, and a gigabit Ethernet line, and what that means is that one lost packet every 1.5 seconds basically causes your gigabit Ethernet line to become a 21 megabit/sec Ethernet line. Congratulations, you just returned to 1985. And if you have 200 megabits/sec of incoming data, all that data is piled up in the network stack somewhere between you and the cameras, and you're in big trouble.

For streaming data, UDP thus handles a lossy network much more gracefully than TCP does. At least, that's true if you don't care about data loss due to redundancy in the data stream. I.e., frame 1000 in a video stream is very similar to frame 999 and frame 1001, so if frame 1000 goes missing it can be recontructed as simply the duplicate of the preceding or following frame. So what if you *do* care about data loss?

The reality is that if you *do* care about data loss, you're much more likely to get good results when streaming data if you use out-of-band data recovery. That is, if you've tried filling your frame, you already have packets for a couple more frames incoming, and you're missing a couple of packets, request packet retransmission directly via a NAK rather than wait for the sender to timeout waiting for an ACK. Thus *one* reason why most reliable multicast protocols rely on recipients proactively sending a NAK rather than time out waiting for an ACK before sending a packet. You can combine the two if you wish -- in this case, ACK would let the sender know that all packets in the frame were received thus it doesn't need to retain the frame in order to potentially service a NAK -- if you're really wanting both reliability and performance. But TCP doesn't do that. Which is why, on a lossy network, TCP is probably the worst protocol you could use for streaming data, and why the person who said "TCP is best for lossy networks" was wrong, at least for this application.

-ELG

Setting up bridging and vlans in Fedora 13

In the previous half of this, I talked about how Fedora's NetworkManager interfered with complex configurations, and discussed how to disable it. Now I'll show you how to define a network that consists of:

• Two Ethernet ports, bridged:
  • eth0 - to public network
  • eth1 - to internal network transparently bridged to public network
• VLAN 4000 on internal network, *NOT* bridged to public network
• Bridge to VLAN 4000 for my KVM virtual machines to connect to, *NOT* bridged to public network.

Now, to do all of this we take advantage of an interesting feature of the Linux networking stack -- bridges aren't "really" bridges. If a packet arrives at the physical eth1 hardware, it gets dispatched to either eth1.4000 or eth1 based upon the VLAN tag. Only those packets that are dispatched to eth1 actually make it onto the bridge to go to eth0. In other words, the Linux bridging code is a *logical* bridge, not a *physical* bridge -- it is not the physical ports that are connected, it is the logical Ethernet devices inside the Linux networking stack that are connected, and eth1.4000 and eth1 just happen to be connected to the same physical port but otherwise are logically distinct with the dispatching to the logical Ethernet device happening based upon the VLAN header (or lack thereof).

So, here we go:

/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BRIDGE=br0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1:
DEVICE=eth1
BRIDGE=br0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1.4000:
VLAN=yes
DEVICE=eth1.4000
BRIDGE=br1
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-br0:
DEVICE=br0
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=dhcp

/etc/sysconfig/network-scripts/ifcfg-br1:
DEVICE=br1
TYPE=Bridge
USERCTL=yes
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.22.2
NETMASK=255.255.255.0

And there you are. Two bridges, one of which has a single port (eth1.4000) for virtual machines to attach to, one of which bridges eth0 and eth1 so that the machine plugged to eth1 on this cluster can also make it out to the outside world (via that bridge) as well as having the VM's on both systems communicate with each other (via the bridge attached to eth1.4000 on the 192.168.22.x network). Internal VM cluster communications stay internal, either within br1 or on VLAN 4000 that never gets routed to the outside world (we'd need an eth0.4000 to bridge it to the outside world -- but we're not going to do that). This does introduce a single point of failure for the cluster -- the cluster controller -- but it's one that's managable, if we need to talk to the outside world after the cluster controller dies we can simply plug in the red interconnect cable to a smart switch that blocks VLAN 4000 rather than into the cluster controller if the cluster controller goes down.

Now, there's other things that can be done here. Perhaps br1 could be given an IP alias that migrates to another cluster controller if the current cluster controller goes down. Perhaps we have multiple gigabit Ethernet ports that we want to bond together into a high speed bond device. There's all sorts of possibilities allowed by Red Hat's "old school" /etc/sysconfig/network-scripts system, and it won't stop you. The same, alas, cannot be said of the new "better" NetManager system, which would simply throw up its hands in disgust if you asked it to do anything more complicated than a single network port attached to a single network.

-ELG

NetworkManager Sucks

Do a Google on the above search term, and you get over 20,000 results. NetworkManager is yet another attempt by the inept Linux desktop crew to make the Linux desktop look Windows-ish, and is as Fail as you might imagine anything desktop-ish from that gang. I mean, c'mon. We're still talking about an OS that cannot dynamically reconfigure its desktop when you plug an external monitor into your laptop, something that both Windows 7 and MacOS have absolutely no problem with. You expect them to get something as simple as a network manager correct? Uhm, no.

Stuck in Red Hat Enterprise Linux server-land for so many years, I didn't have to deal with NetworkManager. But now I'm doing some work with kvm/qemu which requires me to have the latest and greatest kvm/qemu, which requires me to get bleeding edge on my Linux distribution because mismatches between your kernel version and the userland tools can cause some weird issues, like seemingly random kernel panics where old tools don't fill out new fields and system calls end up randomly crashing (this should never happen, BTW, the kernel should check all inputs and reject any calls that would result in a kernel panic, but I have first-hand proof that this is happening). And bleeding edge either Ubuntu or Fedora means you get to deal with NetworkManager.

By and large folks who have a single network card plus a single WiFi adapter in their Linux box and don't intend to do anything unusual will have no problems with NetworkManager. But I wanted to set up a configuration that was unusual. My Linux server has two gigabit NIC's in it. One goes out to the corporate network. I wanted the other to be a direct connection to the Ethernet port on my Macbook Pro, and then have the two be bridged so it also appears I'm directly on the corporate network. I also wanted to set up a private non-routed VLAN tied to the specific network port between my Macbook Pro and the Linux server so I could set up a private network for file sharing between the two systems -- it's still far more pleasant to do editing, email, word processing, etc. on the Mac and use the Linux box as just a big bucket of bytes. Netatalk is not perfect but works "good enough" for this particular application.

All of this is functionality that Red Hat Enterprise Linux had implemented correctly by RHEL4 days (I know that because I backported most of the RHEL4 stuff to RHEL3 to get all of this functionality working in RHEL3 back in the old days for the Resilience firewalls). That is, we're talking about things that have worked correctly for over six years. Unless -- unless you're using a system that has its network cards managed by NetworkManager, at which point you're SOL unless you disable NetworkManager, because the system absolutely refuses to bring up bridges and vlans if you have NetworkManager enabled.

So my first inclination was to simply take an axe to NetworkManager and go back to the old way of doing things, which as I point out has been working for years. That is, "rpm -e NetworkManager". At which point I find out that half of bloody Gnome seems to have a dependency upon NetworkManager, albeit two dependencies removed, and while Gnome is evil it's the lesser of two evils (hmm, sounds like politics there, eh?). So, I settled for the simple way of disabling it:

• # chkconfig NetworkManager off
• # service NetworkManager stop

At that point I no longer have any networks configured other than localhost (eep!), so then I get to set up the bridging and VLAN by hand in the /etc/sysconfig/network-scripts files and bring everything up (GUI tools? Linux penguins don't need no freepin' GUI tools, we just cat - >somefile when we want to configure Linux ;-). But I'll talk about that in my next post.

-ELG