Load Balancing Digest

Note: Just a quick disclaimer here, I worked for KEMP in 2006.  I’m still Mr. Neutral when it comes to these devices, but I wanted to make sure there was full disclosure.  As with all reviews, I receive no payment whatsoever for reviews.  Only a big suitcase full of $100s would change that.

KEMP LoadMaster 1500

KEMP is a member of the value market vendors, and their LM-1500 is what put them on the map. Released in late 2005, the $2,500 price tag gained it a lot of fans. At the time, it was the only Layer 7 load balancer at that price range (a few more have introduced products at that price).

The LM-1500 has been reviewed before, but they’ve got some new code coming out that introduces some new functionality (including caching and compression), and they were kind enough to send me over a box with the soon-to-be-released code. (If you have a product you’d like to see reviewed on lbdigest.com, feel free to contact me [tony at lb digest dot com]).

Getting Started

The LM-1500 can be initially configured through either the VGA port with a USB keyboard, or through the serial port.  After it’s given an IP address, the rest of the configuration is done through a web interface.  There’s also a way to configure from the start with a web interface, as it comes up with a 192.168.1.100 IP automatically.

Administration

There is a command line interface, but virtually all of what you need to do can be best done through the web interface.  One of the nice touches they’ve added is the “download root certificate”, which installs a trusted CA cert on your browser to get rid of the annoying but ubiquitous self-signed certificate warning.

Creating a Virtual Service (VIP) is pretty straight-forward.  Give it an IP address and port, select your options (L7, cookies, etc.), and so on. There’s also configuration options for a “sorry server” (if non of your regular servers are up, the LM will send traffic to a sorry server, which can have a some sort of “sorry, we’re not working right now page”.

There’s also a handy stats section, reporting on the various performance metrics of the device.  Most of these metrics are also available through SNMP (for PRTG/MRTG, etc.).

App Delivery Features

The LM-1500 does the standard Layer 4 load balancing, as well as the more advanced cookie-based persistence and web switching.  The only drawback is that you cannot do both web switching and cookie-based persistence at the same time.

There are quite a few options available to a virtual service configuration, including transparency (full-NAT or half-NAT), health checking, and a “sorry server”, for when all the servers in your main web app farm goes down.

SSL certificate management works well as well, making it pretty easy to add/remove/change SSL certificates assigned to virtual servers. The LoadMaster automatically installs a self-signed certificate when you turn on SSL termination, and gives you the option to install CA certs and intermediate certs.  The LoadMaster 1500 does not include an RSA SSL ASIC, however (see the box section for more specifics).

Network Architecture

As I’ve said before in other reviews, one of the aspects missing in in a lot of reviews is how the product fits into the network, so I make a point to include some details.

The LM-1500 operates only in Layer-3 route-path mode, it cannot be used as a Layer-2 bridge-path load balancer.  You must either use the LM-1500 as the default gateway, or use it in non-transparent mode.   One of the easiest ways to put an LM-1500 into a network is the one-armed mode, where the Virtual Service IPs (VIPs) and the real servers all sit on the same subnet.

The LM-1500 also works well in two-armed mode, which is typically when you have the Virtual Service IPs on a public subnet and the real servers on a private RFC1918 address space (such as 192.168.x.x).

The Box

The box itself is of solid metal construction, and doesn’t look or feel “cheap”.  The bright gold is unmistakable, and in addition to being prominent in a data center, it would probably keep me safe on my late night training runs.

Stat-wise, the system is powered by a VIA Eden chip, with 512 MB of RAM which is more than sufficient for this class of system.  On-board storage is solid-state flash, so there are no moving parts (other than a fan) and the system boots very quickly.

There are three 10/100 Fast Ethernet interfaces, each operating in routed mode (the LoadMasters don’t do bridged) so each interface would each be on a different subnet.

The system boasts on on-board SSL chip, however the chip only does the symmetric “bulk” encryption algorithm AES.  Most of the heavy lifting in an SSL connection is done on the asymmetric RSA operations when an SSL connection is established.  While the AES bulk encryption would help tremendously in long lasting download-type connections, it’s of little use for rapid-fire of short lived connections, which is what most web connections are.

Caching and Compression

The newest addition to the LM-1500 (and the other LoadMaster models) is the addition of caching and compression as well as web application firewall abilities.

In its initial deployment, there’s not much you can configure with the caching and compression, other than turning it on or off per virtual service.  I was able to verify that cached objects came from the LoadMaster, and not the server.

With the LoadMaster, compression is actually another method of caching.  Instead of the on-the-fly compression that most other products do, objects are cached by the LoadMaster, and compressable objects are compressed once in the cache.  This doesn’t get you the benefits of on-the-fly compression for dynamic web pages, but it’s much easier on the CPU (there’s no compression ASIC on the LM-1500).

For compression, I used the simple apache.gif file that comes with the Apache distribution.  Normally, it’s 2410 bytes.

Content-Type: image/gif
Content-Length: 2410

Turning on compression, and I see the LoadMaster send back a gzip’d image.

Content-Type: image/gif
Content-Encoding: gzip
Content-Length: 1795

That’s a slight decrease, but not really a fair test since there’s not a lot to compress.  So to give a better test, I saved the main page of lbdigest.com, and the HTML file came out to about 38K.  I put it up as a static page on my test server, and accessed it through my browser.

 Content-Length: 38180

Turning on compression, I accessed the page again:

 Content-Encoding: gzip
 Content-Length: 11041

From 38k to 11k, that’s more than a 3 to 1 compression ratio on the main page.  Not too shabby.  If your clients were on a 56k dialup line, or connecting from a PC in say Bali, Indonesia, that 3 to 1 savings could mean a much faster page load.

With caching and compression, your mileage may vary quite a bit, depending on the nature of your users and the nature of your content.  It could help tremendously, or it could end up slowing your site down significantly, so it’s something that’s best to test first (this is true for any product that does caching).

Wish List:

My wish list would be the ability to exclude/include file extensions (no PHP, yes on JPG, etc.) and some better reporting of cache and compression statistics.

IPS/Web Application Firewall

Another new addition to the LoadMaster is the addition of IPS/WAF (Web Application Firewall) capabilities (the only other value market product to have this capability is the Barracuda, which I have not personally tested).  The LoadMaster uses SNORT-compatible rules (you can get a limited set for free at snort.org) in order to catch malicious requests.  For instance, try to go to “/etc/passwd” (http://domain.com/etc/passwd) as the URI, and the connection will be blocked at the LoadMaster, and won’t be forwarded to the server. Reporting is a basic with this first release as well, with blocked requests being reported through SYSLOG.

08-01-2008 10:10:39 Invalid URL '/etc/passwd' - WEB-MISC /etc/passwd

Having a web application firewall is part of the new PCI-DSS recommendations.

Wish List:

The ability to pull automatic updates and the ability to get your subscription through KEMP (rather than finding SNORT rules on your own) would be my wishlist for this feature.

Conclusion

Overall, this is a very solid release.  The new features (caching, compression, WAF) are going to be very handy for the SMB.  They are a bit sparse in their configuration, but work quite well as a first release.  The same code will run on all of the LoadMaster line, with the only difference is that the LM-2500 and up have SSL accelerator cards (for the RSA heavy lifting, not just AES).

Availability of the new release will be in the next few weeks.

I saw this on Slashdot today, where a bunch of hackers developed a tool for stealing session IDs in Gmail.  By default, gmail authentication is encrypted, but the rest of your session is not.  In the requests that you send to gmail is included a session ID cookie, which is in the clear.  With your gmail session cookie, I can put it into my browser, and gmail would think I’m you, without needing to re-authenticate. I could then peruse your craig’s list personal responses.  I’m guessing that would be bad.

So now Gmail will allow you to do all SSL, all the time.  This isn’t just a gmail problem, but one that affects all logged-in sessions.  I’m guessing gmail has a pretty high-end SSL accelerator in operation for this.

I was doing my obessive-compulsive reloading of Digg’s main page, and came accross this article on load testing with open source tools.  Good read.

It’s about that time again, time for another load balancing/applicaiton delivery controller survey. This one is geared more towards the SMB (small and medium-sized business market, but anyone can take the survey.  OK, perhaps not everyone, but if you run a load balancer/ADC you qualify.

So if you get a chance, head on over to the SMB Load Balancer Survey and give it a look-through.  It’s pretty quick, and should take less than 5 minutes.

I was on a run in my new home of Portland, Oregon when I came upon an AMC Pacer parked on the street.  And I thought, how like the old Alteons the AMC Pacer is.  They’re both rather odd looking, and have a rather fanatical fan following (as evidenced in it being in a starring role in the Wayne’s World movies).

There was a post on the lb-l mailing list recently about them, and it’s interesting to see how the old Alteons are still going strong in the market.

So, when you’re not looking, stuff happens.  I turn my head for a minute, and I come to find Foundry Networks gets bought by Brocade.  Ok, not bought yet, but in definate agreement to. Also, one of my favorite bloggers, Fake Steve Jobs, calls it a blog and starts posting under his real name.

I’m assuming Brocade is interested in Foundry’s abiltiy to throw packets around, and not necessarily the L7 gear, but it’ll be interesting to see how that all works out.

I realized that this was a dupe, but I think the topic is a little more effective.  Plus, I haven’t written anything in a while.  Every craftsman has their tools, and I’ve got three in particular that I use:

• Telnet (Layer 4 TCP testing utility, Layer 7 interface)
• OpenSSL (When Layer 4 is wrapped with encryption)
• HTTP Analyzers (several to choose from, to look at headers)

Telnet is particularly handy, as I can test through Layer 7 (especially if I type “GET /” and hit return twice).  Ping is pretty useless, as is traceroute, but Telnet works wonders.

I’ve documented the tools and how I use them in the latest addition to the brand new lbwiki.com:

Troubleshooting Tools

I’ve been getting a steady stream of emails recently from various vendors on various bits of industry news, and I’ve been remiss in posting them.  So I’m going to play a little catch up.

First up is news from A10 Networks, and they report that they’ve recieved $23 million in Series C funding.  Details are here.

EtherChannel is a strange creature.  It’s generally known as a way to load balance Ethernet traffic over multiple links.  Need more than 1 Gigabit of throughput, but don’t have 10 Gigabit interfaces yet?  Then bond two Gigabit links together in an EtherChannel link, and you’ve got 2 Gigabits.  That’s the idea, at least.  However, it’s not quite that simple. In many cases, it doesn’t behave the way that people would expect it to, and for that, EtherChannel is both feared and loathed.

Recently, this came up on the load balancing mailing list.  An Alteon was setup with EtherChannel, but the traffic only went over one link.   As strange as it seems at first, this is actually the way EtherChannel works in his environment, and technically it’s working correctly.  To understand why, you need to understand EtherChannel.

I could go into a long treatise on the various components of EtherChannel, but Wikipedia has it pretty well covered, so instead, I came up with the 3 Things You Need To Know About Etherchannel. Know these 3 things, and you can Google the rest.

#1: Etherchannel may not always balance load evenly (or at all) across multiple links.

Simply turning on EtherChannel does not necessarily create even load distribution across multiple bonded links.   You may get nearly perfectly distributed load, or you may get all your traffic running over a single link.  There are tweaks you can do to resolve uneven distribution depending on the situation, but there are also situations where load distribution isn’t possible at all.

#2: How Etherchannel divvies up the traffic

EtherChannel does not work by round-robin, but instead works by hashing a characteristic of an Ethernet frame. The hash determines which of the links the frame travels to in order to get to the other side.

The most basic characteristic to hash on is the source or the destination MAC address. Most devices that are capable of EtherChannel support one of those two modes (or both).

In addition, some advanced switches can hash based on source or destination IP address, and even source or destination IP address and port configuration. These are typically more advanced Layer 2/3 combo devices.  Someone from the load balancing mailing list posted a link to this article on Cisco’s site which explains which modes are available on their line of switches.

As you can probably guess at this point, that a lack of diversity of whatever characteristic you use can cause uneven load distribution.  If you’re doing the hash on the MAC address, but only one MAC address is used to send traffic to only one other MAC address, that traffic will always go over the same link.  The other links in the EtherChannel group won’t get used at all.

In fact, if you have a single MAC address, IP address, and TCP port sending traffic to a single MAC address, IP address, and TCP port (such as an FTP transfer from one host to another) load distribution is actually impossible.  Everything hashes out the same link.

Most likely, however, one of the methods (source or destination on the MAC, IP, or IP/port) will typically get you pretty good load distribution.  If it’s available on your device, that is.  Again, many devices only support source/desination MAC addresses.

#3: The sending side alone decides which link the traffic goes over

When a device determines that a frame is to go over an EtherChannel link to reach its destination, the device sending the frame is the one that chooses which of the available individual links the frame goes down. The receiving end has absolutely no say in the matter.

As a result, to obtain proper load distribution, you must configure each end individually for the best hashing characteristic available.  One device may support only MAC address hashing, while the other device may support MAC and IP.  You can mix and match.  The other ends don’t care how the frames get to them.

I tend to separate out infrastructures into two different beasts:  A site, and a super-site.  Super-sites get tremendous amounts of traffic (1 Gigabit+) and typically are associated with a brand, such as Amazon.com, Google.com, or what have you.

On Slashdot today was a posting about the infrastructure for the super-site Wikipedia.  And what load balancer do they use?  Why, Linux Virtual Server of course.