HAProxy

Quick links

Thanks for your support !

News

Dec 28th, 2012 : Development 1.5-dev17

I broke a few things in dev16 which have been repaired in 1.5-dev17. This is a very small version with few changes. All 1.5-dev users are then encouraged to upgrade to dev17.

Dec 24th, 2012 : Development 1.5-dev16

Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John Rood have done at Picturae ICT we could finally spot the freeze bug after one week of restless digging ! This bug was amazingly hard to reproduce in general and would only affect POST requests under certain circumstances that I never could reproduce despite many efforts. It is likely that other users were affected too but did not notice it because end users did not complain (I'm thinking about webmail and file sharing environments for example). During this week of code review and testing, around 10 other minor to medium bugs related to the polling changes could be fixed.

Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a global error stack that must constantly be flushed (surely they never heard how errno works). The result is that some SSL errors could cause another SSL session to break as a side effect of this error. This issue was reported by J. Maurice (wiz technologies) who first encountered it when playing with the tests on ssllabs.com.

Another bug present since 1.4 concerns the premature close of the response when the server responds before the end of a POST upload. This happens when the server responds with a redirect or with a 401, sometimes the client would not get the response. This has been fixed.

Krzysztof Rutecki reported some issues on client certificate checks, because the check for the presence of the certificate applies to the connection and not just to the session. So this does not match upon session resumption. Thus another ssl_c_used ACL was added to check for such sessions.

Among the other nice additions, it is now possible to log the result of any sample fetch method using "%[]". This allows to log SSL certificates for example. And similarly, passing such information to HTTP headers was implemented too, as "http-request add-header" and "http-request set-header", using the same format as the logs. This also becomes useful for combining headers !

Some people have been asking for logging the amount of uploaded data from the client to the server, so this is now available as the "%U" log-format tag. Some other log-format tags were deprecated and replaced with easier to remind ones. The old ones still work but emit a warning suggesting the replacement.

And last, the stats HTML version was improved to present detailed information using hover tips instead of title attributes, allowing multi-line details on the page. The result is nicer, more readable and more complete, as can be seen on the demo page.

All 1.5-dev users are then encouraged to upgrade to dev16. Update: minor last-minute regression on the stats page, please use the latest snapshot instead.

Dec 12th, 2012 : Development 1.5-dev15

This is an incremental fixup on top of dev14 to address the few remaining bugs that were reported since its release, and particularly the high CPU usage that a few users have reported. Some SSL issues were also fixed and its cache was improved to use 4 times less memory. The conditions to enable compression were tightened. The strange server errors that were logged and counted for years were in fact client errors, and that was fixed. SSL handshake errors are now logged. Tracking layer 7 information is now possible ; it was limited to "src" till now. It will allow people behind proxies to benefit from some scraping or DoS protection.

All 1.5-dev users are then encouraged to upgrade to dev15.

Nov 26th, 2012 : Development 1.5-dev14

This is a quick fixup for all the bugs that were reported in dev13. All users are encouraged to upgrade to dev14 and to drop both dev12 and dev13 !

Nov 22th, 2012 : Development 1.5-dev13 with Compression!

This is the largest development version ever issued, 295 patches in 2 months! We managed to keep the Exceliance team busy all the time, which means that the code is becoming more modular with less cross-dependences, I really like this ! First, we got an amazing amount of feedback from early adopters of dev12. It seems like SSL was expected for too long a time. We really want to thank all those who contributed patches, feedback, configs, cores (yes there were) and even live gdb access, you know who you are and you deserve a big thanks for this! Git log says there were 55 bugs fixed since dev12 (a few of them might have been introduced in between). Still, this means that dev12 should be avoided as much as possible, which is why I redirected many of you to more recent snapshots. These bugs aside, I'm proud to say that the whole team did a really great job which could be summarized like this :

1. SSL:
   • many more features ; client and server certificates supported on both sides with CA and CRL checks. Most of the information available in SSL can be used in ACLs for access control. Some information such as protocol and ciphers can be reported in the logs. These information are still not added to HTTP logs though, a lot of config work is still needed.
   • cache life time and maximum concurrent SSL connections can be set. Unfortunately OpenSSL happily dereferences NULL malloc returns and causes the process to die if memory becomes scarce. So we can only limit its maximum amount of connections if we want to limit the memory it uses.
   • TLS NPN was implemented with the help from Simone Bordet from Jetty, and can be used to offload SSL/TLS for SPDY and to direct to a different server depending on the protocol chosen by the client.
   • Ivan Ristic from ssllabs and Andy Humphreys from Robinson-way provided very valuable help in diagnosing and fixing some nasty issues with aborts on failed handshakes and improve from an E-grade to an A-grade.
2. HTTP Compression
   • HTTP payload compression was implemented at Exceliance to achieve bandwidth usage reduction and reduce page load time on congested or small links. Compression is extremely CPU and memory intensive, so we spent most of the time developing dynamic adaptations. It is possible to limit the maximum RAM dedicated to compression, the CPU usage threshold and bandwidth thresholds above which compression is disabled. It is even possible to adjust some of these settings from the stats socket and to monitor bandwidth savings in real time. Proceeding like this ensures a high reliability at low cost and with little added latency. I've put it on the haproxy web site with nice bandwidth savings (72% avg on compressible objects, 50% on average, considering that most downloads are compressed sources). I must say I'm very happy of this new feature which will reduce bandwidth costs in hosted infrastructures ! And it goes back to the origins of haproxy in zprox 14 years ago :-)
3. Health checks
   • SSL is now usable with health checks. By default it is enabled if the server has the "ssl" keyword and no "port" nor "addr" setting. It can be forced using "check-ssl" otherwise. So now running an HTTPS health check simply consists in using "option httpchk" with "ssl" on the server.
   • send-proxy is also usable with health checks, with the same rules as above, and the "check-send-proxy" directive to force it. The checks also respect the updated spec which suggests sending real addresses with health checks instead of sending unknown addresses. This makes it compatible with some products such as postfix 2.10 for example.
4. Polling
   • speculative polling was generalized to all pollers, and sepoll disappeared as it was superseded by epoll. The main reason for this important change is the way OpenSSL works and the fact that it can easily get stuck with some data in buffers with no I/O event to unblock them. So we needed to engage into this difficult change. I'd have preferred to delay it to 1.6 if I was offered the choice ! But in the end this is good because it's done and it improves both performance and reliability. Even select() and poll() are now fast.
   • the maxaccept setting was too low on some platforms to achieve the highest possible performance, so it was doubled to 64 and is now per listener so that it automatically adjusts to the number of processes the listener is bound to. This ensures both best performance in single process mode, and quite good fairness in multi-process mode.
5. Platform improvements
   • Linux 3.6 TCP Fast Open is supported on listeners ("tfo" bind keyword). This is used to allow compatible clients to re-establish a TCP connection in a single packet and save one round-trip. The kernel code for this is still young, I'd be interested in any feedback.
   • use of accept4() on Linux >= 2.6.28 saves one system call.
6. Process management
   • stats socket can now be bound to specific processes. This is useful to monitor a specific process only.
   • "bind-process" now supports ranges instead of silently ignoring them.
   • "cpu-map" establishes a mapping between process numbers and CPU cores. This is important when running SSL offloaders on dedicated processes because you don't want them to pollute the low-latency L7 core.
7. Misc : "redirect scheme" makes it easier to redirect between http and https, config error reporting was improved for "bind" and "server" lines by enumerating the list of supported options dynamically.

I must say I'm much more confident in dev13 than I was with dev12 and I have already upgraded the main web site which has been upgraded every few days with recent snapshots. I've build and run it on Linux i586/x86_64/armv5/v7, OpenBSD/amd64 and Solaris/sparc without any issue anymore.

To all those running SSL tests on dev12, please drop it for dev13. I don't think we introduced regressions (but that's still possible), but I know for sure that we fixed a lot! The usual changelog and source are available at the usual place.

Sept 10th, 2012 : Development 1.5-dev12 with SSL!!!

The main, long-awaited, feature this time is native SSL support on both sides, with SNI and multi-process session sharing. The work took several months to be done at Exceliance because it required a major rewrite of the lower connection layers in order to support multiple data layers. This was a very painful task, but doing so allowed us to shrink the SSL patch from several thousands of lines of hardly maintainable code to a few hundreds of SSL-specific code. The code supports the Server Name Indication TLS extension (SNI), which consists in presenting the certificate which matches the host name requested by the client. This also works with wildcard certificates, of course. The certificates can be loaded from a directory, which makes it more convenient to load hundreds or thousands at a time. And since they are loaded into a binary tree, there is no lookup overhead even if there are hundreds of thousands, which is very convenient for massive hosting providers.

In current state, the code does not yet support checking certificates, which also means that connecting to an SSL server is only useful if the LAN is safe (in short, it's only useful if the server absolutely wants to get the connection to port 443). But the Exceliance team is actively working on this.

We took care of correctly arranging connection and data layers. Right now it's perfectly possible to chain multiple layers of haproxy servers to offload more SSL, using SSL-ID affinity and the PROXY protocol in order not to lose the client's source address. Doing this with off-the shelf hardware can result in quite a cheap SSL offloader even for huge loads. We measured 4000 TPS on SSLv3 on an Atom D510 and have not yet run the tests on larger hardware.

Among the other features in this version, we can list IPv6 transparent mode, "base" pattern/acl to match a concatenation of the Host header and the URI, "urlp_val" ACL to match a URL parameter's value, support for the "nice" keyword on "bind" lines to change the priority of sessions using this bind line (useful to limit SSL CPU impact), the ability to clear/feed stick-table entries on the stats CLI (which got lost forgotten in a dead branch), and the usual set of halog features and optims.

The changelog is available for more information, though there are a lot of commits to transform the connection layers. Users who need SSL should really give it a try. While we got a number of useful reports on the mailing list and could fix some issues, it is very likely that some bugs remain, so if you observe abnormal behaviours, please report your experiences there.

On the stable branch side, 1.4.22 was silently released one month ago with a number of small fixes and a number of minor feature improvements, such as the ability for putting a server in soft-stop mode from the stats web page in admin mode, and support for the "httponly" and "secure" flags on cookies.

June 4th, 2012 : Development 1.5-dev11

A large number of bugs were fixed again since 1.5-dev10, some of them being regressions from 1.5-dev8 and later versions. See the changelog for more information, but nobody should be running on dev9 nor dev10. Minor harmless features were added in dev11, such as new actions on the stats page, a few new cookie options, and some minor improvements on URI hashing and server recovery mode. Users should really upgrade, as I don't want to waste time trying to spot stupid bugs in configs that are notoriously broken.

May 21st, 2012 : Stable 1.4.21

A number of old bugs were reported recently. Some of them are quite problematic because they can lead to crashes while parsing configuration or when starting up, which is even worse considering that startup scripts will generally not notice it. Among the bugs that 1.4.21 fixes, we can list : risk of crash if using reqrep/rsprep and having tune.bufsize manually configured larger than what was compiled in, risk of crash when using header captures on a TCP frontend (uncaught invalid configuration), risk of crash when some servers are declared with checks in a farm which does not use an LB algorithm (eg: "option transparent" or "dispatch"), "balance source" did not correctly hash IPv6 addresses resulting in IPv4 connections to IPv6 listeners always having the same hash. Some other minor fixes and improvements were merged. While it's very likely that almost nobody is affected by the bugs above, troubleshooting them is annoying enough to justify an upgrade.

May 8th, 2012 : Development 1.5-dev9

Many new features were added since 1.5-dev7 (I forgot to announce dev8 here). Let's summarize this shortly : new logging subsystem with customizable log formats, a unique-ID generator, full rework of the buffers and HTTP message storage, merge of the ACL and pattern fetch code, ACL support for IPv6 addresses, cookies, URL parameters and arbitrary payload, support for specifying a precise occurrence in fetch functions, much better error reporting for ACL parsing errors, the long-awaited "use-server" directive, minor improvements to the error capture reports, and a significant number of bugfixes. Please give it a test.

March 10th, 2012 : Stable 1.4.20

A few bugs were reported since 1.4.19 was released, and some were found in 1.5 during development. Servers tracking disabled servers would still be used while disabled. Zero-weight servers could still dequeue requests pending in the backend's queue. The build was broken on FreeBSD since 1.4.19. Since the introduction of client keep-alive, a server would not pick a pending requests after releasing a connection if it keeps exactly maxconn-1 connections, which is problematic with low maxconn values. POST requests smaller than the buffer would experience an undesirable additional delay of 200ms due to a flag being left unconditionally enabled on the buffer. Sometimes when sending data wrapping across the buffer, haproxy would fail to merge TCP segments into a single one, which results in a few PUSH packets that can sometimes be observed during chunked-encoded transfers (this was just a missed optimization). 1.4.20 was released with all these changes. Some of them are important enough to justify an upgrade, eventhough they've been here for a very long time.

January 8th, 2012 : Stable 1.4.19

A few bugs were fixed since 1.4.18, and they impacted users so I wanted to release something now eventhough none of them is critical. First, Sagi Bashari fixed the usage of alternative header name for the "forwardfor" option. An incompatibility between server tracking and slowstart, was diagnosed by Ludovic Levesque : the weight would remain at the lowest level forever. Daniel Rankov reported that option nolinger did not work in backends. It looks like it has been the case for a very long time now. An issue in the string indexing in ebtrees was diagnosed by Julien Thomas. It is used in ACLs could theorically affect the ACL code though it has no visible effect since all patterns in the same ACL are interchangeable. Timothy Garnett reported an issue where Ruby clients were experiencing an extra delay in response time. After analyzing some network traces, it appeared that Ruby likes to send POST requests in multiple incomplete packets, waiting for the first one to be ACKed before pushing the rest, which is incompatible with the delayed ACK. Since we get the incomplete request, we can notice that it's missing data and re-enable quick ACKs to make the client send the rest ASAP. Obvously the client should be fixed as its behaviour makes it very sensible to network latency. Brian Lagoni reported that TProxy broke after Linux 2.6.34 kernel, because the address family was previously assumed to be AF_INET and was not set in HAProxy. Last bug, I was fed up with HAProxy blocking invalid server responses which were sent without headers. I finally understood that it was because some requests were sent with a "\0" in the URI which HAProxy did not block, and Apache considered the request line truncated and ignored the HTTP version, resulting in HTTP/0.9. So the request parser was modified to reject control characters in the URI (the standard forbids other characters but we can't change too much in a stable version without risking breaking some setups). One minor feature was merged. Mark Lamourine worked on a solution to send a server's name in a header when connections are established to a server. I know this can be useful in some silo-like setups and the code does not present any risk of regression so I accepted to include it in 1.4. So 1.4.19 was released with all these changes. If you have no problem with current version, there is no need to upgrade.

September 16th, 2011 : Stable 1.4.18

The fix for the space parsing in the headers that I made of 1.4.17 was not complete, because it results in negative header lengths being returned for headers that are exclusively composed of spaces. I have checked the whole code to see if this can have any nasty effect, and I couldn't find one, since everytime, we check the length before the contents (we're saved by an optimization). Still, I don't like having dangerous code lie around, especially in stable versions. I know for instance that some people apply custom patches on top of it and may get trapped. So i have issued 1.4.18 with that fix. I also included the recent patch from Finn Arne Gangstad to split domain names on ':' too, as I agree that whenever a port is specified, the host name cannot easily be checked. And I added a match for header length so that it's now a lot easier to check for an empty header. The rest are just usual doc and halog updates. I don't think there is any specific reason to rush on this new version, but if you're in the process of upgrading an older one, please avoid 1.4.17 and use 1.4.18 instead.

September 10th, 2011 : Development 1.5-dev7

Five months have elapsed since 1.5-dev6. A massive amount of changes was merged since then. Most of them were cleanups and optimizations. A number of changes were dedicated to making listeners more autonomous. The immediate effect is a more robust handling of resource saturation, and the second effect is the removal of the 10-years old maintain_proxies() function which was harming performance and hard to get over. Halog was improved too (faster with more filters). A significant number of external contributions were merged, among them the stats socket updates to clear session-table keys by values. There are too many changes to list, but nothing too dangerous, so I'd say it's the 1.5-dev version I trust the most today. Please give it a test.

September 5th, 2011 : Stable 1.4.17

Last week an issue was discovered with an application emitting spaces after the content-length value, which caused haproxy to report an error when parsing it. After some checks, it appeared that haproxy ought to ignore these spaces, so this was addressed. It was an opportunity to improve invalid request and responses captures, so that any message rejected for its malformation can be captured. A new minor feature making the X-Forwarded-For header addition conditional was added because users had to resort to complex tricks to do that. Last, halog was updated to latest version. Due to the issue with the header above, I released 1.4.17. Quite frankly most users don't need to upgrade. However it's better to use this one for new deployments.

August 6th, 2011 : Stable 1.3.26 and status updates

Previous 1.3 version was released 14 months ago, the same day as 1.4.8. Since then, a number of fixes went into 1.4 and a part of them were queued for 1.3. These fixes are not *that much* important but are still worth a release. Thus, both 1.3.26 and 1.3.15.13 were released and are available as source and precompiled binaries for Linux/x86 and Solaris/sparc.

I realized that I don't use 1.3 anymore myself, and for this reason I'm afraid of the risk of introducing regressions with future backports. So I decided that it was time to turn 1.3 into a "critical fixes only" status after 2.5 years of stable releases and 5 years of existence, meaning that minor fixes will probably never get there anymore, and that future releases, if any, will be focused on important bugs. That does not mean it's not supported anymore, but that fixes will come at a very slow pace and that new deployments are encouraged to use 1.4 if they expect a responsive support.

I'm also switching the 1.3.14 and 1.2 branches to the "unmaintained" status since nobody appears to be using them anymore (last fixes were more than 2 and 3 years ago respectively).

August 5th, 2011 : Stable 1.4.16

Since 1.4.15 was released 2 months ago, very few minor bugs were detected. They were so minor that it was worth waiting for other ones to be found, but after some time, there wasn't any point making users wait any more, so I released 1.4.16. A few minor improvements were also made based on feedback from users. Among the changes, MySQL checks now support Mysqld versions after 5.5, health checks support for multi-packet response has been fixed, the HTTP 200 status can be configured for monitor responses, a new http-no-delay option has been added to work around buggy HTTP implementations that assume packet-based transfers, chunked-encoded transfers have been optimised a bit, the stats interface now support URL-encoded forms, and halog correctly handles truncated files. There is no real emergency to upgrade.

June 7th, 2011 : Country IP Blocks needs help

Quite a few HAProxy users rely on geolocation lists freely provided by Country IP blocks, either to add a request header indicating the origin country, or to select the datacenter closest to the client. Now this nice service needs some money to continue operations otherwise they're forced to close. They're asking for donations. Their service has been offered for free with a high quality to many HAProxy users for some time now, I think it would really be fair that these users in turn help their nice provider. They need $2000 before next week, this certainly is achievable if all big site using their lists donate $100 each to keep them alive. Never forget that for any free software or service on the Net, there are always people working hard to keep the service alive and who have to pay bills at the end of the month.

May 31st, 2011 : HAProxy participates to IPv6 day

About two weeks ago I registered to participate to the World IPv6 day. The concept is very simple : on June 8th, many web sites will be available both in IPv4 and IPv6. Why only that day ? Because there exists some places where IPv6 can be resolved but not reached, causing the dual-stack sites to be unreachable from these places. By having many sites running IPv6 on the same day, network admins will notice the problem comes from their site and not from the outside since many sites will be unreachable at the same time.

HAProxy was running dual-stack a few years ago but I had to revert this due to many problem reports. Still some visitors might have noticed the little green image indicating to them if their browser can connect to IPv6. Since I noticed on the participants list that some sites were already running with dual-stack enabled, I decided to enable it here again in advance, so that I'll be able to revert it in case some visitors report any issue. If no issue is reported until June 8th, I'll probably leave that enabled.

Unfortunately, the Dedibox serving as a cache for the web site is in a network that is not yet IPv6-enabled. That's really a shame, considering that we upgraded it from an old box that was on an IPv6-capable network. I really don't understand what's happening at Free for taking that long a time to enable IPv6 on all their network segments, it does not seem to be on their top priority list. But since the site is running at home behind my Nerim internet access which has been IPv6-enabled for something like 10 years now, I could announce the ADSL endpoint address in the DNS.

Enabling IPv6 on your web site really is trivial with HAProxy. You just have to add "bind :::80" to your frontend and announce the IPv6 address as an AAAA record in your DNS zone, and that's all. No readdressing, no routing changes, nothing fancy. And you can even get IPv4/IPv6 statistics like here. BTW, I know for sure that some of the World IPv6 Day participants have done exactly that with their HAProxy config too :-)

Apr 8th, 2011 : stable 1.4.15 & 1.5-dev6

Two annoying bugs were detected on 1.4 at Exosec, one week apart. The first one limits the usable content-length to 32-bit on 32-bit platforms, despite the efforts made in the code to support 64-bit quantities everywhere. It was then fixed in 1.4.14. Yesterday, while working on the backport of 1.4 fixes to 1.3, I spotted that the patch to fix the issues with spaces in cookies that was merged in 1.4.9 introduced a regression due to a typo. In some circumstances, a malformed header sent by the server can crash haproxy when cookie-based persistence is enabled. Thus 1.4.15 was released as an emergency update to address this. The bug has never been reported because it's extremely unlikely to appear, unless a server tries to provoke it on purpose.

In the mean time, 1.5-dev4 was released with a huge amount of fixes and architectural reorganizations (too many to list here), which were needed to continue the work towards server-side keep-alive. 1.5-dev5 enabled server-side IPv6 support and fixed a number of remaining bugs. 1.5-dev6 was finally released to address the last regressions reported on the list yesterday as well as the important bug above.

Now, everyone should have understood that all users of 1.4 >= 1.4.9 or 1.5 > 1.5-dev3 must upgrade.

Please consult the 1.4 CHANGELOG and the 1.5 CHANGELOG for more information.

Mar 9th, 2011 : stable 1.4.13

Many annoying bugs were discovered both when working on 1.5-dev and by users. Some headers were not correctly processed after removal of the last header (issue reported to the list by Stefan Behte), disabling a disabled proxy from the CLI could result in a segfault (reported by Bryan Talbot, fixed by Cyril Bont), "balance url_param" was completely broken on POST requests (reported by Bryan Talbot too), it is theorically possible to get HTTP chunk size wrong if only the CR is sent as the last byte of the buffer, waiting for the LF to wrap around in a subsequent packet, ACLs loaded from a file did not correctly close the file descriptor upon success (reported by Bertrand Jacquin), the recently added srv_id ACL could segfault if the server is not known (reported by Herv Commowick), rlimits were not correctly updated for listening sockets (reported by the loadbalancer.org team), the stats page in admin mode did not support multi-packet requests (fixed by Cyril).

1.4.12 was released with all those fixes, and Hank A. Paulson reported a crash with pattern files with empty lines caused by a regression introduced into 1.4.11 by a fix for correctly handling empty lines. So 1.4.13 was released a few hours later to avoid any issue.

I'd like to thank all of these contributors, because well-detailed bug reports are equally important as code contributions. Once again, all users of 1.4 are encouraged to upgrade in order to avoid boring troubleshooting of stupid bugs. This time I have added Sparc builds too, as there are still requests for those. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Feb 10th, 2011 : stable 1.4.11

While working on keep-alive on 1.5, several issues were discovered, some of which were found to also affect 1.4. For this reason, I preferred to delay the next 1.4 release to until 1.5-dev4 was complete but development has recently stalled, so I preferred to release 1.4.11 anyway. One bug is tagged as critical because it can cause a session to remain indefinitely upon certain conditions that are hard but not impossible to trigger when a server dies. A bug in the ebtree code could cause stick tables to not always match arbitrary length keys. Cyril Bont has definitely fixed the http-pretend-keepalive option to correctly handle the situations where it was combined with httpclose or in tunnel mode. Until now, it was common to see the client wait for the server to close the connection before returning, causing very bad performance. Since all combinations were extensively tested, I think we should be OK now. Some error conditions were fixed to report correct flags in the logs (eg: client aborts in the middle of HTTP trunking used to report chunking errors). There was an issue with server connection error processing which prevented pending connections from being processed when maxconn was set to 1, because the current connection was counted in them. The error capture from the stats socket is now also able to report incorrectly chunked data. This helps troubleshooting faulty applications. Also the error captures now include an error counter to ease processing with external monitoring scripts. Joe Williams added a global "log-send-hostname" statement which makes it possible to pass the hostname field in the emitted syslog messages. Other various minor improvements on the config parser were merged too.

With all that, all users of 1.4 are strongly encouraged to upgrade. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Nov 11th, 2010 : devel 1.5-dev3

Haproxy 1.5-dev3 was released with everything that went into 1.4.9, plus some added bonus that were mainly developped at Exceliance, among which support for binding to UNIX socket on the accept side so that Haproxy can now receive connections over a UNIX socket. This is particularly useful when combined with stunnel, for which a patch is also available. The new PROXY protocol was implemented in order to permit stunnel to forward transport-level information to haproxy, such as the protocol, source and destinations of an incoming connection, so that haproxy can make use of that everywhere internally (acls, logs, transparent, ...) instead of stunnel's address. The main advantage over the x-forwarded-for patch is that it now supports keep-alive and is not limited to HTTP anymore. When combined with the UNIX socket, it can make haproxy and stunnel integrate seamlessly and reliably, provided that this patch is applied to stunnel. Stick tables can now learn from responses, which enables SSL-ID stickiness. And more importantly, stick-tables can now be synchronized in a multi-master fashion between multiple haproxy instances. Also, during soft-restarts, the new process learns the table from the old one so that restarts do not lose that precious information anymore. This tough task was the second half of the large work co-sponsored by Exceliance and LoadBalancer.org.

Oct 29th, 2010 : stable 1.4.9

Four months after 1.4.8 was released, some minor issues have accumulated and a new release was necessary. It was also an opportunity to add some long-awaited minor feature improvements.

Among the issues that were fixed, a listener could be left in an unrecoverable state in case of memory shortage during an accept(). POST requests that were followed by a CRLF (forbidden) in a late packet could cause some TCP resets to be emitted on Linux due to those two unread bytes (diagnosed with Dietrich Hasselhorn). Servers that were disabled while processing requests could still drain new requests from the global queue. HTTP header handling for ACLs did not correctly consider quotes and used to consider commas within quotes as a list delimitor. A server with address 0.0.0.0 used to rely on the system to connect to this address (which is always itself). Now it forwards the connection the same way as in the transparent mode. Various error reports and logs were fixed or improved, and many doc typos were fixed.

Now concerning the improvements, Krzysztof Oledzki improved his netsnmp-perl plugin to support listening sockets, and Mathieu Trudel's Cacti templates were merged. Judd Montgomery and Cyril Bont's work to support setting servers up/down from the stats interface has been merged too. Gabor Lekeny added LDAPv3 health checks. Herv Commowick improved the MySQL check to support a complete login sequence with a real username. When option "abortonclose" is set and a client disconnects while waiting for the server, now we forward the close notification to the server. That way the server can decide whether to continue or close. This is important for servers dealing with long polling requests. The Explicit Content Validation (ECV) check code was finally merged after 18 months of reviewd and fixes by various people. That was one major cause for delaying this release. Health checks can now rely on a string that is looked up in server responses. Persistence cookies now support inactivity timeouts and time to live. This is needed with some new terminals such as iPhones where the browser is never closed and the terminal sticks to the same server forever (which is particularly undesired during a partial outage). Also, we now have a new "preserve" option for cookies in "insert" mode, which indicate that if the server sets the cookie, then we let it pass unaffected. This allows servers to terminate persistence upon logout. Last, the "halog" utility was improved to support per-url and per-termination code statistics. This means that it's now trivial to know what URLs take the most processing time.

Version 1.4.9 was released with all that, with sources and Linux binaries at the usual places. Some of these fixes will slip into 1.3 too.

Oct 23th, 2010 : new httperf results : 572000 reqs/s

This morning I came across this interesting post from Kristian Lyngstol about the performance tests he ran on the Varnish cache. What struck me was the number of requests per second Kristian managed to reach : 275000, not less. I'm not surprized at all that Varnish can withstand such high rates, it's known for being very fast. My surprize came from the fact that Kristian managed to find fast enough tools to run this test. My old injector is limited to around 100k requests per second on my machines, as it does not support keep-alive, and Apache's ab to around 150k with keep-alive enabled. And when I managed to reach 2 millions requests per second, I was feeding a constant stream of pipelined requests with netcat, which is particularly inconvenient to use.

Kristian said he used httperf. I tried it in the past but did not manage to get good numbers out of it. He said he found some "httperf secrets", so that made me want to try again. First tests were limited to approximately 50000 requests per second with httperf at 100% CPU. Something close to my memories. But reading the man, I found that httperf can work in a session-based mode with the "--wsess" parameter, where it also support HTTP pipelining. Hmmm nice, we'll be less sensible to packet round-trips :-) So I tried again with haproxy simply doing redirects. Performance was still limited to 50000 requests per second.

In fact, there appears to be a default limit of 50000 requests per second when "--rate" is not specified. I set it to 1 million and ran the test again. Result: about 158000 requests per second at 100% CPU and with haproxy at 44%. Since my machine is a Core2 Quad at 3 GHz, I fired 3 httperf against one haproxy process. The load reached a max of 572000 requests/s with an average around 450000 requests per second. This time, haproxy and all 3 httperf were using 100% CPU. What an improvement!

These tests mean nothing at all for real world uses of course, because when you have many clients, they won't send you massive amounts of pipelined requests. However it's very nice to be able to stress-test the HTTP engine for regression testing. And this will be an invaluable measurement tool to test the end-to-end keep-alive when it's finished. I still have to figure out the meaning of some options and how to make the process less verbose. Right