haproxy and tomcat intermittent hangs

I am trying to run haproxy in front of tomcat on a Solaris x86 box, but I am getting intermittent failures. At seemingly random intervals, the request just hangs until haproxy times out the connection.

I thought maybe it was my app, but I've been able to reproduce it with the tomcat manager app, and hitting tomcat directly there is no problems at all.

Hitting it repeatedly with curl will cause the error within 10-15 tries

  curl -ikL http://admin:admin@<my server>:81/manager/status

haproxy is running on port 81, tomcat on port 7000. haproxy returns a 504 gateway timeout to the client, and puts this into the log file:

Sep  7 21:39:53 localhost haproxy[16887]: xxx.xxx.xxx.xxx:65168 [07/Sep/2009:21:39:23.005] http_proxy http_proxy/tomcat7000 5/0/0/-1/30014 504 194 - - sHNN 0/0/0/0/0 0/0 "GET /manager/status HTTP/1.1"

Tomcat shows nothing, no error in the logs and no indication that the request ever makes it to the tomcat server. The request count is not incremented, the manager app only shows activity on one thread, serving up the manager app.

Here are my haproxy and tomcat connector settings, I've been playing with both a good deal trying to chase down the issue, so they may not be ideal, but they definitely don't seem like they should cause this error.

server.xml

<Connector 
    port="7000" protocol="HTTP/1.1"
    enableLookups="false" maxKeepAliveRequests="1"
    connectionLinger="10"
/>

haproxy config

global
  log loghost local0
  chroot /var/haproxy

listen http_proxy :81
  mode http
  log global
  option httplog
  option httpclose
  clitimeout  150000
  srvtimeout  30000
  contimeout  3000 
  balance roundrobin
  cookie SERVERID insert
  server tomcat7000 127.0.0.1:7000 cookie server00 check inter 2000

From serverfault Lorin

• If you turn on the haproxy statistics server, the extra information may help you debug

  • add "stats enable" to your "listen" section
  • browse to http://your-proxy:81/haproxy?stats
  Lorin : Hey, thanks for responding. The only thing I was really able to see from that was the number of requests that failed, and that haproxy thinks it is failing in the response, which is odd since tomcat never gets the request. Is there something in particular I should look for?
  From Casey

• Well, you should run tcpdump between haproxy and netcat to figure out what's happening. Haproxy says it did not get the response, which may be discuted, but it is clear it has connected and sent the request otherwise it could not be waiting for a header. The fact that tomcat does not see any incoming request is troubling, because at least I could have accepted a bug on haproxy causing it to timeout on responses, but here we're saying that tomcat has no reason to respond because it does not consider it got a request. Either there's a bug on tomcat itself, or something strange on your network (two servers on the same IP ?), because at least it should tell you that it received the connection and the request! Tcpdump will sort that out pretty easily. Also, what version of haproxy are you running on ? versions between 1.3.16 and 1.3.19 had several issues among which a random timeout occurring on the last packet of response (but here tomcat says it has not responded, but anyway it's better to have things fixed).

  Willy

  From $pageUsers[$post.author].name