Wednesday, January 19, 2011

Unusual Apache->Tomcat caching issue.

Right now, I have an Apache setup sitting in front of Tomcat to handle caching. This setup has been given to an external service to manage, and since the transition, I've noticed odd behavior. Specifically, when I request a swf file from the web server, I hit the Apache cache (good), but occasionally I'll receive a truncated file. Once I receive this truncated file, the cache will NOT refresh until I manually delete the cache and let the swf pull down from tomcat again.

The external service claims that the configuration is fine, but I don't see any way this could be happening aside from improper configuration. Now, there are two apache and two tomcat servers under a load balancer, and occasionally one apache cache will break while another does not (leading to 50% of all requests getting bad, truncated data).

Where should I start looking to debug this issue? What could POSSIBLY be causing this odd behavior?

Edit: Inspecting the logs, tomcat throws this:

java.io.IOException: Bad file number
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:199)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.FilterInputStream.read(FilterInputStream.java:90)
        at org.apache.catalina.servlets.DefaultServlet.copyRange(DefaultServlet.java:1968)
        at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:1714)
        at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:809)
        at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:325)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:209)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
        at org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.invoke(SessionValve55.java:57)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:619)

followed by 

access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:00:27:32 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:01:27:33 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:01:39:53 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:02:27:38 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -

So apache is caching the bad file size. What could possibly be causing this, and possibly separate, how do I ensure that this exception does not get written to cache?

  • wget, curl or file save as this file?

    Do other large files work?

    If there's a support contract, I would consider requesting the vendor debug this collaboratively.

    Barry : This seems to be the answer: http://mail-archives.apache.org/mod_mbox/tomcat-dev/200808.mbox/%3Cbug-45601-78@https.issues.apache.org/bugzilla/%3E
    iftrue : Is that still not fixed in 6.0.20?
    iftrue : wget and save as both pull down the cached apache file and fail. There aren't many other files on the server that get hit often as large as this file (in fact this file probably gets hit most frequently).
    From Barry
  • Tomcat's inability to serve large files has been fixed in trunk, 5.5.x, and the fix fixing the bug is committed to 6.0.27.

    From iftrue

0 comments:

Post a Comment