java.util.ConcurrentModificationException

Hibernate JIRA | Zacharias J. Beckman | 1 decade ago
  1. 0

    When 'debug' level logging is turned on we get ConcurrentModificationExceptions during a session.delete(object) call. The problem goes away if we turn off 'debug' logging (degrading it to 'warn,' for example). This is 100% reproducible here. Following is a cut-and-paste from our own JIRA issue tracking system: ISSUE: Persister.delete() method causes ConcurrentModificationException & server 'packs up' There is a consistently reproduced problem with the Hibernate tier code (either something to do with Persister.java or with the way we are using Hibernate, hopefully not with Hibernate itself). The problem is easily reproducible on the IS but much less reproducible on my own development server. (My development server has a slightly different configuration -- it is possible that it is more "properly" configured than the IS but that seems an unlikely cause). When running integration tests, the IS will fail on the testDeleteDocument test. The failure throws a ConcurrentModificationException from within Hibernate, as called by Persister.delete() on an object. Once the CME is thrown, the server pretty well packs up -- it is no longer able to get a good database connection (transaction level failures, "net.sf.hibernate.TransactionException: Could not register Synchronization." The server appears to recover after a period of time (probably timing something out), but the only quick solution is a server reboot. The client side stack trace of the CME exception is attached in a comment, below. Note that these tests work fine on my development configuration. My developoment configuration is slightly more "textbook" than the IS server, which is based on the JBoss default configuration. I can't account for any reason it would behave differently. One possible step to resolve would be to use the "textbook" configuration on the integration server. --- COMMENT: The following is a consistently reproduced stack trace on the IS. This bug does not appear on my development server: [junit] Testcase: testDeleteDocument took 0.092 sec [junit] Caused an ERROR [junit] null [junit] java.util.ConcurrentModificationException [junit] at java.util.HashMap$HashIterator.nextEntry(HashMap.java:782) [junit] at java.util.HashMap$ValueIterator.next(HashMap.java:812) [junit] at net.sf.hibernate.impl.Printer.toString(Printer.java:82) [junit] at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2228) [junit] at net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2186) [junit] at net.sf.hibernate.transaction.JTATransaction.commit(JTATransaction.java:52) [junit] at com.legaledge.harmony.model.objects.DocumentTest.tearDown(Unknown Source) [junit] at org.apache.cactus.ServletTestCase.runCactusTest(ServletTestCase.java:295) [junit] at org.apache.cactus.ServletTestCase.runBare(ServletTestCase.java:263) [junit] at org.apache.cactus.server.AbstractWebTestCaller.doTest(AbstractWebTestCaller.java:156) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody0(AbstractWebTestController.java:130) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody1$advice(AbstractWebTestController.java:151) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest(AbstractWebTestController.java) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody2(ServletTestRedirector.java:138) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody3$advice(ServletTestRedirector.java:151) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost(ServletTestRedirector.java) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody0(ServletTestRedirector.java:109) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody1$advice(ServletTestRedirector.java:151) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet(ServletTestRedirector.java) [junit] at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) [junit] at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) [junit] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247) [junit] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193) [junit] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417) [junit] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197) [junit] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781) [junit] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549) [junit] at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605) [junit] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677) [junit] at java.lang.Thread.run(Thread.java:534) --- COMMENT: Further experimentation found the apparent cause of this problem. It remains a problem but we've got a workaround. Someone on the Hibernate lists had mentioned that they ran into CME problems when Hibernate was put into 'debugging' log level. This, they explained, happens because Hibernate becomes very chatty and iterates over all manner of things, printing out values for the log messages. Apprently, it iterates over a list that is not supposed to be iterated (probably the list from which an object is deleted). This causes a ConcurrentModificationException to be thrown. I have changed the IS logging level to "WARN" on the Hibernate logs. This will curtail the amount of information available, but it seems to be a valid workaround for now. I'll report the error on Hibernate, as well, and hopefully someone will fix it. Unfortunately, this means that putting Hibernate into 'debug' log level will actually cause more bugs in the system. It might make 'debug' logging in Hibernate very difficult to use...

    Hibernate JIRA | 1 decade ago | Zacharias J. Beckman
    java.util.ConcurrentModificationException
  2. 0

    When 'debug' level logging is turned on we get ConcurrentModificationExceptions during a session.delete(object) call. The problem goes away if we turn off 'debug' logging (degrading it to 'warn,' for example). This is 100% reproducible here. Following is a cut-and-paste from our own JIRA issue tracking system: ISSUE: Persister.delete() method causes ConcurrentModificationException & server 'packs up' There is a consistently reproduced problem with the Hibernate tier code (either something to do with Persister.java or with the way we are using Hibernate, hopefully not with Hibernate itself). The problem is easily reproducible on the IS but much less reproducible on my own development server. (My development server has a slightly different configuration -- it is possible that it is more "properly" configured than the IS but that seems an unlikely cause). When running integration tests, the IS will fail on the testDeleteDocument test. The failure throws a ConcurrentModificationException from within Hibernate, as called by Persister.delete() on an object. Once the CME is thrown, the server pretty well packs up -- it is no longer able to get a good database connection (transaction level failures, "net.sf.hibernate.TransactionException: Could not register Synchronization." The server appears to recover after a period of time (probably timing something out), but the only quick solution is a server reboot. The client side stack trace of the CME exception is attached in a comment, below. Note that these tests work fine on my development configuration. My developoment configuration is slightly more "textbook" than the IS server, which is based on the JBoss default configuration. I can't account for any reason it would behave differently. One possible step to resolve would be to use the "textbook" configuration on the integration server. --- COMMENT: The following is a consistently reproduced stack trace on the IS. This bug does not appear on my development server: [junit] Testcase: testDeleteDocument took 0.092 sec [junit] Caused an ERROR [junit] null [junit] java.util.ConcurrentModificationException [junit] at java.util.HashMap$HashIterator.nextEntry(HashMap.java:782) [junit] at java.util.HashMap$ValueIterator.next(HashMap.java:812) [junit] at net.sf.hibernate.impl.Printer.toString(Printer.java:82) [junit] at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2228) [junit] at net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2186) [junit] at net.sf.hibernate.transaction.JTATransaction.commit(JTATransaction.java:52) [junit] at com.legaledge.harmony.model.objects.DocumentTest.tearDown(Unknown Source) [junit] at org.apache.cactus.ServletTestCase.runCactusTest(ServletTestCase.java:295) [junit] at org.apache.cactus.ServletTestCase.runBare(ServletTestCase.java:263) [junit] at org.apache.cactus.server.AbstractWebTestCaller.doTest(AbstractWebTestCaller.java:156) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody0(AbstractWebTestController.java:130) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody1$advice(AbstractWebTestController.java:151) [junit] at org.apache.cactus.server.AbstractWebTestController.handleRequest(AbstractWebTestController.java) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody2(ServletTestRedirector.java:138) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody3$advice(ServletTestRedirector.java:151) [junit] at org.apache.cactus.server.ServletTestRedirector.doPost(ServletTestRedirector.java) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody0(ServletTestRedirector.java:109) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody1$advice(ServletTestRedirector.java:151) [junit] at org.apache.cactus.server.ServletTestRedirector.doGet(ServletTestRedirector.java) [junit] at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) [junit] at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) [junit] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247) [junit] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193) [junit] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417) [junit] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174) [junit] at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) [junit] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) [junit] at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) [junit] at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197) [junit] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781) [junit] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549) [junit] at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605) [junit] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677) [junit] at java.lang.Thread.run(Thread.java:534) --- COMMENT: Further experimentation found the apparent cause of this problem. It remains a problem but we've got a workaround. Someone on the Hibernate lists had mentioned that they ran into CME problems when Hibernate was put into 'debugging' log level. This, they explained, happens because Hibernate becomes very chatty and iterates over all manner of things, printing out values for the log messages. Apprently, it iterates over a list that is not supposed to be iterated (probably the list from which an object is deleted). This causes a ConcurrentModificationException to be thrown. I have changed the IS logging level to "WARN" on the Hibernate logs. This will curtail the amount of information available, but it seems to be a valid workaround for now. I'll report the error on Hibernate, as well, and hopefully someone will fix it. Unfortunately, this means that putting Hibernate into 'debug' log level will actually cause more bugs in the system. It might make 'debug' logging in Hibernate very difficult to use...

    Hibernate JIRA | 1 decade ago | Zacharias J. Beckman
    java.util.ConcurrentModificationException
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Nodes in cluster somtimes hang

    Google Groups | 9 years ago | svo...@tiscali.dk
    java.util.ConcurrentModificationException
  5. 0

    Jetty - Java HTTP Servlet Server / Mailing Lists

    sourceforge.net | 1 year ago
    java.util.ConcurrentModificationException

    4 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.ConcurrentModificationException

      No message provided

      at java.util.HashMap$HashIterator.nextEntry()
    2. Java RT
      HashMap$ValueIterator.next
      1. java.util.HashMap$HashIterator.nextEntry(HashMap.java:782)
      2. java.util.HashMap$ValueIterator.next(HashMap.java:812)
      2 frames
    3. net.sf.hibernate
      JTATransaction.commit
      1. net.sf.hibernate.impl.Printer.toString(Printer.java:82)
      2. net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2228)
      3. net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2186)
      4. net.sf.hibernate.transaction.JTATransaction.commit(JTATransaction.java:52)
      4 frames
    4. com.legaledge.harmony
      DocumentTest.tearDown
      1. com.legaledge.harmony.model.objects.DocumentTest.tearDown(Unknown Source)
      1 frame
    5. org.apache.cactus
      ServletTestRedirector.doGet
      1. org.apache.cactus.ServletTestCase.runCactusTest(ServletTestCase.java:295)
      2. org.apache.cactus.ServletTestCase.runBare(ServletTestCase.java:263)
      3. org.apache.cactus.server.AbstractWebTestCaller.doTest(AbstractWebTestCaller.java:156)
      4. org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody0(AbstractWebTestController.java:130)
      5. org.apache.cactus.server.AbstractWebTestController.handleRequest_aroundBody1$advice(AbstractWebTestController.java:151)
      6. org.apache.cactus.server.AbstractWebTestController.handleRequest(AbstractWebTestController.java)
      7. org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody2(ServletTestRedirector.java:138)
      8. org.apache.cactus.server.ServletTestRedirector.doPost_aroundBody3$advice(ServletTestRedirector.java:151)
      9. org.apache.cactus.server.ServletTestRedirector.doPost(ServletTestRedirector.java)
      10. org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody0(ServletTestRedirector.java:109)
      11. org.apache.cactus.server.ServletTestRedirector.doGet_aroundBody1$advice(ServletTestRedirector.java:151)
      12. org.apache.cactus.server.ServletTestRedirector.doGet(ServletTestRedirector.java)
      12 frames
    6. JavaServlet
      HttpServlet.service
      1. javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
      2. javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
      2 frames
    7. Glassfish Core
      StandardPipeline$StandardPipelineValveContext.invokeNext
      1. org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
      2. org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
      3. org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
      4. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      5. org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      6. org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      7. org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      8. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      8 frames
    8. JBoss Application Server Tomcat
      JBossSecurityMgrRealm.invoke
      1. org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
      1 frame
    9. Glassfish Core
      StandardPipeline$StandardPipelineValveContext.invokeNext
      1. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      2. org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
      3. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      3 frames
    10. org.jboss.web
      ContainerStatsValve.invoke
      1. org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
      1 frame
    11. Glassfish Core
      StandardPipeline$StandardPipelineValveContext.invokeNext
      1. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      2. org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      3. org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      4. org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
      5. org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
      6. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      7. org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
      8. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      9. org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
      10. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      10 frames
    12. JBoss Application Server Tomcat
      SecurityAssociationValve.invoke
      1. org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
      1 frame
    13. Glassfish Core
      ContainerBase.invoke
      1. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      2. org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
      3. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
      4. org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      5. org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      6. org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
      7. org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
      8. org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
      9. org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
      9 frames
    14. org.apache.coyote
      CoyoteAdapter.service
      1. org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:197)
      1 frame
    15. Grizzly HTTP
      TcpWorkerThread.runIt
      1. org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781)
      2. org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
      3. org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605)
      3 frames
    16. Tomcat Util
      ThreadPool$ControlRunnable.run
      1. org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
      1 frame
    17. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:534)
      1 frame