org.archive.crawler.admin.CrawlJob$MBeanCrawlController

JIRA | Gordon Mohr | 9 years ago
  1. 0

    Productions crawl (2BC) found in 'pausing' state, 7 threads still waiting on frontier lock to complete pause for days. Can't unpause. SIGQUIT thread dump reveals deadlock: Found one Java-level deadlock: ============================= "RMI TCP Connection(128)-208.70.29.245": waiting to lock monitor 0x080a7120 (object 0x9bed2a28, a org.archive.crawler.admin.CrawlJob$MBeanCrawlController), which is held by "PoolThread-1" "PoolThread-1": waiting to lock monitor 0x080a70bc (object 0x9bed3270, a org.archive.crawler.frontier.BdbFrontier), which is held by "ToeThread #127: " "ToeThread #127: ": waiting to lock monitor 0x080a7120 (object 0x9bed2a28, a org.archive.crawler.admin.CrawlJob$MBeanCrawlController), which is held by "PoolThread-1" Java stack information for the threads listed above: =================================================== "RMI TCP Connection(128)-208.70.29.245": at org.archive.crawler.framework.CrawlController.requestCrawlPause(CrawlController.java:1441) - waiting to lock <0x9bed2a28> (a org.archive.crawler.admin.CrawlJob$MBeanCrawlController) at org.archive.crawler.admin.CrawlJob.pause(CrawlJob.java:1087) at org.archive.crawler.admin.CrawlJob.invoke(CrawlJob.java:1866) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:431) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1366) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) "PoolThread-1": at org.archive.crawler.frontier.AbstractFrontier.unpause(AbstractFrontier.java:351) - waiting to lock <0x9bed3270> (a org.archive.crawler.frontier.BdbFrontier) at org.archive.crawler.framework.CrawlController.requestCrawlResume(CrawlController.java:1481) - locked <0x9bed2a28> (a org.archive.crawler.admin.CrawlJob$MBeanCrawlController) at org.archive.crawler.admin.CrawlJob.resume(CrawlJob.java:1093) at org.archive.crawler.admin.CrawlJobHandler.resumeJob(CrawlJobHandler.java:646) at org.archive.crawler.jspc.admin.console.action_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:358) at org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:342) at org.archive.crawler.admin.ui.RootFilter.doFilter(RootFilter.java:67) at org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:334) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:286) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1807) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:525) at org.mortbay.http.HttpContext.handle(HttpContext.java:1757) at org.mortbay.http.HttpServer.service(HttpServer.java:879) at org.mortbay.http.HttpConnection.service(HttpConnection.java:789) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:960) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:806) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:218) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:300) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:511) "ToeThread #127: ": at org.archive.crawler.framework.CrawlController.toePaused(CrawlController.java:1786) - waiting to lock <0x9bed2a28> (a org.archive.crawler.admin.CrawlJob$MBeanCrawlController) at org.archive.crawler.frontier.AbstractFrontier.preNext(AbstractFrontier.java:623) - locked <0x9bed3270> (a org.archive.crawler.frontier.BdbFrontier) at org.archive.crawler.frontier.WorkQueueFrontier.next(WorkQueueFrontier.java:631) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:144) Found 1 deadlock.

    JIRA | 9 years ago | Gordon Mohr
    org.archive.crawler.admin.CrawlJob$MBeanCrawlController

    Root Cause Analysis

    1. org.archive.crawler.admin.CrawlJob$MBeanCrawlController

      No message provided

      at org.archive.crawler.admin.CrawlJob.pause()
    2. org.archive.crawler
      CrawlJob.invoke
      1. org.archive.crawler.admin.CrawlJob.pause(CrawlJob.java:1087)
      2. org.archive.crawler.admin.CrawlJob.invoke(CrawlJob.java:1866)
      2 frames
    3. Java RT
      Thread.run
      1. com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
      2. com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
      3. com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:431)
      4. javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
      5. javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
      6. javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
      7. java.security.AccessController.doPrivileged(Native Method)
      8. javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1366)
      9. javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
      10. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      11. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      12. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      13. java.lang.reflect.Method.invoke(Method.java:597)
      14. sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
      15. sun.rmi.transport.Transport$1.run(Transport.java:159)
      16. java.security.AccessController.doPrivileged(Native Method)
      17. sun.rmi.transport.Transport.serviceCall(Transport.java:155)
      18. sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
      19. sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
      20. sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
      21. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      22. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      23. java.lang.Thread.run(Thread.java:619)
      23 frames