com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN

JIRA | Paul Jack | 9 years ago
  1. 0

    Running a largish crawl (10K seeds) using basic-seed-sites for a few seconds, then hit "Terminate". ToeThreads continue to process while final reports are generated: Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.ciia.org/robots.txt 200 411 text/plain Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.warchild.ca/robots.txt 404 1315 text/html Nov 15, 2007 7:04:42 PM org.archive.crawler.framework.StatisticsTracker writeReportFile INFO: wrote report: /home/pjack/Projects/builds_heritrix/jobs/active-basic_seed_sites-20071115190415/./frontier-report.txt Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.economia.cl/robots.txt 404 398 text/html; charset=iso-8859-1 Also the ToeThreads continue to process URIs even after everything is shut down: Nov 15, 2007 7:04:43 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.presidencia.gub.uy/robots.txt 404 4184 text/html Nov 15, 2007 7:04:43 PM org.archive.crawler.framework.ToeThread run SEVERE: Fatal exception in ToeThread #9: http://www.minag.gob.pe/robots.txt com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.StoredCollection.getFirstOrLast(StoredCollection.java:550) at com.sleepycat.collections.StoredSortedKeySet.last(StoredSortedKeySet.java:106) at org.archive.settings.file.PrefixFinder.last(PrefixFinder.java:91) at org.archive.settings.file.PrefixFinder.find(PrefixFinder.java:63) at org.archive.settings.file.FileSheetManager.findConfig(FileSheetManager.java:499) at org.archive.crawler.datamodel.CrawlURI.setStateProvider(CrawlURI.java:1509) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:294) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:152) All outstanding ToeThreads should finish before CrawlController.completeStop() fires.

    JIRA | 9 years ago | Paul Jack
    com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN
  2. 0

    Running a largish crawl (10K seeds) using basic-seed-sites for a few seconds, then hit "Terminate". ToeThreads continue to process while final reports are generated: Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.ciia.org/robots.txt 200 411 text/plain Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.warchild.ca/robots.txt 404 1315 text/html Nov 15, 2007 7:04:42 PM org.archive.crawler.framework.StatisticsTracker writeReportFile INFO: wrote report: /home/pjack/Projects/builds_heritrix/jobs/active-basic_seed_sites-20071115190415/./frontier-report.txt Nov 15, 2007 7:04:42 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.economia.cl/robots.txt 404 398 text/html; charset=iso-8859-1 Also the ToeThreads continue to process URIs even after everything is shut down: Nov 15, 2007 7:04:43 PM org.archive.modules.fetcher.FetchHTTP innerProcess INFO: GET http://www.presidencia.gub.uy/robots.txt 404 4184 text/html Nov 15, 2007 7:04:43 PM org.archive.crawler.framework.ToeThread run SEVERE: Fatal exception in ToeThread #9: http://www.minag.gob.pe/robots.txt com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.StoredCollection.getFirstOrLast(StoredCollection.java:550) at com.sleepycat.collections.StoredSortedKeySet.last(StoredSortedKeySet.java:106) at org.archive.settings.file.PrefixFinder.last(PrefixFinder.java:91) at org.archive.settings.file.PrefixFinder.find(PrefixFinder.java:63) at org.archive.settings.file.FileSheetManager.findConfig(FileSheetManager.java:499) at org.archive.crawler.datamodel.CrawlURI.setStateProvider(CrawlURI.java:1509) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:294) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:152) All outstanding ToeThreads should finish before CrawlController.completeStop() fires.

    JIRA | 9 years ago | Paul Jack
    com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN
  3. 0

    I started heritrix 1.14.4 on Friday morning to crawl only text/html files, beginning with yahoo.com On Monday morning i saw that heritrix just stopped crawling, heap space looks fine. The log says (last entry): 2010-12-10T18:04:49.904Z -5 - http://kr.promotion.yahoo.com/worldcup2010/ RRLLLLLLLE http://kr.sports.yahoo.com/event/wc2010/ no-type #020 - - - err=com.sleepycat.util.RuntimeExceptionWrapper com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.82) fetchTarget of 0x50/0x8590e3 parent IN=266461 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x77/0x6dcaa0 parent.getDirty()=true state=0 com.sleepycat.je.log.LogFileNotFoundException: (JE 3.3.82) 0x50/0x8590e3 (JE 3.3.82) Couldn't open file /home/oli/Desktop/heritrix-1.14.4/jobs/nixdefault-20101210100246162/state/00000050.jdb: /home/oli/Desktop/heritrix-1.14.4/jobs/nixdefault-20101210100246162/state/00000050.jdb (Too many open files) at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:466) at com.sleepycat.collections.StoredContainer.getValue(StoredContainer.java:306) at com.sleepycat.collections.StoredMap.get(StoredMap.java:227) at org.archive.util.ObjectIdentityBdbCache.getOrUse(ObjectIdentityBdbCache.java:264) at org.archive.util.ObjectIdentityBdbCache.getOrUse(ObjectIdentityBdbCache.java:75) at org.archive.crawler.datamodel.ServerCache.getServerFor(ServerCache.java:100) at org.archive.crawler.datamodel.ServerCache.getServerFor(ServerCache.java:124) at org.archive.crawler.prefetch.PreconditionEnforcer.considerDnsPreconditions(PreconditionEnforcer.java:227) at org.archive.crawler.prefetch.PreconditionEnforcer.innerProcess(PreconditionEnforcer.java:111) at org.archive.crawler.framework.Processor.process(Processor.java:109) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:306) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:154) but its the date from yesterday. I think there are some file handles left open - i use the write-to-harddisk-writer. I changed the /etc/security/limits.conf regarding the FAQ but the process is still doing nothing. It is just idle, consuming memory. when i go to "reports" in the web frontend i get: An error occured java.util.NoSuchElementException java.util.NoSuchElementException at java.util.TreeMap.key(TreeMap.java:1223) at java.util.TreeMap.firstKey(TreeMap.java:284) at java.util.TreeSet.first(TreeSet.java:394) at org.archive.crawler.framework.ToePool.singleLineReportTo(ToePool.java:276) at org.archive.util.ArchiveUtils.singleLineReport(ArchiveUtils.java:728) at org.archive.crawler.framework.ToePool.singleLineReport(ToePool.java:296) at org.archive.crawler.framework.CrawlController.oneLineReportThreads(CrawlController.java:1628) at org.archive.crawler.admin.CrawlJob.getThreadOneLine(CrawlJob.java:930) at org.archive.crawler.jspc.admin.reports_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:358) at org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:342) at org.archive.crawler.admin.ui.RootFilter.doFilter(RootFilter.java:67) at org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java:334) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:286) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1807) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:525) at org.mortbay.http.HttpContext.handle(HttpContext.java:1757) at org.mortbay.http.HttpServer.service(HttpServer.java:879) at org.mortbay.http.HttpConnection.service(HttpConnection.java:789) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:960) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:806) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:218) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:300) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:511) When i go to "Logs" they are displayed as empty. But they aren't (last entry local-errors.log): 2010-12-11T01:20:43.747Z -2 - http://www.kzone.com.au/js/prototype.js RRLLLLRE http://www.kzone.com.au/ no-type #049 - - - le:IOException@HTTP java.io.IOException: RIS already open for ToeThread #49: http://www.kzone.com.au/js/prototype.js at org.archive.io.RecordingInputStream.open(RecordingInputStream.java:88) at org.archive.util.HttpRecorder.inputWrap(HttpRecorder.java:148) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:756) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:109) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:306) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:154) I'm not sure whats going on but i think its one bug which leads to so many problems. I will test it with heritrix 3 soon.

    JIRA | 6 years ago | oliver z.
    com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.82) fetchTarget of 0x50/0x8590e3 parent IN=266461 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x77/0x6dcaa0 parent.getDirty()=true state=0 com.sleepycat.je.log.LogFileNotFoundException: (JE 3.3.82) 0x50/0x8590e3 (JE 3.3.82) Couldn't open file /home/oli/Desktop/heritrix-1.14.4/jobs/nixdefault-20101210100246162/state/00000050.jdb: /home/oli/Desktop/heritrix-1.14.4/jobs/nixdefault-20101210100246162/state/00000050.jdb (Too many open files)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Attempt to use non-open Environment object().

    Oracle Community | 1 decade ago | 535385
    com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.0.12) Attempt to use non-open Environment object().
  6. 0

    TransactionRunner that doesn't commit

    Oracle Community | 8 years ago | 665498
    com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.75) Lock expired. Locker 8893280 -1_Thread-1_ThreadLocker: waited for lock on database=storedEntriesDb LockAddr:1143493 node=23 type=READ grant=WAIT_NEW timeoutMillis=500 startTime=1235854157063 endTime=1235854157564 Owners: [<LockInfo locker="12835244 7_main_Txn" type="WRITE"/>] Waiters: []

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.sleepycat.util.RuntimeExceptionWrapper

      (JE 3.2.44) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN

      at com.sleepycat.collections.StoredContainer.convertException()
    2. Berkeley DB Java Edition
      StoredSortedKeySet.last
      1. com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447)
      2. com.sleepycat.collections.StoredCollection.getFirstOrLast(StoredCollection.java:550)
      3. com.sleepycat.collections.StoredSortedKeySet.last(StoredSortedKeySet.java:106)
      3 frames
    3. org.archive.settings
      FileSheetManager.findConfig
      1. org.archive.settings.file.PrefixFinder.last(PrefixFinder.java:91)
      2. org.archive.settings.file.PrefixFinder.find(PrefixFinder.java:63)
      3. org.archive.settings.file.FileSheetManager.findConfig(FileSheetManager.java:499)
      3 frames
    4. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.datamodel.CrawlURI.setStateProvider(CrawlURI.java:1509)
      2. org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:294)
      3. org.archive.crawler.framework.ToeThread.run(ToeThread.java:152)
      3 frames