java.lang.NullPointerException

JIRA | Michael Magin | 10 years ago
  1. 0

    At the time I terminated the crawl via the UI, this appeared in heritrix_out: 10/24/2006 15:18:35 +0000 SEVERE org.archive.crawler.framework.ToeThread run Fatal exception in ToeThread #131: http://adserver1.christianitytoday.com/bservers//pageid=330330096/acc_rando m=691830669/AAMB1/site=music/area=article/position=top_1/guid=0008B29029790 5351347729361626364 java.lang.NullPointerException at org.archive.crawler.frontier.RecoveryJournal.writeLine(RecoveryJournal.java :190) at org.archive.crawler.frontier.RecoveryJournal.finishedSuccess(RecoveryJourna l.java:154) at org.archive.crawler.frontier.RecoveryJournal.finishedSuccess(RecoveryJourna l.java:146) at org.archive.crawler.frontier.AbstractFrontier.doJournalFinishedSuccess(Abst ractFrontier.java:394) at org.archive.crawler.frontier.WorkQueueFrontier.finished(WorkQueueFrontier.j ava:865) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:159) Disk was not full or broken. I didn't notice any other negative side-effects of this. (end-of-crawl reports were generated correctly) --- Gordon commented in email: I only see one place where RecoveryJournal's 'out' field is nulled: its close() method, line 506. So it's likely that another thread closed the journal before this ToeThread wrote to it. If the problem were that the RecoveryJournal was closed while threads were obviously still in-progress, I would expect to see this all the time -- the surprise here is then would be that it's not more common. Since it's not, perhaps it's a different kind of timing issue -- some thread starting a new URI at a moment where the terminate thinks the thread has been successfully held.

    JIRA | 10 years ago | Michael Magin
    java.lang.NullPointerException
  2. 0

    NullPointerException at org.archive.crawler.processor.recrawl.PersistLogProcessor.finalTasks(PersistLogProcessor.java:87) 03/09/2009 17:07:46 +0000 INFO org.archive.crawler.admin.CrawlJob postDeregister org.archive.crawler:host=crawling10.us.archive.org,jmxport=9093,mother=h1236289378518,name=1104-20090309170725217,type=CrawlService.Job unregistered from MBeanServerId=crawling10.us.archive.org_1236143748023, SpecificationVersion=1.4, ImplementationVersion=1.6.0_03-b05, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #75: " java.lang.NullPointerException at org.archive.crawler.processor.recrawl.PersistLogProcessor.finalTasks(PersistLogProcessor.java:87) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Exception in thread "ToeThread #63: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #61: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #64: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #60: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031)

    JIRA | 8 years ago | Noah Levitt
    java.lang.NullPointerException
  3. 0

    Ftp entries in an arc file look like this currently: ftp://ftp.ksl.stanford.edu/welcome.msg 171.64.71.195 20081121190026 no-type 56 ***** ***** Stanford Knowledge Systems Laboratory ***** There is no header, only body content. When heritrix encounters an error trying to download a file, for example: 550 foo: Permission denied. it throws an exception which propagates to the logs: 11/21/2008 19:00:45 +0000 SEVERE org.archive.crawler.fetcher.FetchFTP innerProcess FTP server reported problem. org.archive.net.FTPException: FTP error code: 550 at org.archive.net.ClientFTP.openDataConnection(ClientFTP.java:130) at org.archive.crawler.fetcher.FetchFTP.fetch(FetchFTP.java:312) at org.archive.crawler.fetcher.FetchFTP.innerProcess(FetchFTP.java:252) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Heritrix still tries to write to the ARC, but fails because there is no content: 11/21/2008 19:00:45 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem java.lang.NullPointerException occured when trying to process 'ftp://ftp.ksl.stanford.edu/dev/ticotsord' at step ABOUT_TO_BEGIN_PROCESSOR in Archiver java.lang.NullPointerException at org.archive.crawler.writer.ARCWriterProcessor.innerProcess(ARCWriterProcessor.java:122) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) So there is no record in the arc file at all. But this "550 foo: Permission denied." is essentially equivalent to a HTTP 403. It should be archived somehow and should not spew stack traces in the logs. So I propose we include a "header" section in the arc for ftp transactions. "550 foo: Permission denied." would go there. On a successful get, the message would be something like "150 Binary data connection for /welcome.msg (76.103.251.45,57342) (56 bytes)." Would this break anything?

    JIRA | 8 years ago | Noah Levitt
    java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    The following exception stack occurred when terminating a small test crawl via the web UI. A subsequent crawl terminated normally on same settings. com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN occured when trying to process 'http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif' at step ABOUT_TO_BEGIN_PROCESSOR in HTTP com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread run Fatal exception in ToeThread #29: http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif java.lang.NullPointerException at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)

    JIRA | 9 years ago | Kristinn SigurĂ°sson
    java.lang.NullPointerException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.NullPointerException

      No message provided

      at org.archive.crawler.frontier.RecoveryJournal.writeLine()
    2. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.frontier.RecoveryJournal.writeLine(RecoveryJournal.java:190)
      2. org.archive.crawler.frontier.RecoveryJournal.finishedSuccess(RecoveryJournal.java:154)
      3. org.archive.crawler.frontier.RecoveryJournal.finishedSuccess(RecoveryJournal.java:146)
      4. org.archive.crawler.frontier.AbstractFrontier.doJournalFinishedSuccess(AbstractFrontier.java:394)
      5. org.archive.crawler.frontier.WorkQueueFrontier.finished(WorkQueueFrontier.java:865)
      6. org.archive.crawler.framework.ToeThread.run(ToeThread.java:159)
      6 frames