java.lang.NullPointerException

JIRA | Michael Stack | 10 years ago
  1. 0

    Below is back and forth between Igor and Stack on an NPE found by Igor serializing out cookies. Looks like we came a cross a cookie that had a null value. No need for the patch. It is not critical to anything that we are doing. i. > Probably not a regression. I think ye've just found case where a cookie value comes back null, something we've not come across before. Doing an append with null to MutableString throws NPE: > > > > [bregeon] ~ > bsh -classpath ~/workspace/heritrix/lib/dsi.unimi.it-1.2.0.jar BeanShell 2.0b4 - by Pat Niemeyer (pat@pat.net) > bsh % new it.unimi.dsi.mg4j.util.MutableString(); > bsh % ms = new it.unimi.dsi.mg4j.util.MutableString(); > bsh % ms.append(null); > // Error: // Uncaught Exception: Method Invocation ms.append : at Line: 3 : in file: <unknown file> : ms .append ( null ) Target exception: java.lang.NullPointerExceptionjava.lang.NullPointerException > at it.unimi.dsi.mg4j.util.MutableString.append(MutableString.java:811) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 9) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at bsh.Reflect.invokeMethod(Unknown Source) > at bsh.Reflect.invokeObjectMethod(Unknown Source) > at bsh.Name.invokeMethod(Unknown Source) > at bsh.BSHMethodInvocation.eval(Unknown Source) > at bsh.BSHPrimaryExpression.eval(Unknown Source) > at bsh.BSHPrimaryExpression.eval(Unknown Source) > at bsh.Interpreter.run(Unknown Source) > at bsh.Interpreter.main(Unknown Source) > > > I'm guessing the problematic value just didn't make it into the cookie.dump. > > Want me to add patch for the NPE? > > St.Ack > > > Igor Ranitovic wrote: >> Hi Stack, >> >> In our latest IQ crawl we used save-cookies option. We have not done this in long time and it seems that we found a regression or new bug. >> At the end of the crawl, Mike notice that reports are not being generated hours after the termination. Looking at heritrix_out revealed the NPE within the Ubicrawler mutable string: >> Exception in thread "ToeThread #61: " java.lang.NullPointerException >> at it.unimi.dsi.mg4j.util.MutableString.append(MutableString.java:828) >> at org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1278) >> at org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1231) >> at org.archive.crawler.fetcher.FetchHTTP.finalTasks(FetchHTTP.java:963) >> at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlC ontroller.java:1669) >> at org.archive.crawler.framework.CrawlController.completeStop(CrawlController. java:1026) >> at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJ ob.java:793) >> at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java :1810) >> at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) >> Exception in thread "StatLogger" java.lang.NullPointerException >> at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsE vent(CrawlJob.java:763) >> at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(Abstr actTracker.java:185) >> at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(Statist icsTracker.java:303) >> at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:142) >> at java.lang.Thread.run(Thread.java:595) >> >> >> We did get some of the cookies dumped to the disk. See cookies.dump and the rest of the >> configuration files are at crawling004:/0/loc-jobs/IQ-100/ >> >> i. >

    JIRA | 10 years ago | Michael Stack
    java.lang.NullPointerException
  2. 0

    Exception in thread "StatLogger" java.lang.NullPointerException Happened in a start-then-immediate-stop crawl. The first stack trace ("Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN") is normal for this case, the others aren't. heritrix-1.15.4 from svn 2010-01-22 2010-02-20 20:39:54.911 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: addJob("http://aiapp201:6080/seam/resource/crawlRequestOrder.jar?crawlRequestId=16803", "1786", "manually crawl using 3 seeds from Tennessee Electronic Newspapers", "", ) 2010-02-20 20:39:57.230 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: startCrawling() 2010-02-20 20:39:58.348 INFO thread-12 org.archive.crawler.admin.CrawlJob.postRegister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=9093,mother=Heritrix,name=1786-20100220203955121,type=CrawlService.Job registered to MBeanServerId=crawling105.us.archive.org_1266647414512, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems 2010-02-20 20:40:25.755 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: terminateCurrentJob() 2010-02-20 20:40:28.525 SEVERE thread-12 org.archive.crawler.processor.recrawl.PersistProcessor.populatePersistEnvFromLog() caught exception after loading 77028 urls from the persist log (perhaps crawl was stopped by user?) com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.82) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:466) at com.sleepycat.collections.StoredContainer.handleException(StoredContainer.java:458) at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:325) at com.sleepycat.collections.StoredMap.put(StoredMap.java:257) at org.archive.crawler.processor.recrawl.PersistProcessor.populatePersistEnvFromLog(PersistProcessor.java:212) at org.archive.crawler.processor.recrawl.PersistProcessor.copyPersistSourceToHistoryMap(PersistProcessor.java:316) at org.archive.crawler.processor.recrawl.PersistLoadProcessor.initStore(PersistLoadProcessor.java:81) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.initialTasks(PersistOnlineProcessor.java:62) at org.archive.crawler.framework.CrawlController.runProcessorInitialTasks(CrawlController.java:1666) at org.archive.crawler.framework.CrawlController.requestCrawlStart(CrawlController.java:1011) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1150) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1127) at java.lang.Thread.run(Thread.java:619) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.openCursor(Database.java:516) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:369) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:54) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:317) ... 10 more 2010-02-20 20:40:28.530 INFO thread-12 org.archive.crawler.processor.recrawl.PersistLoadProcessor.initStore() Loaded deduplication information for 77028 previously fetched urls from http://aiapp201:6080/persist-manager/persist/1786 2010-02-20 20:40:28.649 INFO thread-14 org.archive.crawler.admin.CrawlJob.postDeregister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=9093,mother=Heritrix,name=1786-20100220203955121,type=CrawlService.Job unregistered from MBeanServerId=crawling105.us.archive.org_1266647414512, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #1: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:89) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1680) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1035) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:806) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1821) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at com.sleepycat.je.Database.closeInternal(Database.java:406) at com.sleepycat.je.Database.close(Database.java:315) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:87) ... 5 more 2010-02-20 20:40:28.688 INFO thread-15 org.archive.crawler.Heritrix.invoke() JMX invoke: completedJobs() 2010-02-20 20:40:28.761 INFO thread-15 org.archive.crawler.Heritrix.invoke() JMX invoke: crawlendReport("20100220203955121", "crawl-report", ) Exception in thread "StatLogger" java.lang.NullPointerException at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent(CrawlJob.java:776) at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(AbstractTracker.java:187) at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(StatisticsTracker.java:323) at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:144) at java.lang.Thread.run(Thread.java:619)

    JIRA | 7 years ago | Noah Levitt
    java.lang.NullPointerException
  3. 0

    Not sure of the cause or how to reproduce. 2010-06-12 19:29:26.579 INFO thread-55 org.archive.crawler.admin.CrawlJob.postDeregister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=6692,mother=Heritrix,name=1632-20100609192344804,type=CrawlService.Job unregistered from MBeanServerId=crawling105.us.archive.org_1276102367817, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #25: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:89) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1682) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1035) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:806) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1823) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at com.sleepycat.je.Database.closeInternal(Database.java:406) at com.sleepycat.je.Database.close(Database.java:315) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:87) ... 5 more 2010-06-12 19:29:26.602 INFO thread-91 org.archive.crawler.Heritrix.invoke() JMX invoke: completedJobs() 2010-06-12 19:29:26.628 INFO thread-91 org.archive.crawler.Heritrix.invoke() JMX invoke: crawlendReport("20100609192344804", "crawl-report", ) Exception in thread "StatLogger" java.lang.NullPointerException at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent(CrawlJob.java:776) at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(AbstractTracker.java:187) at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(StatisticsTracker.java:323) at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:144) at java.lang.Thread.run(Thread.java:619)

    JIRA | 6 years ago | Noah Levitt
    java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Exception in thread "StatLogger" java.lang.NullPointerException Happened in a start-then-immediate-stop crawl. The first stack trace ("Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN") is normal for this case, the others aren't. heritrix-1.15.4 from svn 2010-01-22 2010-02-20 20:39:54.911 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: addJob("http://aiapp201:6080/seam/resource/crawlRequestOrder.jar?crawlRequestId=16803", "1786", "manually crawl using 3 seeds from Tennessee Electronic Newspapers", "", ) 2010-02-20 20:39:57.230 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: startCrawling() 2010-02-20 20:39:58.348 INFO thread-12 org.archive.crawler.admin.CrawlJob.postRegister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=9093,mother=Heritrix,name=1786-20100220203955121,type=CrawlService.Job registered to MBeanServerId=crawling105.us.archive.org_1266647414512, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems 2010-02-20 20:40:25.755 INFO thread-11 org.archive.crawler.Heritrix.invoke() JMX invoke: terminateCurrentJob() 2010-02-20 20:40:28.525 SEVERE thread-12 org.archive.crawler.processor.recrawl.PersistProcessor.populatePersistEnvFromLog() caught exception after loading 77028 urls from the persist log (perhaps crawl was stopped by user?) com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.82) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:466) at com.sleepycat.collections.StoredContainer.handleException(StoredContainer.java:458) at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:325) at com.sleepycat.collections.StoredMap.put(StoredMap.java:257) at org.archive.crawler.processor.recrawl.PersistProcessor.populatePersistEnvFromLog(PersistProcessor.java:212) at org.archive.crawler.processor.recrawl.PersistProcessor.copyPersistSourceToHistoryMap(PersistProcessor.java:316) at org.archive.crawler.processor.recrawl.PersistLoadProcessor.initStore(PersistLoadProcessor.java:81) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.initialTasks(PersistOnlineProcessor.java:62) at org.archive.crawler.framework.CrawlController.runProcessorInitialTasks(CrawlController.java:1666) at org.archive.crawler.framework.CrawlController.requestCrawlStart(CrawlController.java:1011) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1150) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1127) at java.lang.Thread.run(Thread.java:619) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.openCursor(Database.java:516) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:369) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:54) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:317) ... 10 more 2010-02-20 20:40:28.530 INFO thread-12 org.archive.crawler.processor.recrawl.PersistLoadProcessor.initStore() Loaded deduplication information for 77028 previously fetched urls from http://aiapp201:6080/persist-manager/persist/1786 2010-02-20 20:40:28.649 INFO thread-14 org.archive.crawler.admin.CrawlJob.postDeregister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=9093,mother=Heritrix,name=1786-20100220203955121,type=CrawlService.Job unregistered from MBeanServerId=crawling105.us.archive.org_1266647414512, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #1: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:89) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1680) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1035) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:806) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1821) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at com.sleepycat.je.Database.closeInternal(Database.java:406) at com.sleepycat.je.Database.close(Database.java:315) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:87) ... 5 more 2010-02-20 20:40:28.688 INFO thread-15 org.archive.crawler.Heritrix.invoke() JMX invoke: completedJobs() 2010-02-20 20:40:28.761 INFO thread-15 org.archive.crawler.Heritrix.invoke() JMX invoke: crawlendReport("20100220203955121", "crawl-report", ) Exception in thread "StatLogger" java.lang.NullPointerException at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent(CrawlJob.java:776) at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(AbstractTracker.java:187) at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(StatisticsTracker.java:323) at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:144) at java.lang.Thread.run(Thread.java:619)

    JIRA | 7 years ago | Noah Levitt
    java.lang.NullPointerException
  6. 0

    Not sure of the cause or how to reproduce. 2010-06-12 19:29:26.579 INFO thread-55 org.archive.crawler.admin.CrawlJob.postDeregister() org.archive.crawler:host=crawling105.us.archive.org,jmxport=6692,mother=Heritrix,name=1632-20100609192344804,type=CrawlService.Job unregistered from MBeanServerId=crawling105.us.archive.org_1276102367817, SpecificationVersion=1.4, ImplementationVersion=1.6.0_16-b01, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #25: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:89) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1682) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1035) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:806) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1823) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.82) There are open cursors against the database. They will be closed. Exception while closing cursors: com.sleepycat.je.DatabaseException: (JE 3.3.82) Cursor has been closed. at com.sleepycat.je.Database.closeInternal(Database.java:406) at com.sleepycat.je.Database.close(Database.java:315) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:87) ... 5 more 2010-06-12 19:29:26.602 INFO thread-91 org.archive.crawler.Heritrix.invoke() JMX invoke: completedJobs() 2010-06-12 19:29:26.628 INFO thread-91 org.archive.crawler.Heritrix.invoke() JMX invoke: crawlendReport("20100609192344804", "crawl-report", ) Exception in thread "StatLogger" java.lang.NullPointerException at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent(CrawlJob.java:776) at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(AbstractTracker.java:187) at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(StatisticsTracker.java:323) at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:144) at java.lang.Thread.run(Thread.java:619)

    JIRA | 6 years ago | Noah Levitt
    java.lang.NullPointerException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.NullPointerException

      No message provided

      at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent()
    2. org.archive.crawler
      AbstractTracker.run
      1. org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsEvent(CrawlJob.java:763)
      2. org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(AbstractTracker.java:185)
      3. org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(StatisticsTracker.java:303)
      4. org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:142)
      4 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:595)
      1 frame