java.lang.NullPointerException

JIRA | Michael Stack | 1 decade ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Below is back and forth between Igor and Stack on an NPE found by Igor serializing out cookies. Looks like we came a cross a cookie that had a null value. No need for the patch. It is not critical to anything that we are doing. i. > Probably not a regression. I think ye've just found case where a cookie value comes back null, something we've not come across before. Doing an append with null to MutableString throws NPE: > > > > [bregeon] ~ > bsh -classpath ~/workspace/heritrix/lib/dsi.unimi.it-1.2.0.jar BeanShell 2.0b4 - by Pat Niemeyer (pat@pat.net) > bsh % new it.unimi.dsi.mg4j.util.MutableString(); > bsh % ms = new it.unimi.dsi.mg4j.util.MutableString(); > bsh % ms.append(null); > // Error: // Uncaught Exception: Method Invocation ms.append : at Line: 3 : in file: <unknown file> : ms .append ( null ) Target exception: java.lang.NullPointerExceptionjava.lang.NullPointerException > at it.unimi.dsi.mg4j.util.MutableString.append(MutableString.java:811) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 9) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at bsh.Reflect.invokeMethod(Unknown Source) > at bsh.Reflect.invokeObjectMethod(Unknown Source) > at bsh.Name.invokeMethod(Unknown Source) > at bsh.BSHMethodInvocation.eval(Unknown Source) > at bsh.BSHPrimaryExpression.eval(Unknown Source) > at bsh.BSHPrimaryExpression.eval(Unknown Source) > at bsh.Interpreter.run(Unknown Source) > at bsh.Interpreter.main(Unknown Source) > > > I'm guessing the problematic value just didn't make it into the cookie.dump. > > Want me to add patch for the NPE? > > St.Ack > > > Igor Ranitovic wrote: >> Hi Stack, >> >> In our latest IQ crawl we used save-cookies option. We have not done this in long time and it seems that we found a regression or new bug. >> At the end of the crawl, Mike notice that reports are not being generated hours after the termination. Looking at heritrix_out revealed the NPE within the Ubicrawler mutable string: >> Exception in thread "ToeThread #61: " java.lang.NullPointerException >> at it.unimi.dsi.mg4j.util.MutableString.append(MutableString.java:828) >> at org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1278) >> at org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1231) >> at org.archive.crawler.fetcher.FetchHTTP.finalTasks(FetchHTTP.java:963) >> at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlC ontroller.java:1669) >> at org.archive.crawler.framework.CrawlController.completeStop(CrawlController. java:1026) >> at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJ ob.java:793) >> at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java :1810) >> at org.archive.crawler.framework.ToeThread.run(ToeThread.java:190) >> Exception in thread "StatLogger" java.lang.NullPointerException >> at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.progressStatisticsE vent(CrawlJob.java:763) >> at org.archive.crawler.framework.AbstractTracker.progressStatisticsEvent(Abstr actTracker.java:185) >> at org.archive.crawler.admin.StatisticsTracker.progressStatisticsEvent(Statist icsTracker.java:303) >> at org.archive.crawler.framework.AbstractTracker.run(AbstractTracker.java:142) >> at java.lang.Thread.run(Thread.java:595) >> >> >> We did get some of the cookies dumped to the disk. See cookies.dump and the rest of the >> configuration files are at crawling004:/0/loc-jobs/IQ-100/ >> >> i. >

    JIRA | 1 decade ago | Michael Stack
    java.lang.NullPointerException
  2. 0

    The following exception stack occurred when terminating a small test crawl via the web UI. A subsequent crawl terminated normally on same settings. com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN occured when trying to process 'http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif' at step ABOUT_TO_BEGIN_PROCESSOR in HTTP com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread run Fatal exception in ToeThread #29: http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif java.lang.NullPointerException at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)

    JIRA | 10 years ago | Kristinn Sigurðsson
    java.lang.NullPointerException
  3. 0

    The following exception stack occurred when terminating a small test crawl via the web UI. A subsequent crawl terminated normally on same settings. com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN occured when trying to process 'http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif' at step ABOUT_TO_BEGIN_PROCESSOR in HTTP com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread run Fatal exception in ToeThread #29: http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif java.lang.NullPointerException at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)

    JIRA | 10 years ago | Kristinn Sigurðsson
    java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    From Kris: It's me again :-) Discovered a potential NPE when terminating a job. The Frontier hangs around for the threads to finish (at least it is supposed too) but (as the following stacktrace shows), the CrawlController or more likely the CrawlScope (unsure which) does not: java.lang.NullPointerException at org.archive.crawler.postprocessor.Postselector.schedule(Postselector.java:2 69) at org.archive.crawler.postprocessor.Postselector.handleLinkCollection(Postsel ector.java:358) at org.archive.crawler.postprocessor.Postselector.innerProcess(Postselector.ja va:166) at org.archive.crawler.framework.Processor.process(Processor.java:102) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:255) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:131) Exception in thread "ToeThread #5" java.lang.NullPointerException at org.archive.crawler.framework.ToeThread.run(ToeThread.java:137) The following patch 'handles' it (if in a very simplistic way), basically just catcht the NPE, and decide that the URI is not within scope if this occurs. Maybe the CrawlController (and it should not be null) should throw an EndedException on getScope() when the crawl has been terminated? Not really a big bug, but we really should have the crawler finish at least semi-gracefully, although I should note that it did not prevent the crawl reports from being written. - Kris Index: Postselector.java =================================================================== RCS file: /cvsroot/archive-crawler/ArchiveOpenCrawler/src/java/org/archive/crawler/po stprocessor/Postselector.java,v retrieving revision 1.13 diff -u -r1.13 Postselector.java --- Postselector.java 27 Oct 2004 00:47:23 -0000 1.13 +++ Postselector.java 17 Nov 2004 15:45:25 -0000 @@ -266,21 +266,26 @@ * @return true if CandidateURI was accepted by crawl scope, false otherwise */ private boolean schedule(CandidateURI caUri) { - if(getController().getScope().accepts(caUri)) { - logger.finer("Accepted: "+caUri); - getController().getFrontier().schedule(caUri); - return true; - } else { - // Run the curi through another set of filters to see - // if we should log it to the scope rejection log. - if (logger.isLoggable(Level.INFO)) { - CrawlURI curi = (caUri instanceof CrawlURI)? - (CrawlURI)caUri: new CrawlURI(caUri.getUURI()); - if (filtersAccept(this.rejectLogFilters, curi)) { - logger.info("Rejected " + curi.getUURI().toString()); + try{ + if(getController().getScope().accepts(caUri)) { + logger.finer("Accepted: "+caUri); + getController().getFrontier().schedule(caUri); + return true; + } else { + // Run the curi through another set of filters to see + // if we should log it to the scope rejection log. + if (logger.isLoggable(Level.INFO)) { + CrawlURI curi = (caUri instanceof CrawlURI)? + (CrawlURI)caUri: new CrawlURI(caUri.getUURI()); + if (filtersAccept(this.rejectLogFilters, curi)) { + logger.info("Rejected " + curi.getUURI().toString()); + } } } + } catch(NullPointerException e){ + // Return false if this happens. Most likely the crawl is ending. } + return false; }

    JIRA | 1 decade ago | Michael Stack
    java.lang.NullPointerException

    Root Cause Analysis

    1. java.lang.NullPointerException

      No message provided

      at it.unimi.dsi.mg4j.util.MutableString.append()
    2. it.unimi.dsi
      MutableString.append
      1. it.unimi.dsi.mg4j.util.MutableString.append(MutableString.java:828)
      1 frame
    3. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1278)
      2. org.archive.crawler.fetcher.FetchHTTP.saveCookies(FetchHTTP.java:1231)
      3. org.archive.crawler.fetcher.FetchHTTP.finalTasks(FetchHTTP.java:963)
      4. org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1669)
      5. org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1026)
      6. org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:793)
      7. org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1810)
      8. org.archive.crawler.framework.ToeThread.run(ToeThread.java:190)
      8 frames