java.lang.NullPointerException

JIRA | (sourceforge) | 10 years ago
  1. 0

    I get an NPE in quotaEnforcer.checkQuotas whenever I crawl the Danish site: http://tv.sputnik.dk. 2006-11-07T15:39:50.692Z -5 - clsid:A9FC132B-096D-460B-B7D5-1DB0FAE0C062 XRE http://tv.sputnik.dk/?returnurl=http://tv.sputnik.dk/player/license/channel /2089519/clip/1782918.html&cancelurl=http://tv.sputnik.dk/page/2040550/chan nel/2089519/category/720677-0/clip/1782918/index.html no-type #034 - - - err=java.lang.NullPointerException' Nov 7, 2006 3:39:50 PM org.archive.crawler.framework.ToeThread recoverableProblem SEVERE: Problem java.lang.NullPointerException occured when trying to process 'clsid:A9FC132B-096D-460B-B7D5-1DB0FAE0C062' at step ABOUT_TO_BEGIN_PROCESSOR in QuotaEnforcer java.lang.NullPointerException at org.archive.crawler.prefetch.QuotaEnforcer.checkQuotas(QuotaEnforcer.java:2 57) at org.archive.crawler.prefetch.QuotaEnforcer.innerProcess(QuotaEnforcer.java: 239) at org.archive.crawler.framework.Processor.process(Processor.java:103) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:304) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:153) Nov 7, 2006 3:39:50 PM dk.netarkivet.harvester.harvesting.DomainnameQueueAssignmentPolicy getClassKey FINE: Illegal class key candidate default... for clsid:6BF52A52-394A-11d3-B153-00C04F79FAA6 Nov 7, 2006 3:39:50 PM org.archive.crawler.framework.ToeThread recoverableProblem SEVERE: Problem java.lang.NullPointerException occured when trying to process 'clsid:6BF52A52-394A-11d3-B153-00C04F79FAA6' at step ABOUT_TO_BEGIN_PROCESSOR in QuotaEnforcer java.lang.NullPointerException at org.archive.crawler.prefetch.QuotaEnforcer.checkQuotas(QuotaEnforcer.java:2 57) at org.archive.crawler.prefetch.QuotaEnforcer.innerProcess(QuotaEnforcer.java: 239) at org.archive.crawler.framework.Processor.process(Processor.java:103) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:304) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:153) Note that we use our own QueueAssignmentPolicy. This may cause the problem as well. But the code fragment in question: >protected boolean checkQuotas(final CrawlURI curi, > final CrawlSubstats.HasCrawlSubstats >hasStats, > final int CAT) { > CrawlSubstats substats = >hasStats.getSubstats(); > should to avoid the NPE be replaced by protected boolean checkQuotas(final CrawlURI curi, final CrawlSubstats.HasCrawlSubstats hasStats, final int CAT) { if (substats == null) { return false; } CrawlSubstats substats = hasStats.getSubstats();

    JIRA | 10 years ago | (sourceforge)
    java.lang.NullPointerException
  2. 0

    NullPointerException at org.archive.crawler.processor.recrawl.PersistLogProcessor.finalTasks(PersistLogProcessor.java:87) 03/09/2009 17:07:46 +0000 INFO org.archive.crawler.admin.CrawlJob postDeregister org.archive.crawler:host=crawling10.us.archive.org,jmxport=9093,mother=h1236289378518,name=1104-20090309170725217,type=CrawlService.Job unregistered from MBeanServerId=crawling10.us.archive.org_1236143748023, SpecificationVersion=1.4, ImplementationVersion=1.6.0_03-b05, SpecificationVendor=Sun Microsystems Exception in thread "ToeThread #75: " java.lang.NullPointerException at org.archive.crawler.processor.recrawl.PersistLogProcessor.finalTasks(PersistLogProcessor.java:87) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Exception in thread "ToeThread #63: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #61: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #64: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031) at org.archive.crawler.admin.CrawlJob$MBeanCrawlController.completeStop(CrawlJob.java:801) at org.archive.crawler.framework.CrawlController.toeEnded(CrawlController.java:1817) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:186) Caused by: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1458) at com.sleepycat.je.Database.sync(Database.java:424) at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:83) ... 5 more Exception in thread "ToeThread #60: " java.lang.RuntimeException: com.sleepycat.je.DatabaseException: (JE 3.3.75) Can't call Database.sync: Database state can't be DbState.CLOSED must be DbState.OPEN at org.archive.crawler.processor.recrawl.PersistOnlineProcessor.finalTasks(PersistOnlineProcessor.java:86) at org.archive.crawler.framework.CrawlController.runProcessorFinalTasks(CrawlController.java:1676) at org.archive.crawler.framework.CrawlController.completeStop(CrawlController.java:1031)

    JIRA | 8 years ago | Noah Levitt
    java.lang.NullPointerException
  3. 0

    Ftp entries in an arc file look like this currently: ftp://ftp.ksl.stanford.edu/welcome.msg 171.64.71.195 20081121190026 no-type 56 ***** ***** Stanford Knowledge Systems Laboratory ***** There is no header, only body content. When heritrix encounters an error trying to download a file, for example: 550 foo: Permission denied. it throws an exception which propagates to the logs: 11/21/2008 19:00:45 +0000 SEVERE org.archive.crawler.fetcher.FetchFTP innerProcess FTP server reported problem. org.archive.net.FTPException: FTP error code: 550 at org.archive.net.ClientFTP.openDataConnection(ClientFTP.java:130) at org.archive.crawler.fetcher.FetchFTP.fetch(FetchFTP.java:312) at org.archive.crawler.fetcher.FetchFTP.innerProcess(FetchFTP.java:252) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Heritrix still tries to write to the ARC, but fails because there is no content: 11/21/2008 19:00:45 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem java.lang.NullPointerException occured when trying to process 'ftp://ftp.ksl.stanford.edu/dev/ticotsord' at step ABOUT_TO_BEGIN_PROCESSOR in Archiver java.lang.NullPointerException at org.archive.crawler.writer.ARCWriterProcessor.innerProcess(ARCWriterProcessor.java:122) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) So there is no record in the arc file at all. But this "550 foo: Permission denied." is essentially equivalent to a HTTP 403. It should be archived somehow and should not spew stack traces in the logs. So I propose we include a "header" section in the arc for ftp transactions. "550 foo: Permission denied." would go there. On a successful get, the message would be something like "150 Binary data connection for /welcome.msg (76.103.251.45,57342) (56 bytes)." Would this break anything?

    JIRA | 8 years ago | Noah Levitt
    java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    The following exception stack occurred when terminating a small test crawl via the web UI. A subsequent crawl terminated normally on same settings. com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread recoverableProblem Problem com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN occured when trying to process 'http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif' at step ABOUT_TO_BEGIN_PROCESSOR in HTTP com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:447) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:380) at org.apache.commons.httpclient.cookie.CookieSpecBase.match(CookieSpecBase.java:607) at org.apache.commons.httpclient.HttpMethodBase.addCookieRequestHeader(HttpMethodBase.java:1193) at org.apache.commons.httpclient.HttpMethodBase.addRequestHeaders(HttpMethodBase.java:1327) at org.apache.commons.httpclient.HttpMethodBase.writeRequestHeaders(HttpMethodBase.java:2056) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1939) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1000) at org.archive.httpclient.HttpRecorderGetMethod.execute(HttpRecorderGetMethod.java:116) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.archive.crawler.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:500) at org.archive.crawler.framework.Processor.process(Processor.java:112) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) Caused by: com.sleepycat.je.DatabaseException: (JE 3.2.23) Can't open a cursor Database state can't be DbState.CLOSED must be DbState.OPEN at com.sleepycat.je.Database.checkRequiredDbState(Database.java:1069) at com.sleepycat.je.Database.openCursor(Database.java:359) at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:364) at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:30) at com.sleepycat.collections.DataCursor.init(DataCursor.java:171) at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:59) at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299) ... 15 more 07/05/2007 21:02:25 +0000 SEVERE org.archive.crawler.framework.ToeThread run Fatal exception in ToeThread #29: http://www.landsbokasafn.is/Apps/WebObjects/HI.woa/wa/header_logo_neg.gif java.lang.NullPointerException at org.archive.crawler.framework.ToeThread.run(ToeThread.java:157)

    JIRA | 9 years ago | Kristinn SigurĂ°sson
    java.lang.NullPointerException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.NullPointerException

      No message provided

      at org.archive.crawler.prefetch.QuotaEnforcer.checkQuotas()
    2. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.prefetch.QuotaEnforcer.checkQuotas(QuotaEnforcer.java:257)
      2. org.archive.crawler.prefetch.QuotaEnforcer.innerProcess(QuotaEnforcer.java:239)
      3. org.archive.crawler.framework.Processor.process(Processor.java:103)
      4. org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:304)
      5. org.archive.crawler.framework.ToeThread.run(ToeThread.java:153)
      5 frames