java.io.IOException: Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs

GitHub | nlevitt | 6 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    GitHub comment 167#239924423

    GitHub | 6 months ago | nlevitt
    java.io.IOException: Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs
  2. 0

    Possible race condition when creating writable directory. In the example below, multiple threads attempt to write the the same not-yet-existant directory. This results in a NullPointerException, and the data is not written. In this example, the dns information seems to have been cached and used, but not written. The warc numbers seem to increment as well, even though the write failed, leading to non-contiguous numbers. 2012-05-09 17:55:04.657 SEVERE thread-17 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.lang.NullPointerException occured when trying to process 'dns:www.newt.org' at step ABOUT_TO_BEGIN_PROCESSOR in Which have stack trace: java.lang.NullPointerException        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:284)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424        at org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)        at org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:446)        at org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:204)        at org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:181)        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:231)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)

    JIRA | 5 years ago | Adam Miller
    java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424
  3. 0

    Possible race condition when creating writable directory. In the example below, multiple threads attempt to write the the same not-yet-existant directory. This results in a NullPointerException, and the data is not written. In this example, the dns information seems to have been cached and used, but not written. The warc numbers seem to increment as well, even though the write failed, leading to non-contiguous numbers. 2012-05-09 17:55:04.657 SEVERE thread-17 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.lang.NullPointerException occured when trying to process 'dns:www.newt.org' at step ABOUT_TO_BEGIN_PROCESSOR in Which have stack trace: java.lang.NullPointerException        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:284)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424        at org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)        at org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:446)        at org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:204)        at org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:181)        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:231)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)

    JIRA | 5 years ago | Adam Miller
    java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Heritrix issue with an external location

    GitHub | 2 weeks ago | liuqingli
    java.lang.IllegalStateException: java.io.IOException: Failed to create directory: /sfm-data/containers/91e060938deb/jobs

    Root Cause Analysis

    1. java.io.IOException

      Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs

      at org.archive.util.FileUtils.ensureWriteableDirectory()
    2. webarchive-commons
      FileUtils.ensureWriteableDirectory
      1. org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)
      1 frame
    3. org.archive.modules
      WriterPoolProcessor.calcOutputDirs
      1. org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:457)
      1 frame
    4. webarchive-commons
      WriterPoolMember.checkSize
      1. org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:203)
      2. org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:180)
      2 frames
    5. org.archive.modules
      ProcessorChain.process
      1. org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:219)
      2. org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:195)
      3. org.archive.modules.Processor.process(Processor.java:142)
      4. org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
      4 frames
    6. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.framework.ToeThread.run(ToeThread.java:152)
      1 frame