java.io.IOException: Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs

GitHub | nlevitt | 4 months ago
  1. 0

    GitHub comment 167#239924423

    GitHub | 4 months ago | nlevitt
    java.io.IOException: Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs
  2. 0

    Possible race condition when creating writable directory. In the example below, multiple threads attempt to write the the same not-yet-existant directory. This results in a NullPointerException, and the data is not written. In this example, the dns information seems to have been cached and used, but not written. The warc numbers seem to increment as well, even though the write failed, leading to non-contiguous numbers. 2012-05-09 17:55:04.657 SEVERE thread-17 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.lang.NullPointerException occured when trying to process 'dns:www.newt.org' at step ABOUT_TO_BEGIN_PROCESSOR in Which have stack trace: java.lang.NullPointerException        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:284)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424        at org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)        at org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:446)        at org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:204)        at org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:181)        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:231)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)

    JIRA | 5 years ago | Adam Miller
    java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424
  3. 0

    Possible race condition when creating writable directory. In the example below, multiple threads attempt to write the the same not-yet-existant directory. This results in a NullPointerException, and the data is not written. In this example, the dns information seems to have been cached and used, but not written. The warc numbers seem to increment as well, even though the write failed, leading to non-contiguous numbers. 2012-05-09 17:55:04.657 SEVERE thread-17 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.lang.NullPointerException occured when trying to process 'dns:www.newt.org' at step ABOUT_TO_BEGIN_PROCESSOR in Which have stack trace: java.lang.NullPointerException        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:284)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151) java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424        at org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)        at org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:446)        at org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:204)        at org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:181)        at org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:231)        at org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:209)        at org.archive.modules.Processor.process(Processor.java:142)        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)

    JIRA | 5 years ago | Adam Miller
    java.io.IOException: Failed to create directory: /3/crawldata/e12/20120509175424
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Failed to create directory: /1/ait-h3-jobs/gh-167/20160815201316/warcs

      at org.archive.util.FileUtils.ensureWriteableDirectory()
    2. webarchive-commons
      FileUtils.ensureWriteableDirectory
      1. org.archive.util.FileUtils.ensureWriteableDirectory(FileUtils.java:677)
      1 frame
    3. org.archive.modules
      WriterPoolProcessor.calcOutputDirs
      1. org.archive.modules.writer.WriterPoolProcessor.calcOutputDirs(WriterPoolProcessor.java:457)
      1 frame
    4. webarchive-commons
      WriterPoolMember.checkSize
      1. org.archive.io.WriterPoolMember.createFile(WriterPoolMember.java:203)
      2. org.archive.io.WriterPoolMember.checkSize(WriterPoolMember.java:180)
      2 frames
    5. org.archive.modules
      ProcessorChain.process
      1. org.archive.modules.writer.WARCWriterProcessor.write(WARCWriterProcessor.java:219)
      2. org.archive.modules.writer.WARCWriterProcessor.innerProcessResult(WARCWriterProcessor.java:195)
      3. org.archive.modules.Processor.process(Processor.java:142)
      4. org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
      4 frames
    6. org.archive.crawler
      ToeThread.run
      1. org.archive.crawler.framework.ToeThread.run(ToeThread.java:152)
      1 frame