java.io.FileNotFoundException: /crawl/heritrix/heritrix-0.6.0/jobs/crs-20040427190708335/disk/scratch/bphc .hrsa.gov.ff0 (Too many open files)

JIRA | Michael Stack | 10 years ago
  1. 0

    I happen to have a seed list of nearly 1024 entries. Not totally surprisingly, Heritrix behaves a little oddly with that many seeds. First, crawls with either 0.6.0 or the latest CVS build fail because too many files are opened almost immediately, and then neither socket operations nor file logging are able to proceed. A typical exception: java.io.FileNotFoundException: /crawl/heritrix/heritrix-0.6.0/jobs/crs-20040427190708335/disk/scratch/bphc .hrsa.gov.ff0 (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.archive.io.FlipFileOutputStream.<init>(FlipFileOutputStream.java:69) at org.archive.io.DiskBackedByteQueue.initializeStreams(DiskBackedByteQueue.ja va:67) at org.archive.util.DiskQueue.<init>(DiskQueue.java:100) at org.archive.util.DiskBackedQueue.<init>(DiskBackedQueue.java:59) at org.archive.crawler.basic.KeyedQueue.<init>(KeyedQueue.java:76) at org.archive.crawler.basic.Frontier.keyedQueueFor(Frontier.java:927) at org.archive.crawler.basic.Frontier.scheduleForRetry(Frontier.java:1333) at org.archive.crawler.basic.Frontier.finished(Frontier.java:676) at org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:200) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:124) You can get past that by allowing a larger number of open files for the process (which requires running Heritrix with root privilege), as in: # (ulimit -n 4096; JAVA_OPTS=-Xmx320 bin/heritrix -p 9876)

    JIRA | 10 years ago | Michael Stack
    java.io.FileNotFoundException: /crawl/heritrix/heritrix-0.6.0/jobs/crs-20040427190708335/disk/scratch/bphc .hrsa.gov.ff0 (Too many open files)
  2. 0

    TeamCity is leaking caches/git file descriptors

    YouTrack | 6 years ago
    java.io.FileNotFoundException: /data/teamcity/BuildServer/system/pluginData/customDataStorage/buildTypes/bt2613/485653500 (Too many open files)
  3. 0

    FileNotFoundException during saving of PDF file

    GitHub | 5 years ago | mbastian
    java.io.FileNotFoundException: C:\****\Untilted.pdf (Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Marker Issues

    GitHub | 4 years ago | TheSeize
    java.io.FileNotFoundException: plugins/dynmap/web/tiles/_markers_/marker_final.json.new (No such file or directory)
  6. 0

    Trouble with deploying Sonatype Nexus to Tomcat6 on Gentoo (log4j)

    Stack Overflow | 7 years ago | John
    java.io.FileNotFoundException: /dev/null/sonatype-work/nexus/conf/log4j.properties (Not a directory)

  1. eti22 3 times, last 2 weeks ago
  2. tyson925 49 times, last 1 week ago
  3. MoYapro 1 times, last 1 month ago
  4. musketyr 3 times, last 1 month ago
  5. iridic 12 times, last 1 month ago
23 more registered users
72 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.FileNotFoundException

    /crawl/heritrix/heritrix-0.6.0/jobs/crs-20040427190708335/disk/scratch/bphc .hrsa.gov.ff0 (Too many open files)

    at java.io.FileOutputStream.open()
  2. Java RT
    FileOutputStream.<init>
    1. java.io.FileOutputStream.open(Native Method)
    2. java.io.FileOutputStream.<init>(FileOutputStream.java:179)
    3. java.io.FileOutputStream.<init>(FileOutputStream.java:131)
    3 frames
  3. webarchive-commons
    DiskBackedQueue.<init>
    1. org.archive.io.FlipFileOutputStream.<init>(FlipFileOutputStream.java:69)
    2. org.archive.io.DiskBackedByteQueue.initializeStreams(DiskBackedByteQueue.java:67)
    3. org.archive.util.DiskQueue.<init>(DiskQueue.java:100)
    4. org.archive.util.DiskBackedQueue.<init>(DiskBackedQueue.java:59)
    4 frames
  4. org.archive.crawler
    ToeThread.run
    1. org.archive.crawler.basic.KeyedQueue.<init>(KeyedQueue.java:76)
    2. org.archive.crawler.basic.Frontier.keyedQueueFor(Frontier.java:927)
    3. org.archive.crawler.basic.Frontier.scheduleForRetry(Frontier.java:1333)
    4. org.archive.crawler.basic.Frontier.finished(Frontier.java:676)
    5. org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:200)
    6. org.archive.crawler.framework.ToeThread.run(ToeThread.java:124)
    6 frames