java.lang.OutOfMemoryError

JIRA | Gordon Mohr | 9 years ago
  1. 0

    Vinay reports getting this OOME stack barely more than an hour after a starting a busy crawl that's run without problems in nearly identical configuration the last 3 months: java.lang.OutOfMemoryError Stacktrace: java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:123) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:46) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58) at org.archive.io.WriterPoolMember$CompressedStream.(WriterPoolMember.java:518) at org.archive.io.WriterPoolMember.preWriteRecordTasks(WriterPoolMember.java:367) ...etc... Other info collected: 'top' line for JVM (look at VIRT: so much larger than heap and at 4GB zone): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12228 webcrawl 18 0 4064m 2.3g 46m S 0.0 60.3 130:38.15 /usr/lib/jdk1.6.0_01/bin/java -Dj 5 of last 6 URLs shown in crawl.log are >50MB files, 4 of 5 are compressed video (gzip stress?) Unsure why this is triggered now, but looks like this JVM issue: "Instantiating Inflater/Deflater causes OutOfMemoryError; finalizers not called promptly enough" http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189 Paradoxically, a larger heap -- by limiting the need for heap GC -- could make this problem worse, as could anything else that increases the proportion of crawler effort using gzip/native space compared to heap space. (This is a dedup crawl writing both ARCs and WARCs, so relatively lot of compression is occurring.)

    JIRA | 9 years ago | Gordon Mohr
    java.lang.OutOfMemoryError
  2. 0

    Vinay reports getting this OOME stack barely more than an hour after a starting a busy crawl that's run without problems in nearly identical configuration the last 3 months: java.lang.OutOfMemoryError Stacktrace: java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:123) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:46) at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:58) at org.archive.io.WriterPoolMember$CompressedStream.(WriterPoolMember.java:518) at org.archive.io.WriterPoolMember.preWriteRecordTasks(WriterPoolMember.java:367) ...etc... Other info collected: 'top' line for JVM (look at VIRT: so much larger than heap and at 4GB zone): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12228 webcrawl 18 0 4064m 2.3g 46m S 0.0 60.3 130:38.15 /usr/lib/jdk1.6.0_01/bin/java -Dj 5 of last 6 URLs shown in crawl.log are >50MB files, 4 of 5 are compressed video (gzip stress?) Unsure why this is triggered now, but looks like this JVM issue: "Instantiating Inflater/Deflater causes OutOfMemoryError; finalizers not called promptly enough" http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189 Paradoxically, a larger heap -- by limiting the need for heap GC -- could make this problem worse, as could anything else that increases the proportion of crawler effort using gzip/native space compared to heap space. (This is a dedup crawl writing both ARCs and WARCs, so relatively lot of compression is occurring.)

    JIRA | 9 years ago | Gordon Mohr
    java.lang.OutOfMemoryError
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Server performance

    Google Groups | 2 years ago | Sean McGrath
    java.lang.OutOfMemoryError

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      No message provided

      at java.util.zip.Deflater.init()
    2. Java RT
      GZIPOutputStream.<init>
      1. java.util.zip.Deflater.init(Native Method)
      2. java.util.zip.Deflater.<init>(Deflater.java:123)
      3. java.util.zip.GZIPOutputStream.<init>(GZIPOutputStream.java:46)
      4. java.util.zip.GZIPOutputStream.<init>(GZIPOutputStream.java:58)
      4 frames
    3. webarchive-commons
      WriterPoolMember.preWriteRecordTasks
      1. org.archive.io.WriterPoolMember$CompressedStream.<init>(WriterPoolMember.java:518)
      2. org.archive.io.WriterPoolMember.preWriteRecordTasks(WriterPoolMember.java:367)
      2 frames