java.io.IOException: write beyond end of stream

Apache's JIRA Issue Tracker | William Kinney | 7 years ago
  1. 0

    Per [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO.

    Apache's JIRA Issue Tracker | 7 years ago | William Kinney
    java.io.IOException: write beyond end of stream
  2. 0

    Hadoop hdfs throw exception while cloing sequencefile

    Stack Overflow | 1 year ago | Jun Wang
    java.io.IOException: write beyond end of stream
  3. 0

    Groovy provides convenience methods in Process like consumeProcessOutput that spawns off threads to handle an external process' I/O. Unfortunately, these threads are sort of dumped on the floor, leaving the user with no way to manage race conditions where threads finish at different times. For example, the following code tries to capture stdout into a compressed file: Process p = ['mysqldump', 'big_database'].execute() def os = new GZIPOutputStream(new File('dbdump.sql.gz').newOutputStream()) p.consumeProcessOutput os, System.err p.waitFor() os.close() This can result in the following exception. This is because while the process itself may terminate after Process.waitFor(), the thread writing the corresponding output stream may not be finished writing yet. So if I call OutputStream.close() immediately after Process.waitFor(), there is a chance that the gzip stream may not be fully written yet, so the consumer thread tries to write to an output stream that has already been closed. The trouble is, there is no way for me to tell when the consumer threads are done processing. Exception in thread "Thread-27" groovy.lang.GroovyRuntimeException: exception while dumping process stream at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10063) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: write beyond end of stream at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:104) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72) at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10060) ... 1 more There are two possible solutions to this: 1. Process.waitFor should also wait for the consumer threads to finish. 2. Find a way to expose the consumer thread objects so the user can call Thread.join() on them.

    Apache's JIRA Issue Tracker | 8 years ago | Christopher Wong
    groovy.lang.GroovyRuntimeException: exception while dumping process stream
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Groovy provides convenience methods in Process like consumeProcessOutput that spawns off threads to handle an external process' I/O. Unfortunately, these threads are sort of dumped on the floor, leaving the user with no way to manage race conditions where threads finish at different times. For example, the following code tries to capture stdout into a compressed file: Process p = ['mysqldump', 'big_database'].execute() def os = new GZIPOutputStream(new File('dbdump.sql.gz').newOutputStream()) p.consumeProcessOutput os, System.err p.waitFor() os.close() This can result in the following exception. This is because while the process itself may terminate after Process.waitFor(), the thread writing the corresponding output stream may not be finished writing yet. So if I call OutputStream.close() immediately after Process.waitFor(), there is a chance that the gzip stream may not be fully written yet, so the consumer thread tries to write to an output stream that has already been closed. The trouble is, there is no way for me to tell when the consumer threads are done processing. Exception in thread "Thread-27" groovy.lang.GroovyRuntimeException: exception while dumping process stream at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10063) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: write beyond end of stream at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:104) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72) at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10060) ... 1 more There are two possible solutions to this: 1. Process.waitFor should also wait for the consumer threads to finish. 2. Find a way to expose the consumer thread objects so the user can call Thread.join() on them.

    Apache's JIRA Issue Tracker | 8 years ago | Christopher Wong
    groovy.lang.GroovyRuntimeException: exception while dumping process stream
  6. 0

    Re: HDFS Sink stops writing events because HDFSWriter failed to append and close a file

    flume-user | 4 years ago | Jeff Lord
    java.io.IOException: write beyond end of stream*

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      write beyond end of stream

      at org.apache.hadoop.io.compress.CompressorStream.write()
    2. Hadoop
      CompressorStream.write
      1. org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:61)
      1 frame
    3. Java RT
      FilterOutputStream.write
      1. java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
      2. java.io.DataOutputStream.write(DataOutputStream.java:90)
      3. java.io.FilterOutputStream.write(FilterOutputStream.java:80)
      3 frames
    4. Hadoop
      TestCodecPerformance.main
      1. org.apache.hadoop.io.compress.TestCodecPerformance.runCodecPerformance(TestCodecPerformance.java:161)
      2. org.apache.hadoop.io.compress.TestCodecPerformance.runCodecPerformances(TestCodecPerformance.java:91)
      3. org.apache.hadoop.io.compress.TestCodecPerformance.main(TestCodecPerformance.java:271)
      3 frames