java.io.IOException: write beyond end of stream

flume-user | Shady Xu | 1 year ago
  1. 0

    write beyond end of stream

    flume-user | 1 year ago | Shady Xu
    java.io.IOException: write beyond end of stream
  2. 0

    Groovy provides convenience methods in Process like consumeProcessOutput that spawns off threads to handle an external process' I/O. Unfortunately, these threads are sort of dumped on the floor, leaving the user with no way to manage race conditions where threads finish at different times. For example, the following code tries to capture stdout into a compressed file: Process p = ['mysqldump', 'big_database'].execute() def os = new GZIPOutputStream(new File('dbdump.sql.gz').newOutputStream()) p.consumeProcessOutput os, System.err p.waitFor() os.close() This can result in the following exception. This is because while the process itself may terminate after Process.waitFor(), the thread writing the corresponding output stream may not be finished writing yet. So if I call OutputStream.close() immediately after Process.waitFor(), there is a chance that the gzip stream may not be fully written yet, so the consumer thread tries to write to an output stream that has already been closed. The trouble is, there is no way for me to tell when the consumer threads are done processing. Exception in thread "Thread-27" groovy.lang.GroovyRuntimeException: exception while dumping process stream at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10063) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: write beyond end of stream at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:104) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72) at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10060) ... 1 more There are two possible solutions to this: 1. Process.waitFor should also wait for the consumer threads to finish. 2. Find a way to expose the consumer thread objects so the user can call Thread.join() on them.

    Apache's JIRA Issue Tracker | 8 years ago | Christopher Wong
    groovy.lang.GroovyRuntimeException: exception while dumping process stream
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Per [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO.

    Apache's JIRA Issue Tracker | 7 years ago | William Kinney
    java.io.IOException: write beyond end of stream
  5. 0

    Groovy provides convenience methods in Process like consumeProcessOutput that spawns off threads to handle an external process' I/O. Unfortunately, these threads are sort of dumped on the floor, leaving the user with no way to manage race conditions where threads finish at different times. For example, the following code tries to capture stdout into a compressed file: Process p = ['mysqldump', 'big_database'].execute() def os = new GZIPOutputStream(new File('dbdump.sql.gz').newOutputStream()) p.consumeProcessOutput os, System.err p.waitFor() os.close() This can result in the following exception. This is because while the process itself may terminate after Process.waitFor(), the thread writing the corresponding output stream may not be finished writing yet. So if I call OutputStream.close() immediately after Process.waitFor(), there is a chance that the gzip stream may not be fully written yet, so the consumer thread tries to write to an output stream that has already been closed. The trouble is, there is no way for me to tell when the consumer threads are done processing. Exception in thread "Thread-27" groovy.lang.GroovyRuntimeException: exception while dumping process stream at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10063) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: write beyond end of stream at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:104) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72) at org.codehaus.groovy.runtime.DefaultGroovyMethods$ByteDumper.run(DefaultGroovyMethods.java:10060) ... 1 more There are two possible solutions to this: 1. Process.waitFor should also wait for the consumer threads to finish. 2. Find a way to expose the consumer thread objects so the user can call Thread.join() on them.

    Apache's JIRA Issue Tracker | 8 years ago | Christopher Wong
    groovy.lang.GroovyRuntimeException: exception while dumping process stream

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      write beyond end of stream

      at com.hadoop.compression.lzo.LzopOutputStream.write()
    2. com.hadoop.compression
      LzopOutputStream.write
      1. com.hadoop.compression.lzo.LzopOutputStream.write(LzopOutputStream.java:134)
      1 frame
    3. Java RT
      OutputStream.write
      1. java.io.OutputStream.write(OutputStream.java:75)
      1 frame
    4. Flume
      BodyTextEventSerializer.write
      1. org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
      1 frame
    5. org.apache.flume
      BucketWriter$9.call
      1. org.apache.flume.sink.hdfs.HDFSCompressedDataStream.append(HDFSCompressedDataStream.java:126)
      2. org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
      3. org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
      4. org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
      5. org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
      6. org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
      6 frames
    6. Java RT
      Thread.run
      1. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      2. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      3. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      4. java.lang.Thread.run(Thread.java:724)
      4 frames