org.apache.spark.shuffle.FetchFailedException: Error in opening FileSegmentManagedBuffer{file=/tmp/spark-b5507e64-ccaf-4082-8c7e-373bd7945133/executor-690303d2-011f-4a03-a21f-62479c2b9cb2/blockmgr-42f641bb-db6e-4fd8-913b-c343e792a704/31/shuffle_0_78_0.data, offset=608, length=16}

GitHub | car2008 | 6 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    GitHub comment 197#244256304

    GitHub | 6 months ago | car2008
    org.apache.spark.shuffle.FetchFailedException: Error in opening FileSegmentManagedBuffer{file=/tmp/spark-b5507e64-ccaf-4082-8c7e-373bd7945133/executor-690303d2-011f-4a03-a21f-62479c2b9cb2/blockmgr-42f641bb-db6e-4fd8-913b-c343e792a704/31/shuffle_0_78_0.data, offset=608, length=16}
  2. 0

    Spark Job failed on YARN -

    Stack Overflow | 1 year ago | Shankar
    org.apache.spark.shuffle.FetchFailedException: Error in opening FileSegmentManagedBuffer{file=/hdfs1/yarn/nm/usercache/phdpentcustcdibtch/appcache/application_1449986083135_60217/blockmgr-34a2e882-6b36-42c6-bcff-03d9bc5ef80b/0c/shuffle_4_42_0.data, offset=5899394, length=46751}
  3. 0

    GitHub comment 200#106711720

    GitHub | 2 years ago | linxixiong
    org.apache.spark.shuffle.FetchFailedException: Error in opening FileSegmentManagedBuffer{file=/data2/hadoop/hd_space/tmp/nm-local-dir/usercache/linxixiong/appcache/application_1432021926412_9808/spark-82b1829f-7662-4480-82ca-9abf7cc6618d/10/shuffle_5_106_0.data, offset=17462106, length=2529501}
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    GitHub comment 200#88290674

    GitHub | 2 years ago | thanigaiv
    org.apache.spark.shuffle.FetchFailedException: Error in opening FileSegmentManagedBuffer{file=/data/yarn/nm/usercache/root/appcache/application_1426812029369_0893/spark-local-20150331165623-6233/19/shuffle_5_0_0.data, offset=3437, length=532}
  6. 0

    Intermittent java.io.FileNotFoundException on release

    GitHub | 1 year ago | samthebest
    java.io.FileNotFoundException: /var/lib/jenkins/workspace/altmetrics/target/scala-2.10/classes/com/MYCLASS$$anonfun$main$5$$anonfun$apply$3$$anonfun$apply$5.class (No such file or directory)

  1. rp 1 times, last 2 weeks ago
  2. Adrien 6 times, last 2 weeks ago
  3. gamebusterz 8 times, last 3 weeks ago
  4. michallos 1 times, last 1 month ago
  5. tyson925 4 times, last 1 month ago
52 more registered users
43 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.FileNotFoundException

    /tmp/spark-b5507e64-ccaf-4082-8c7e-373bd7945133/executor-690303d2-011f-4a03-a21f-62479c2b9cb2/blockmgr-42f641bb-db6e-4fd8-913b-c343e792a704/31/shuffle_0_78_0.data (there is no file or directory)

    at java.io.FileInputStream.open0()
  2. Java RT
    FileInputStream.<init>
    1. java.io.FileInputStream.open0(Native Method)
    2. java.io.FileInputStream.open(FileInputStream.java:195)
    3. java.io.FileInputStream.<init>(FileInputStream.java:138)
    3 frames
  3. Spark
    FileSegmentManagedBuffer.createInputStream
    1. org.apache.spark.network.buffer.FileSegmentManagedBuffer.createInputStream(FileSegmentManagedBuffer.java:98)
    1 frame
  4. Spark
    ShuffleBlockFetcherIterator.next
    1. org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:304)
    2. org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
    2 frames
  5. Scala
    Iterator$$anon$11.hasNext
    1. scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    2. scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    3. scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    3 frames
  6. Spark
    Executor$TaskRunner.run
    1. org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
    2. org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
    3. org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:152)
    4. org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:45)
    5. org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:89)
    6. org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98)
    7. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    8. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    9. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    10. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    11. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    12. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    13. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    14. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    15. org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
    16. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    17. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    18. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    19. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    20. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    21. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    22. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    23. org.apache.spark.scheduler.Task.run(Task.scala:89)
    24. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    24 frames
  7. Java RT
    Thread.run
    1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    3. java.lang.Thread.run(Thread.java:745)
    3 frames