cluster.ClusterTaskSetManager: Loss was due to > java.io.FileNotFoundException > java.io.FileNotFoundException: > /tmp/spark-local-20140417145643-a055/3c/shuffle_1_218_1157 (Too many > open files) > > ulimit -n tells me I can open 32000 files. Here's a plot of lsof on a > worker node during a failed .distinct(): > http://i.imgur.com/wyBHmzz.png , you can see tasks fail when Spark > tries to open 32000 files. > > I never ran into this in 0.7.3. Is there a parameter I can set to tell > Spark to use less than 32000 files? > > On Mon, Mar 24, 2014 at 10:23 AM, Aaron Davidson < > wrote: >> Look up setting ulimit, though note the distinction between soft and hard >> limits, and that updating your hard limit may require changing >> /etc/security/limits.confand restarting each worker. >> >> >> On Mon, Mar 24, 2014 at 1:39 AM, Kane < > wrote: Got a bit further, i think out of memory error was caused by setting spark.spill to false. Now i have this error, is there an easy way to increase file limit for spark, cluster-wide?: java.io.FileNotFoundException: /tmp/spark-local-20140324074221-b8f1/01/temp_1ab674f9-4556-4239-9f21-688dfc9f17d2 (Too many open files)

nabble.com | 7 months ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Apache Spark User List - distinct on huge dataset

    nabble.com | 7 months ago
    cluster.ClusterTaskSetManager: Loss was due to > java.io.FileNotFoundException > java.io.FileNotFoundException: > /tmp/spark-local-20140417145643-a055/3c/shuffle_1_218_1157 (Too many > open files) > > ulimit -n tells me I can open 32000 files. Here's a plot of lsof on a > worker node during a failed .distinct(): > http://i.imgur.com/wyBHmzz.png , you can see tasks fail when Spark > tries to open 32000 files. > > I never ran into this in 0.7.3. Is there a parameter I can set to tell > Spark to use less than 32000 files? > > On Mon, Mar 24, 2014 at 10:23 AM, Aaron Davidson < > wrote: >> Look up setting ulimit, though note the distinction between soft and hard >> limits, and that updating your hard limit may require changing >> /etc/security/limits.confand restarting each worker. >> >> >> On Mon, Mar 24, 2014 at 1:39 AM, Kane < > wrote: Got a bit further, i think out of memory error was caused by setting spark.spill to false. Now i have this error, is there an easy way to increase file limit for spark, cluster-wide?: java.io.FileNotFoundException: /tmp/spark-local-20140324074221-b8f1/01/temp_1ab674f9-4556-4239-9f21-688dfc9f17d2 (Too many open files)

    Root Cause Analysis

    1. cluster.ClusterTaskSetManager

      Loss was due to > java.io.FileNotFoundException > java.io.FileNotFoundException: > /tmp/spark-local-20140417145643-a055/3c/shuffle_1_218_1157 (Too many > open files) > > ulimit -n tells me I can open 32000 files. Here's a plot of lsof on a > worker node during a failed .distinct(): > http://i.imgur.com/wyBHmzz.png , you can see tasks fail when Spark > tries to open 32000 files. > > I never ran into this in 0.7.3. Is there a parameter I can set to tell > Spark to use less than 32000 files? > > On Mon, Mar 24, 2014 at 10:23 AM, Aaron Davidson < > wrote: >> Look up setting ulimit, though note the distinction between soft and hard >> limits, and that updating your hard limit may require changing >> /etc/security/limits.confand restarting each worker. >> >> >> On Mon, Mar 24, 2014 at 1:39 AM, Kane < > wrote: Got a bit further, i think out of memory error was caused by setting spark.spill to false. Now i have this error, is there an easy way to increase file limit for spark, cluster-wide?: java.io.FileNotFoundException: /tmp/spark-local-20140324074221-b8f1/01/temp_1ab674f9-4556-4239-9f21-688dfc9f17d2 (Too many open files)

      at java.io.FileOutputStream.openAppend()
    2. Java RT
      FileOutputStream.<init>
      1. java.io.FileOutputStream.openAppend(Native Method)
      2. java.io.FileOutputStream.<init>(FileOutputStream.java:192)
      2 frames
    3. Spark
      Executor$TaskRunner.run
      1. org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:113)
      2. org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:174)
      3. org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:191)
      4. org.apache.spark.util.collection.ExternalAppendOnlyMap.insert(ExternalAppendOnlyMap.scala:141)
      5. org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:59)
      6. org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:95)
      7. org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:94)
      8. org.apache.spark.rdd.RDD$$anonfun$3.apply(RDD.scala:471)
      9. org.apache.spark.rdd.RDD$$anonfun$3.apply(RDD.scala:471)
      10. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
      11. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      12. org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      13. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161)
      14. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
      15. org.apache.spark.scheduler.Task.run(Task.scala:53)
      16. org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
      17. org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:49)
      18. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
      18 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      3. java.lang.Thread.run(Thread.java:662)
      3 frames