java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory

JIRA | May | 2 years ago
  1. 0

    Hello, I'm unable to find similar issues what I'm experiencing. I'm using CDH5.0.1 and Spark in Cloudera Manager 5.0.1. I compiled SparkR with SPARK_HADOOP_VERSION=2.3.0-mr1-cdh5.0.1 ./install-dev.sh OS is Red Hat 6.3. I have 4 hosts and installed R on all of them. SparkR is only installed on 1 of them. Full Error: Loading required package: SparkR Loading required package: methods Loading required package: rJava [SparkR] Initializing with classpath /usr/local/lib64/R/library/SparkR/sparkr-assembly-0.1.jar 14/06/25 20:11:06 INFO Slf4jLogger: Slf4jLogger started 14/06/25 20:11:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/06/25 20:11:09 INFO FileInputFormat: Total input paths to process : 1 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 0 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Loss was due to java.io.IOException java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) at edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241) at org.apache.spark.rdd.RDD.iterator(RDD.scala:232) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109) at org.apache.spark.scheduler.Task.run(Task.scala:53) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 1 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 2 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 3 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 4 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 5 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 6 (task 0.0:0) 14/06/25 20:11:11 ERROR TaskSetManager: Task 0.0:0 failed 4 times; aborting job Error in .jcall(getJRDD(rdd), "Ljava/util/List;", "collect") : org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times (most recent failure: Exception failure: java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory) Calls: count ... collect -> collect -> .local -> .jcall -> .jcheck -> .Call Execution halted My test R scripts is below: require(SparkR) Sys.setenv(MASTER="<HOST>:7077") Sys.setenv(SPARK_HOME="/hadoop/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/") Sys.setenv(SCALA_HOME="/hadoop/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib") sc <- sparkR.init(Sys.getenv("MASTER")) lines <- textFile(sc, "hdfs://mike-flume1.amers1b.ciscloud/user/hdfs/data.txt") count(lines) wordsPerLine <- lapply(lines, function(line) { length(unlist(strsplit(line, " "))) }) collect(wordsPerLine) Any help would be appreciated.

    JIRA | 2 years ago | May
    java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory
  2. 0

    Hello, I'm unable to find similar issues what I'm experiencing. I'm using CDH5.0.1 and Spark in Cloudera Manager 5.0.1. I compiled SparkR with SPARK_HADOOP_VERSION=2.3.0-mr1-cdh5.0.1 ./install-dev.sh OS is Red Hat 6.3. I have 4 hosts and installed R on all of them. SparkR is only installed on 1 of them. Full Error: Loading required package: SparkR Loading required package: methods Loading required package: rJava [SparkR] Initializing with classpath /usr/local/lib64/R/library/SparkR/sparkr-assembly-0.1.jar 14/06/25 20:11:06 INFO Slf4jLogger: Slf4jLogger started 14/06/25 20:11:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/06/25 20:11:09 INFO FileInputFormat: Total input paths to process : 1 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 0 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Loss was due to java.io.IOException java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) at edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241) at org.apache.spark.rdd.RDD.iterator(RDD.scala:232) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109) at org.apache.spark.scheduler.Task.run(Task.scala:53) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 1 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 2 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 3 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 4 (task 0.0:0) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 5 (task 0.0:1) 14/06/25 20:11:11 WARN TaskSetManager: Lost TID 6 (task 0.0:0) 14/06/25 20:11:11 ERROR TaskSetManager: Task 0.0:0 failed 4 times; aborting job Error in .jcall(getJRDD(rdd), "Ljava/util/List;", "collect") : org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times (most recent failure: Exception failure: java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory) Calls: count ... collect -> collect -> .local -> .jcall -> .jcheck -> .Call Execution halted My test R scripts is below: require(SparkR) Sys.setenv(MASTER="<HOST>:7077") Sys.setenv(SPARK_HOME="/hadoop/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/") Sys.setenv(SCALA_HOME="/hadoop/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib") sc <- sparkR.init(Sys.getenv("MASTER")) lines <- textFile(sc, "hdfs://mike-flume1.amers1b.ciscloud/user/hdfs/data.txt") count(lines) wordsPerLine <- lapply(lines, function(line) { length(unlist(strsplit(line, " "))) }) collect(wordsPerLine) Any help would be appreciated.

    JIRA | 2 years ago | May
    java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory
  3. 0

    RE: Cannot run program "Rscript" using SparkR

    apache.org | 12 months ago
    java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    sonar not generating reports with jenkins

    Stack Overflow | 4 years ago | user1720124
    java.io.IOException: Cannot run program "mvn" (in directory "/root/.jenkins/jobs/test/workspace"): error=2, No such file or directory
  6. 0

    Jenkins not recognizing git binary

    Stack Overflow | 4 years ago | sathya
    java.io.IOException: Cannot run program "git": error=2, No such file or directory

  1. eti22 1 times, last 1 month ago
  2. asdasd 24 times, last 3 months ago
  3. rexgreenza 8 times, last 3 months ago
  4. Benj-AD 1 times, last 4 months ago
  5. Akshay 4 times, last 6 months ago
95 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.IOException

    Cannot run program "Rscript": error=2, No such file or directory

    at java.lang.ProcessBuilder.start()
  2. Java RT
    ProcessBuilder.start
    1. java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    1 frame
  3. edu.berkeley.cs
    RRDD.compute
    1. edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:105)
    1 frame
  4. Spark
    SparkHadoopUtil$$anon$1.run
    1. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
    2. org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
    3. org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
    4. org.apache.spark.scheduler.Task.run(Task.scala:53)
    5. org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
    6. org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
    7. org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
    7 frames
  5. Java RT
    Subject.doAs
    1. java.security.AccessController.doPrivileged(Native Method)
    2. javax.security.auth.Subject.doAs(Subject.java:415)
    2 frames
  6. Hadoop
    UserGroupInformation.doAs
    1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    1 frame
  7. Spark
    Executor$TaskRunner.run
    1. org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
    2. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
    2 frames
  8. Java RT
    Thread.run
    1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    3. java.lang.Thread.run(Thread.java:722)
    3 frames