co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.

Google Groups | Sharanya Santhanam | 7 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    PartitionedFileSet & Spark Scala Program

    Google Groups | 7 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  2. 0

    Unable to persist a Rdd to fileSet in Spark Scala Program

    Google Groups | 7 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  3. 0

    Output directory not set exception when save RDD to hbase with spark

    Stack Overflow | 6 months ago | armnotstrong
    org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    "ERROR 6000, Output location validation failed" using PIG MongoDB-Hadoop Connector on EMR

    Stack Overflow | 2 years ago | d0x
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example
  6. 0

    h1. Description This PIG Script should load a BSON file and save it into another file. {code} REGISTER /home/hadoop/lib/mongo-java-driver-2.13.0.jar REGISTER /home/hadoop/lib/mongo-hadoop-core-1.3.2.jar REGISTER /home/hadoop/lib/mongo-hadoop-pig-1.3.2.jar SET mapreduce.fileoutputcommitter.marksuccessfuljobs false; SET mapreduce.map.speculative false; SET mapreduce.reduce.speculative false; data = LOAD 's3://xxx/xxx.bson' USING com.mongodb.hadoop.pig.BSONLoader(); // That works great STORE data INTO 'file:///tmp/test.pig' USING PigStorage('\t'); // That fails with ERROR 6000 STORE data INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; // That fails with ERROR 6000 STORE data INTO 's3://xxx/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; {code} The failure I get is this one: {code} grunt> STORE example INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; 2015-03-27 11:57:42,078 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT 2015-03-27 11:57:42,079 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more {code} h1. Steps to reproduce: 1. Spawn a cluster on AWS using this bootstrap script: {code} #!/bin/sh wget -P /home/hadoop/lib http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-core-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-pig-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-hive-1.3.2.jar cp /home/hadoop/lib/mongo* /home/hadoop/pig/lib cp /home/hadoop/lib/mongo* /home/hadoop/hive/lib cp /home/hadoop/lib/mongo* /home/hadoop/share/hadoop/common/lib/ {code} You can have just one server (no Tasktracker and Core instnaces are requiered). 2. Upload any bson dump to s3 3. Execute the PIG script above The failure occurs with {{pig -v}} and {{pig -x local -v}}

    JIRA | 2 years ago | Christian Schneider
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.InvalidJobConfException

      Output directory not set.

      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs()
    2. Hadoop
      FileOutputFormat.checkOutputSpecs
      1. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138)[hadoop-mapreduce-client-core-2.6.0-**]
      1 frame
    3. Spark
      PairRDDFunctions.saveAsNewAPIHadoopDataset
      1. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1011)[spark-assembly-1.5.2-**]
      2. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-**]
      3. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-**]
      4. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)[spark-assembly-1.5.2-**]
      5. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)[spark-assembly-1.5.2-**]
      6. org.apache.spark.rdd.RDD.withScope(RDD.scala:310)[spark-assembly-1.5.2**]
      7. org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-**]
      7 frames
    4. co.cask.cdap
      SparkMainWrapper.main
      1. co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext$$anon$2.run(DefaultSparkExecutionContext.scala:224)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      2. co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:197)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      3. co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext.saveAsDataset(DefaultSparkExecutionContext.scala:202)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      4. co.cask.cdap.api.spark.SparkMain$SparkProgramRDDFunctions.saveAsDataset(SparkMain.scala:121)[co.cask.cdap.cdap-api-spark-3.4.2.jar:na]
      5. co.cask.cdap.scala.WordCountProgram.run(WordCountProgram.scala:65)[program.expanded.jar/:na]
      6. co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:92)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      7. co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      7 frames
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[na:1.8.0_60]
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[na:1.8.0_60]
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[na:1.8.0_60]
      4. java.lang.reflect.Method.invoke(Method.java:497)[na:1.8.0_60]
      4 frames
    6. Spark Project YARN Stable API
      ApplicationMaster$$anon$2.run
      1. org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)[spark-assembly-1.5.2-**.1]
      1 frame