co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.

Google Groups | Sharanya Santhanam | 4 months ago
  1. 0

    PartitionedFileSet & Spark Scala Program

    Google Groups | 4 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  2. 0

    Unable to persist a Rdd to fileSet in Spark Scala Program

    Google Groups | 4 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  3. 0

    Output directory not set exception when save RDD to hbase with spark

    Stack Overflow | 3 months ago | armnotstrong
    org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    "ERROR 6000, Output location validation failed" using PIG MongoDB-Hadoop Connector on EMR

    Stack Overflow | 2 years ago | d0x
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example
  6. 0

    h1. Description This PIG Script should load a BSON file and save it into another file. {code} REGISTER /home/hadoop/lib/mongo-java-driver-2.13.0.jar REGISTER /home/hadoop/lib/mongo-hadoop-core-1.3.2.jar REGISTER /home/hadoop/lib/mongo-hadoop-pig-1.3.2.jar SET mapreduce.fileoutputcommitter.marksuccessfuljobs false; SET mapreduce.map.speculative false; SET mapreduce.reduce.speculative false; data = LOAD 's3://xxx/xxx.bson' USING com.mongodb.hadoop.pig.BSONLoader(); // That works great STORE data INTO 'file:///tmp/test.pig' USING PigStorage('\t'); // That fails with ERROR 6000 STORE data INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; // That fails with ERROR 6000 STORE data INTO 's3://xxx/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; {code} The failure I get is this one: {code} grunt> STORE example INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; 2015-03-27 11:57:42,078 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT 2015-03-27 11:57:42,079 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more {code} h1. Steps to reproduce: 1. Spawn a cluster on AWS using this bootstrap script: {code} #!/bin/sh wget -P /home/hadoop/lib http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-core-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-pig-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-hive-1.3.2.jar cp /home/hadoop/lib/mongo* /home/hadoop/pig/lib cp /home/hadoop/lib/mongo* /home/hadoop/hive/lib cp /home/hadoop/lib/mongo* /home/hadoop/share/hadoop/common/lib/ {code} You can have just one server (no Tasktracker and Core instnaces are requiered). 2. Upload any bson dump to s3 3. Execute the PIG script above The failure occurs with {{pig -v}} and {{pig -x local -v}}

    JIRA | 2 years ago | Christian Schneider
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.InvalidJobConfException

      Output directory not set.

      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs()
    2. Hadoop
      FileOutputFormat.checkOutputSpecs
      1. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138)[hadoop-mapreduce-client-core-2.7.2-sfdc-1.0.1.jar:2.7.2-sfdc-1.0.1]
      1 frame
    3. Spark
      PairRDDFunctions.saveAsNewAPIHadoopDataset
      1. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1011)[spark-assembly-1.5.2-*****]
      2. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-***]
      3. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-***]
      4. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)[spark-assembly-1.5.2-*****]
      5. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)[spark-assembly-1.5.2-****]
      6. org.apache.spark.rdd.RDD.withScope(RDD.scala:310)[spark-assembly-1.5.2-****]
      7. org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)[spark-assembly-1.5.2-****]
      7 frames
    4. co.cask.cdap
      SparkMainWrapper.main
      1. co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext$$anon$2.run(DefaultSparkExecutionContext.scala:224)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      2. co.cask.cdap.app.runtime.spark.SparkTransactional.execute(SparkTransactional.java:197)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      3. co.cask.cdap.app.runtime.spark.DefaultSparkExecutionContext.saveAsDataset(DefaultSparkExecutionContext.scala:202)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      4. co.cask.cdap.api.spark.SparkMain$SparkProgramRDDFunctions.saveAsDataset(SparkMain.scala:121)[co.cask.cdap.cdap-api-spark-3.4.2.jar:na]
      5. co.cask.cdap.scala.WordCountProgram.run(WordCountProgram.scala:65)[program.expanded.jar/:na]
      6. co.cask.cdap.app.runtime.spark.SparkMainWrapper$.main(SparkMainWrapper.scala:92)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      7. co.cask.cdap.app.runtime.spark.SparkMainWrapper.main(SparkMainWrapper.scala)[co.cask.cdap.cdap-spark-core-3.4.2.jar:na]
      7 frames
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[na:1.8.0_60]
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[na:1.8.0_60]
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[na:1.8.0_60]
      4. java.lang.reflect.Method.invoke(Method.java:497)[na:1.8.0_60]
      4 frames
    6. Spark Project YARN Stable API
      ApplicationMaster$$anon$2.run
      1. org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)[spark-assembly-1.5.2-******]
      1 frame