org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.

Stack Overflow | armnotstrong | 3 months ago
  1. 0

    Output directory not set exception when save RDD to hbase with spark

    Stack Overflow | 3 months ago | armnotstrong
    org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.
  2. 0

    PartitionedFileSet & Spark Scala Program

    Google Groups | 4 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  3. 0

    Unable to persist a Rdd to fileSet in Spark Scala Program

    Google Groups | 4 months ago | Sharanya Santhanam
    co.cask.tephra.TransactionFailureException: Exception raised in transactional execution. Cause: Output directory not set.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    "ERROR 6000, Output location validation failed" using PIG MongoDB-Hadoop Connector on EMR

    Stack Overflow | 2 years ago | d0x
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example
  6. 0

    h1. Description This PIG Script should load a BSON file and save it into another file. {code} REGISTER /home/hadoop/lib/mongo-java-driver-2.13.0.jar REGISTER /home/hadoop/lib/mongo-hadoop-core-1.3.2.jar REGISTER /home/hadoop/lib/mongo-hadoop-pig-1.3.2.jar SET mapreduce.fileoutputcommitter.marksuccessfuljobs false; SET mapreduce.map.speculative false; SET mapreduce.reduce.speculative false; data = LOAD 's3://xxx/xxx.bson' USING com.mongodb.hadoop.pig.BSONLoader(); // That works great STORE data INTO 'file:///tmp/test.pig' USING PigStorage('\t'); // That fails with ERROR 6000 STORE data INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; // That fails with ERROR 6000 STORE data INTO 's3://xxx/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; {code} The failure I get is this one: {code} grunt> STORE example INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; 2015-03-27 11:57:42,078 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT 2015-03-27 11:57:42,079 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more {code} h1. Steps to reproduce: 1. Spawn a cluster on AWS using this bootstrap script: {code} #!/bin/sh wget -P /home/hadoop/lib http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-core-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-pig-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-hive-1.3.2.jar cp /home/hadoop/lib/mongo* /home/hadoop/pig/lib cp /home/hadoop/lib/mongo* /home/hadoop/hive/lib cp /home/hadoop/lib/mongo* /home/hadoop/share/hadoop/common/lib/ {code} You can have just one server (no Tasktracker and Core instnaces are requiered). 2. Upload any bson dump to s3 3. Execute the PIG script above The failure occurs with {{pig -v}} and {{pig -x local -v}}

    JIRA | 2 years ago | Christian Schneider
    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.InvalidJobConfException

      Output directory not set.

      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs()
    2. Hadoop
      FileOutputFormat.checkOutputSpecs
      1. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138)
      1 frame
    3. Spark
      PairRDDFunctions.saveAsNewAPIHadoopDataset
      1. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1088)
      2. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
      3. org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
      4. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
      5. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
      6. org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
      7. org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
      7 frames
    4. com.xinmeiti.spark
      FilterData.main
      1. com.xinmeiti.spark.weibo.FilterData$.filterData(FilterData.scala:45)
      2. com.xinmeiti.spark.weibo.FilterData$.main(FilterData.scala:20)
      3. com.xinmeiti.spark.weibo.FilterData.main(FilterData.scala)
      3 frames
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:606)
      4 frames
    6. Spark
      SparkSubmit.main
      1. org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      2. org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      3. org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      4. org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      5. org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      5 frames