org.apache.pig.impl.logicalLayer.FrontendException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • h1. Description This PIG Script should load a BSON file and save it into another file. {code} REGISTER /home/hadoop/lib/mongo-java-driver-2.13.0.jar REGISTER /home/hadoop/lib/mongo-hadoop-core-1.3.2.jar REGISTER /home/hadoop/lib/mongo-hadoop-pig-1.3.2.jar SET mapreduce.fileoutputcommitter.marksuccessfuljobs false; SET mapreduce.map.speculative false; SET mapreduce.reduce.speculative false; data = LOAD 's3://xxx/xxx.bson' USING com.mongodb.hadoop.pig.BSONLoader(); // That works great STORE data INTO 'file:///tmp/test.pig' USING PigStorage('\t'); // That fails with ERROR 6000 STORE data INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; // That fails with ERROR 6000 STORE data INTO 's3://xxx/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; {code} The failure I get is this one: {code} grunt> STORE example INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; 2015-03-27 11:57:42,078 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT 2015-03-27 11:57:42,079 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more {code} h1. Steps to reproduce: 1. Spawn a cluster on AWS using this bootstrap script: {code} #!/bin/sh wget -P /home/hadoop/lib http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-core-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-pig-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-hive-1.3.2.jar cp /home/hadoop/lib/mongo* /home/hadoop/pig/lib cp /home/hadoop/lib/mongo* /home/hadoop/hive/lib cp /home/hadoop/lib/mongo* /home/hadoop/share/hadoop/common/lib/ {code} You can have just one server (no Tasktracker and Core instnaces are requiered). 2. Upload any bson dump to s3 3. Execute the PIG script above The failure occurs with {{pig -v}} and {{pig -x local -v}}
    via by Christian Schneider,
  • h1. Description This PIG Script should load a BSON file and save it into another file. {code} REGISTER /home/hadoop/lib/mongo-java-driver-2.13.0.jar REGISTER /home/hadoop/lib/mongo-hadoop-core-1.3.2.jar REGISTER /home/hadoop/lib/mongo-hadoop-pig-1.3.2.jar SET mapreduce.fileoutputcommitter.marksuccessfuljobs false; SET mapreduce.map.speculative false; SET mapreduce.reduce.speculative false; data = LOAD 's3://xxx/xxx.bson' USING com.mongodb.hadoop.pig.BSONLoader(); // That works great STORE data INTO 'file:///tmp/test.pig' USING PigStorage('\t'); // That fails with ERROR 6000 STORE data INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; // That fails with ERROR 6000 STORE data INTO 's3://xxx/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; {code} The failure I get is this one: {code} grunt> STORE example INTO 'file://tmp/whatever.bson' USING com.mongodb.hadoop.pig.BSONStorage; 2015-03-27 11:57:42,078 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: LIMIT 2015-03-27 11:57:42,079 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. 2015-03-27 11:57:42,089 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more {code} h1. Steps to reproduce: 1. Spawn a cluster on AWS using this bootstrap script: {code} #!/bin/sh wget -P /home/hadoop/lib http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0/mongo-java-driver-2.13.0.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-core-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-pig-1.3.2.jar wget -P /home/hadoop/lib https://github.com/mongodb/mongo-hadoop/releases/download/r1.3.2/mongo-hadoop-hive-1.3.2.jar cp /home/hadoop/lib/mongo* /home/hadoop/pig/lib cp /home/hadoop/lib/mongo* /home/hadoop/hive/lib cp /home/hadoop/lib/mongo* /home/hadoop/share/hadoop/common/lib/ {code} You can have just one server (no Tasktracker and Core instnaces are requiered). 2. Upload any bson dump to s3 3. Execute the PIG script above The failure occurs with {{pig -v}} and {{pig -x local -v}}
    via by Christian Schneider,
  • problem using BSON Store
    via by batman,
  • PartitionedFileSet & Spark Scala Program
    via by Sharanya Santhanam,
  • Unable to persist a Rdd to fileSet in Spark Scala Program
    via by Sharanya Santhanam,
  • running another job in hadoop
    via Stack Overflow by Flethuseo
    ,
    • org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias example at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1637) at org.apache.pig.PigServer.registerQuery(PigServer.java:577) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1091) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:543) at org.apache.pig.Main.main(Main.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: <line 10, column 0> Output Location Validation Failed for: 'file://tmp/whatever.bson More info to follow: Output directory not set. at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:317) at org.apache.pig.PigServer.compilePp(PigServer.java:1382) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307) at org.apache.pig.PigServer.execute(PigServer.java:1299) at org.apache.pig.PigServer.access$400(PigServer.java:124) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1632) ... 13 more Caused by: org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:138) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 26 more
    No Bugmate found.