org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory zodiacs already exists

Cloudera Open Source | Patrick Dylan Zwick | 6 years ago
  1. 0

    When I import a table using the command: sqoop import --connect jdbc:mysql://192.168.3.122:3306/fbapp_monster --table zodiacs -m 1 --hive-import --username anihq --password xxxxxx sqoop creates the table and I can access it in hive. If I then type "hive" at the command line and enter hive, when I type "show tables" I can see that it imported the table zodiacs. If I then type drop table zodiacs the table is dropped and no longer shows up when I type "show tables". However, if I then rerun the exact same sqoop import command above, I get an error telling me the zodiacs table already exists. 11/06/02 01:57:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 11/06/02 01:57:28 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 11/06/02 01:57:28 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 11/06/02 01:57:28 INFO tool.CodeGenTool: Beginning code generation 11/06/02 01:57:28 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1 11/06/02 01:57:28 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1 11/06/02 01:57:28 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop 11/06/02 01:57:28 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-core.jar Note: /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 11/06/02 01:57:29 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.java to /home/dzwick/./zodiacs.java 11/06/02 01:57:29 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.jar 11/06/02 01:57:29 WARN manager.MySQLManager: It looks like you are importing from mysql. 11/06/02 01:57:29 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 11/06/02 01:57:29 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 11/06/02 01:57:29 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 11/06/02 01:57:29 INFO mapreduce.ImportJobBase: Beginning import of zodiacs 11/06/02 01:57:29 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1 11/06/02 01:57:30 INFO mapred.JobClient: Cleaning up the staging area hdfs://h14-200.sv3:54310/n1/tmp/mapred/mapred/staging/dzwick/.staging/job_201104142112_9830 11/06/02 01:57:30 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory zodiacs already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) at org.apache.hadoop.mapreduce.Job.submit(Job.java:476) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506) at com.cloudera.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:107) at com.cloudera.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:166) at com.cloudera.sqoop.manager.SqlManager.importTable(SqlManager.java:386) at com.cloudera.sqoop.manager.MySQLManager.importTable(MySQLManager.java:125) at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:350) at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423) at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180) at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:218) at com.cloudera.sqoop.Sqoop.main(Sqoop.java:228) I know a similar bug has been reported before, and a patch has been released, but that patch dealt with the situation where the table being imported already exists. It seems to me that the problem persists even if the original import has been dropped. Any idea what's going on and how to fix this? Thanks, Dylan

    Cloudera Open Source | 6 years ago | Patrick Dylan Zwick
    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory zodiacs already exists
  2. 0

    Help - Exception in thread "main" java.io.IOException: Error opening job | DeZyre

    dezyre.com | 4 months ago
    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/cloudera/class3/emswc/employee3.csv already exists
  3. 0

    MapReduce code cleaning staging area error

    Stack Overflow | 2 years ago | user3642020
    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /axp/rim/score2/dev/cmc_score/rishabh/inp/file2.txt already exists
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    wordcount example error

    Google Groups | 5 years ago | Raja Nagendra Kumar
    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output already exists
  6. 0

    Re: How to use GridMix3

    apache.org | 1 year ago
    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /home/username/Desktop/test already exists

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.FileAlreadyExistsException

      Output directory zodiacs already exists

      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs()
    2. Hadoop
      JobClient$2.run
      1. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
      2. org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872)
      3. org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
      3 frames
    3. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:396)
      2 frames
    4. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
      1 frame
    5. Hadoop
      Job.waitForCompletion
      1. org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
      2. org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
      3. org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
      3 frames
    6. com.cloudera.sqoop
      Sqoop.run
      1. com.cloudera.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:107)
      2. com.cloudera.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:166)
      3. com.cloudera.sqoop.manager.SqlManager.importTable(SqlManager.java:386)
      4. com.cloudera.sqoop.manager.MySQLManager.importTable(MySQLManager.java:125)
      5. com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:350)
      6. com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423)
      7. com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
      7 frames
    7. Hadoop
      ToolRunner.run
      1. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      2. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      2 frames
    8. com.cloudera.sqoop
      Sqoop.main
      1. com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
      2. com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:218)
      3. com.cloudera.sqoop.Sqoop.main(Sqoop.java:228)
      3 frames