java.io.IOException: No FileSystem for scheme: mongodb

JIRA | Navin Viswanath | 2 years ago
  1. 0

    MongoDB Hadoop error : no FileSystem for scheme:mongodb

    Stack Overflow | 2 years ago | Navin Viswanath
    java.io.IOException: No FileSystem for scheme: mongodb
  2. 0

    I'm trying to get a basic Spark example running using mongoDB hadoop connector. I'm using Hadoop version *2.6.0*. I'm using version *1.3.1* of mongo-hadoop. I'm not sure where exactly to place the jars for this Hadoop version. Here are the locations I've tried: - $HADOOP_HOME/libexec/share/hadoop/mapreduce - $HADOOP_HOME/libexec/share/hadoop/mapreduce/lib - $HADOOP_HOME/libexec/share/hadoop/hdfs - $HADOOP_HOME/libexec/share/hadoop/hdfs/lib Here is a snippet of code I'm using to load the mongo collection into hdfs: {code} Configuration bsonConfig = new Configuration(); bsonConfig.set("mongo.job.input.format", "MongoInputFormat.class"); JavaPairRDD<Object,BSONObject> zipData = sc.newAPIHadoopFile("mongodb://127.0.0.1:27017/zipsdb.zips", MongoInputFormat.class, Object.class, BSONObject.class, bsonConfig); {code} I get the following error no matter where the jar is placed: {noformat} Exception in thread "main" java.io.IOException: No FileSystem for scheme: mongodb at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:505) at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:774) at org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:471) {noformat} I dont see any other errors in hadoop logs. I suspect I'm missing something in my configuration, or that Hadoop 2.6.0 is not compatible with this connector. Any help is much appreciated.

    JIRA | 2 years ago | Navin Viswanath
    java.io.IOException: No FileSystem for scheme: mongodb
  3. 0

    I'm trying to get a basic Spark example running using mongoDB hadoop connector. I'm using Hadoop version *2.6.0*. I'm using version *1.3.1* of mongo-hadoop. I'm not sure where exactly to place the jars for this Hadoop version. Here are the locations I've tried: - $HADOOP_HOME/libexec/share/hadoop/mapreduce - $HADOOP_HOME/libexec/share/hadoop/mapreduce/lib - $HADOOP_HOME/libexec/share/hadoop/hdfs - $HADOOP_HOME/libexec/share/hadoop/hdfs/lib Here is a snippet of code I'm using to load the mongo collection into hdfs: {code} Configuration bsonConfig = new Configuration(); bsonConfig.set("mongo.job.input.format", "MongoInputFormat.class"); JavaPairRDD<Object,BSONObject> zipData = sc.newAPIHadoopFile("mongodb://127.0.0.1:27017/zipsdb.zips", MongoInputFormat.class, Object.class, BSONObject.class, bsonConfig); {code} I get the following error no matter where the jar is placed: {noformat} Exception in thread "main" java.io.IOException: No FileSystem for scheme: mongodb at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:505) at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:774) at org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:471) {noformat} I dont see any other errors in hadoop logs. I suspect I'm missing something in my configuration, or that Hadoop 2.6.0 is not compatible with this connector. Any help is much appreciated.

    JIRA | 2 years ago | Navin Viswanath
    java.io.IOException: No FileSystem for scheme: mongodb
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Can anyone guide me on how to correctly set up and send a MapReduce Job to Hadoop from Java?

    Stack Overflow | 2 years ago | rabejens
    java.io.IOException: No FileSystem for scheme: hdfs

  1. tyson925 3 times, last 4 months ago
14 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.IOException

    No FileSystem for scheme: mongodb

    at org.apache.hadoop.fs.FileSystem.getFileSystemClass()
  2. Hadoop
    Path.getFileSystem
    1. org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
    2. org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
    3. org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    4. org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    5. org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    6. org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    7. org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
    7 frames
  3. Hadoop
    FileInputFormat.addInputPath
    1. org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:505)
    1 frame
  4. Spark
    JavaSparkContext.newAPIHadoopFile
    1. org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:774)
    2. org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:471)
    2 frames