java.io.IOException: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos

JIRA | James Wahlin | 3 years ago
  1. 0

    When running the mongo-hadoop example script {{examples/treasury_yield/run_job.sh}} against a {{mongos}} instance it fails with the following exception: {code} java.io.IOException: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos at com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:53) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) at com.mongodb.hadoop.util.MongoTool.run(MongoTool.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig.main(TreasuryYieldXMLConfig.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos at com.mongodb.hadoop.splitter.StandaloneMongoSplitter.calculateSplits(StandaloneMongoSplitter.java:79) at com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:51) ... 19 more {code} It appears that the mongo-hadoop connector is running splitVector against the {{admin}} database and mongos rejects because it is not the same database the split is being executed against. mongod allows this but mongos rejects. Instead we should run this command against the same database passed to the splitVector command. splitVector command execution is not restricted to admin.

    JIRA | 3 years ago | James Wahlin
    java.io.IOException: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos
  2. 0

    When running the mongo-hadoop example script {{examples/treasury_yield/run_job.sh}} against a {{mongos}} instance it fails with the following exception: {code} java.io.IOException: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos at com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:53) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) at com.mongodb.hadoop.util.MongoTool.run(MongoTool.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig.main(TreasuryYieldXMLConfig.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos at com.mongodb.hadoop.splitter.StandaloneMongoSplitter.calculateSplits(StandaloneMongoSplitter.java:79) at com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:51) ... 19 more {code} It appears that the mongo-hadoop connector is running splitVector against the {{admin}} database and mongos rejects because it is not the same database the split is being executed against. mongod allows this but mongos rejects. Instead we should run this command against the same database passed to the splitVector command. splitVector command execution is not restricted to admin.

    JIRA | 3 years ago | James Wahlin
    java.io.IOException: com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos
  3. 0

    Prediction IO on AWS EMR

    Google Groups | 4 years ago | David Houston
    com.mongodb.hadoop.splitter.SplitFailedException: Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos*
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.mongodb.hadoop.splitter.SplitFailedException

      Unable to calculate input splits: doing a splitVector across dbs isn't supported via mongos

      at com.mongodb.hadoop.splitter.StandaloneMongoSplitter.calculateSplits()
    2. com.mongodb.hadoop
      MongoInputFormat.getSplits
      1. com.mongodb.hadoop.splitter.StandaloneMongoSplitter.calculateSplits(StandaloneMongoSplitter.java:79)
      2. com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:51)
      2 frames
    3. Hadoop
      Job$10.run
      1. org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
      2. org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
      3. org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
      4. org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
      5. org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
      5 frames
    4. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:415)
      2 frames
    5. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      1 frame
    6. Hadoop
      Job.waitForCompletion
      1. org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
      2. org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
      2 frames
    7. com.mongodb.hadoop
      MongoTool.run
      1. com.mongodb.hadoop.util.MongoTool.run(MongoTool.java:102)
      1 frame
    8. Hadoop
      ToolRunner.run
      1. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      2. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      2 frames
    9. com.mongodb.hadoop
      TreasuryYieldXMLConfig.main
      1. com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig.main(TreasuryYieldXMLConfig.java:38)
      1 frame
    10. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:601)
      4 frames
    11. Hadoop
      RunJar.main
      1. org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      1 frame