org.springframework.data.mongodb.UncategorizedMongoDbException: Rare case where master=null, probably all servers are down; nested exception is com.mongodb.MongoException: Rare case where master=null, probably all servers are down

JIRA | Atul Gupta | 3 years ago
  1. 0

    We are using replica-set and providing the configuration like the following, but these configuration options does not seem to work. We seem to be getting 2 problems- 1) if all the nodes of the mongo db are down, requests (e.g. mongoTemplate.save) hang for about 60-120 seconds and then throw an exception. Following is the configuration we are using, the behavior is same whether we use mongo options or not. We are using mongo db for logging purpose only which is used for logging requests and responses and want even if all nodes of the mongo db are down, requests should not hang (if these fail, these should exit immediately) 2) We see very fast re-connection attempts being made (hundreds of them in a sec) which clogs the log files very fast and we are not sure if #1 may be caused by this. <mongo:mongo id="mongo" replica-set="${nosql.db.replicaset}"> <mongo:options connect-timeout="5000" max-wait-time="1500" auto-connect-retry="true" socket-keep-alive="true" socket-timeout="1500" slave-ok="true" write-number="1" write-timeout="1000" write-fsync="true"/> </mongo:mongo> The following error is seen in the logs repeatedly- org.springframework.data.mongodb.UncategorizedMongoDbException: Rare case where master=null, probably all servers are down; nested exception is com.mongodb.MongoException: Rare case where master=null, probably all servers are down at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:78) at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1694) at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:375) at org.springframework.data.mongodb.core.MongoTemplate.collectionExists(MongoTemplate.java:437) at com.expedia.globalpayments.svs.checker.MongoDatabaseChecker.check(MongoDatabaseChecker.java:56) at com.expedia.e3.es.payment.common.status.cache.ComponentStatusUpdateTask.update(ComponentStatusUpdateTask.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273) at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:299) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:111) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) Caused by: com.mongodb.MongoException: Rare case where master=null, probably all servers are down at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:435) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:277) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:256) at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:289) at com.mongodb.DB.command(DB.java:259) at com.mongodb.DB.command(DB.java:241) at com.mongodb.DB.command(DB.java:282) at com.mongodb.DB.command(DB.java:196) at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:796) at com.mongodb.DB.authenticate(DB.java:569) at org.springframework.data.mongodb.core.MongoDbUtils.doGetDB(MongoDbUtils.java:114) at org.springframework.data.mongodb.core.MongoDbUtils.getDB(MongoDbUtils.java:75) at org.springframework.data.mongodb.core.SimpleMongoDbFactory.getDb(SimpleMongoDbFactory.java:118) at org.springframework.data.mongodb.core.SimpleMongoDbFactory.getDb(SimpleMongoDbFactory.java:107) at org.springframework.data.mongodb.core.MongoTemplate.getDb(MongoTemplate.java:1286) at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:372) ... 12 more Any help is highly appreciated- Thanks, Atul Gupta

    JIRA | 3 years ago | Atul Gupta
    org.springframework.data.mongodb.UncategorizedMongoDbException: Rare case where master=null, probably all servers are down; nested exception is com.mongodb.MongoException: Rare case where master=null, probably all servers are down
  2. 0

    Running TomEE as Windows service causes JAVA WAR in it to throw "MongoException: Rare case where master=null, probably all servers are down"

    Stack Overflow | 3 years ago | user1264304
    com.mongodb.MongoException: Rare case where master=null, probably all servers are down
  3. 0

    mongoException

    Google Groups | 5 years ago | swc...@sina.com
    com.mongodb.MongoException: can't find a master
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [mongodb-user] mongoException

    Google Groups | 5 years ago | swc...@sina.com
    com.mongodb.MongoException: can't find a master
  6. 0

    The fix for JAVA-656 introduced a regression. In the case where you have use the Mongo constructor that takes a list of ServerAddress, or a MongoURI with a list of server addresses, _masterPortPool starts as null. Before this change, the checkMaster call would block until an initial master was found. Now that it no longer does, there is a race between the background thread and application threads. If an application only does non-primary reads, and the list of servers is a list of mongos, until a write is done, then ensureMaster will never be called and you will keep getting this exception: {noformat} 10-25-2012 17:01:00 [Thread-427] ERROR com.xgen.svc.mms.svc.ping.BatchPrePopulationSvc [run:136] - Rare case where master=null, probably all servers are down com.mongodb.MongoException: Rare case where master=null, probably all servers are down at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:434) ~[mongo.jar:na] at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:276) ~[mongo.jar:na] at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:256) ~[mongo.jar:na] at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:289) ~[mongo.jar:na] at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:274) ~[mongo.jar:na] at com.mongodb.DBCursor._check(DBCursor.java:368) ~[mongo.jar:na] at com.mongodb.DBCursor._hasNext(DBCursor.java:459) ~[mongo.jar:na] at com.mongodb.DBCursor.hasNext(DBCursor.java:484) ~[mongo.jar:na] {noformat} This test demonstrates the problem. {code:java} public class JAVA656Test { public static void main(String[] args) throws UnknownHostException { Mongo mongo = new Mongo(Arrays.asList(new ServerAddress("localhost:30999"))); mongo.setReadPreference(ReadPreference.secondary()); DBCollection collection = mongo.getDB("test").getCollection("JAVA656Test"); for (int i = 0; i < 5; i++) { try { collection.findOne(); } catch (MongoException e) { System.out.println(e.getMessage()); } } System.out.println("Inserting"); collection.insert(new BasicDBObject()); System.out.println("Trying again"); collection.findOne(); System.out.println("Done"); } } {code} Output is: {noformat} Rare case where master=null, probably all servers are down Rare case where master=null, probably all servers are down Rare case where master=null, probably all servers are down Rare case where master=null, probably all servers are down Rare case where master=null, probably all servers are down Inserting Trying again Done {noformat}

    JIRA | 4 years ago | Jeff Yemin
    com.mongodb.MongoException: Rare case where master=null, probably all servers are down

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.mongodb.MongoException

      Rare case where master=null, probably all servers are down

      at com.mongodb.DBTCPConnector$MyPort.get()
    2. MongoDB Java Driver
      DB.authenticate
      1. com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:435)
      2. com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:277)
      3. com.mongodb.DBTCPConnector.call(DBTCPConnector.java:256)
      4. com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:289)
      5. com.mongodb.DB.command(DB.java:259)
      6. com.mongodb.DB.command(DB.java:241)
      7. com.mongodb.DB.command(DB.java:282)
      8. com.mongodb.DB.command(DB.java:196)
      9. com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:796)
      10. com.mongodb.DB.authenticate(DB.java:569)
      10 frames
    3. Spring Data MongoDB - Core
      MongoTemplate.collectionExists
      1. org.springframework.data.mongodb.core.MongoDbUtils.doGetDB(MongoDbUtils.java:114)
      2. org.springframework.data.mongodb.core.MongoDbUtils.getDB(MongoDbUtils.java:75)
      3. org.springframework.data.mongodb.core.SimpleMongoDbFactory.getDb(SimpleMongoDbFactory.java:118)
      4. org.springframework.data.mongodb.core.SimpleMongoDbFactory.getDb(SimpleMongoDbFactory.java:107)
      5. org.springframework.data.mongodb.core.MongoTemplate.getDb(MongoTemplate.java:1286)
      6. org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:372)
      7. org.springframework.data.mongodb.core.MongoTemplate.collectionExists(MongoTemplate.java:437)
      7 frames
    4. com.expedia.globalpayments
      MongoDatabaseChecker.check
      1. com.expedia.globalpayments.svs.checker.MongoDatabaseChecker.check(MongoDatabaseChecker.java:56)
      1 frame
    5. com.expedia.e3
      ComponentStatusUpdateTask.update
      1. com.expedia.e3.es.payment.common.status.cache.ComponentStatusUpdateTask.update(ComponentStatusUpdateTask.java:37)
      1 frame
    6. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    7. Spring Core
      MethodInvoker.invoke
      1. org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
      1 frame
    8. Spring Context Support
      QuartzJobBean.execute
      1. org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:299)
      2. org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:111)
      2 frames
    9. quartz
      SimpleThreadPool$WorkerThread.run
      1. org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      2. org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
      2 frames