com.mongodb.MongoException$Network: Operation on server c151.candidate.6.mongolayer.com:10151 failed

JIRA | Mathieu Carbou | 3 years ago
  1. 0

    Hello, We were impacted by the bugs resolved in version 2.12.1 (https://github.com/mongodb/mongo-java-driver/releases/tag/r2.12.1) Thus we upgraded this morning to the new driver and after the production push we get a lot of errors coming from MongoDriver, which crashes preventing web request to be fulfilled, thus having a high rate of http status code 500 errors. Our application is hosted on Heroku, where there were also no issue at the moment. We are using a MongoHQ Master/Slave replication set running MongoDB 2.6.0, which run well. There were no failover that could cause the issue. I cannot say whether Heroku platform or MongoHQ could have broke the DB connection for some reason. On MongoHQ, the current master is candidate.6, candidate.5 being the slave. There is also another slave and an arbitrer, but they are not used in the connection string. The connection string is: {code:java} "mongodb://<user>:<pass>@candidate.6.mongolayer.com:10151,candidate.5.mongolayer.com:10151/guestful" {code} In any case, here are the exceptions we get: {code:java} Caused by: com.mongodb.MongoException$Network: Read operation to server c151.candidate.6.mongolayer.com:10151 failed on database guestful at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:308) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:279) at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84) at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66) at com.mongodb.DBCollection.findOne(DBCollection.java:870) at com.mongodb.DBCollection.findOne(DBCollection.java:844) at com.mongodb.DBCollection.findOne(DBCollection.java:802) at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at com.guestful.backend.service.DB$GCollection.findOne(DB.groovy:186) at com.guestful.backend.service.DB$GCollection$findOne$12.call(Unknown Source) at com.guestful.backend.rest.service.RestaurantHelper.findOne(RestaurantHelper.groovy:61) at com.guestful.backend.rest.service.RestaurantHelper$findOne.call(Unknown Source) at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilities(RestaurantAvailabilityResource.groovy:125) at com.guestful.backend.rest.RestaurantAvailabilityResource$getAvailabilities.callCurrent(Unknown Source) at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilitiesForRestaurant(RestaurantAvailabilityResource.groovy:44) at sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ... 59 common frames omitted Caused by: java.io.EOFException: null at org.bson.io.Bits.readFully(Bits.java:50) at org.bson.io.Bits.readFully(Bits.java:35) at org.bson.io.Bits.readFully(Bits.java:30) at com.mongodb.Response.<init>(Response.java:42) at com.mongodb.DBPort$1.execute(DBPort.java:141) at com.mongodb.DBPort$1.execute(DBPort.java:135) at com.mongodb.DBPort.doOperation(DBPort.java:164) at com.mongodb.DBPort.call(DBPort.java:135) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:300) ... 109 common frames omitted {code} and {code:java} com.mongodb.MongoException$Network: Operation on server c151.candidate.6.mongolayer.com:10151 failed at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:225) at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:489) at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:428) at com.mongodb.DBCollectionImpl.updateWithCommandProtocol(DBCollectionImpl.java:415) at com.mongodb.DBCollectionImpl.update(DBCollectionImpl.java:267) at com.mongodb.DBCollection.update(DBCollection.java:191) at com.mongodb.DBCollection.update(DBCollection.java:224) at sun.reflect.GeneratedMethodAccessor491.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128) at com.guestful.backend.service.DB$GCollection.update(DB.groovy:202) at com.guestful.backend.service.DB$GCollection.update(DB.groovy) at com.guestful.backend.service.DB$GCollection$update$15.call(Unknown Source) at com.guestful.backend.event.handler.Stats.onAvailabilityCallEvent(Stats.groovy:129) at sun.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at com.ovea.tajin.framework.async.guava.EventHandler.handleEvent(EventHandler.groovy:72) at com.ovea.tajin.framework.async.guava.EventHandler$handleEvent.call(Unknown Source) at com.ovea.tajin.framework.async.guava.AsyncEventBus$1.run(AsyncEventBus.groovy:100) at java_lang_Runnable$run.call(Unknown Source) at com.ovea.tajin.framework.async.guice.AsyncModule$Task.run(AsyncModule.groovy:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.EOFException: null at org.bson.io.Bits.readFully(Bits.java:50) at org.bson.io.Bits.readFully(Bits.java:35) at org.bson.io.Bits.readFully(Bits.java:30) at com.mongodb.Response.<init>(Response.java:42) at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:492) at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:489) at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:218) ... 54 common frames omitted {code} There are very frequent compared to before 2.12.1 driver version, and cause requests to fail.

    JIRA | 3 years ago | Mathieu Carbou
    com.mongodb.MongoException$Network: Operation on server c151.candidate.6.mongolayer.com:10151 failed
  2. 0

    User and support list for MongoDB, a NoSQL database. ()

    gmane.org | 1 year ago
    com.mongodb.MongoException$Network: can't say something
  3. 0

    Mongodb Sharded - Error on 100 threads trying to insert data to one collection

    Google Groups | 6 years ago | Oded Maimon
    com.mongodb.MongoException$Network: can't say something
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    help the mongodb run more and more slowly

    Google Groups | 6 years ago | qiujiawei
    com.mongodb.MongoException$Network: can't say something
  6. 0

    [mongodb-user] Mongodb Sharded - Error on 100 threads trying to insert data to one collection

    Google Groups | 6 years ago | Oded Maimon
    com.mongodb.MongoException$Network: can't say something

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.EOFException

      null

      at org.bson.io.Bits.readFully()
    2. MongoDB Java Driver
      DBCollection.update
      1. org.bson.io.Bits.readFully(Bits.java:50)
      2. org.bson.io.Bits.readFully(Bits.java:35)
      3. org.bson.io.Bits.readFully(Bits.java:30)
      4. com.mongodb.Response.<init>(Response.java:42)
      5. com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:492)
      6. com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:489)
      7. com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:218)
      8. com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:489)
      9. com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:428)
      10. com.mongodb.DBCollectionImpl.updateWithCommandProtocol(DBCollectionImpl.java:415)
      11. com.mongodb.DBCollectionImpl.update(DBCollectionImpl.java:267)
      12. com.mongodb.DBCollection.update(DBCollection.java:191)
      13. com.mongodb.DBCollection.update(DBCollection.java:224)
      13 frames
    3. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor491.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:491)
      3 frames
    4. Groovy
      MetaMethod$doMethodInvoke.call
      1. org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      2. groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      3. groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source)
      3 frames
    5. com.gmongo.internal
      Patcher._invoke
      1. com.gmongo.internal.Patcher._invoke(Patcher.groovy:49)
      1 frame
    6. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:491)
      3 frames
    7. Groovy
      PogoMetaClassSite.callCurrent
      1. org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      2. groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      3. org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      4. groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
      5. org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      5 frames
    8. com.gmongo.internal
      Patcher$__patchInternal_closure1.doCall
      1. com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38)
      1 frame
    9. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:491)
      3 frames
    10. Groovy
      AbstractCallSite.call
      1. org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      2. org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80)
      3. groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108)
      4. groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
      5. org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35)
      6. org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      6 frames
    11. com.guestful.backend
      Stats.onAvailabilityCallEvent
      1. com.guestful.backend.service.DB$GCollection.update(DB.groovy:202)
      2. com.guestful.backend.service.DB$GCollection.update(DB.groovy)
      3. com.guestful.backend.service.DB$GCollection$update$15.call(Unknown Source)
      4. com.guestful.backend.event.handler.Stats.onAvailabilityCallEvent(Stats.groovy:129)
      4 frames
    12. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor542.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:491)
      4. sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source)
      5. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      6. java.lang.reflect.Method.invoke(Method.java:491)
      6 frames
    13. Groovy
      AbstractCallSite.call
      1. org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189)
      2. org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
      3. org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
      4. org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55)
      5. org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
      5 frames
    14. com.ovea.tajin
      AsyncEventBus$1.run
      1. com.ovea.tajin.framework.async.guava.EventHandler.handleEvent(EventHandler.groovy:72)
      2. com.ovea.tajin.framework.async.guava.EventHandler$handleEvent.call(Unknown Source)
      3. com.ovea.tajin.framework.async.guava.AsyncEventBus$1.run(AsyncEventBus.groovy:100)
      3 frames
    15. Unknown
      java_lang_Runnable$run.call
      1. java_lang_Runnable$run.call(Unknown Source)
      1 frame
    16. com.ovea.tajin
      AsyncModule$Task.run
      1. com.ovea.tajin.framework.async.guice.AsyncModule$Task.run(AsyncModule.groovy:154)
      1 frame
    17. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      3. java.lang.Thread.run(Thread.java:724)
      3 frames