com.mongodb.MongoException: Invalid modifier specified$barf

JIRA | Leif Mortenson | 6 years ago
  1. 0

    Hello, I got the following errors, when the application try to insert the data in DB. These error started appearing on initial load but after few minutes everything backs to normal. I found that there were three 500 application errors ( one on each server except its-d01) I looked at the tomcat logs and saw the same three errors which were complaining about mongos. The time of these error entries matches exactly with the error These issues that I noticed got fixed automatically without any changes/intervention from my side. However, what I am not able to comprehend is that if the errors show up in apache logs. tomcat logs and mongos logs, how did the data pertaining to these usernames(a17828, a052l508) still end up in the DB - in spite of these critical errors. Not sure why it happened and how to prevent it from happening in the future. Here is the current configuration:- Shards 2 Configuration Servers :- 3 Mongos :- 4 > db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard01", "host" : "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" } { "_id" : "shard02", "host" : "shard02/its-d02.db.aol.com:4308,its-m02.db.aol.com:4308" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard01" } { "_id" : "historyDb", "partitioned" : true, "primary" : "shard01" } historyDb.History chunks: shard02 29 shard01 29 too many chunksn to print, use verbose if you want to force print { "_id" : "reputationDb", "partitioned" : true, "primary" : "shard01" } reputationDb.Karma chunks: shard02 40 shard01 41 too many chunksn to print, use verbose if you want to force print shard01:PRIMARY> rs.status() { "set" : "shard01", "date" : ISODate("2011-09-22T17:58:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "its-d01.db.aol.com:4308", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "self" : true }, { "_id" : 1, "name" : "its-m01.db.aol.com:4308", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311026, "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") }, { "_id" : 2, "name" : "its-m02.db.aol.com:4309", "health" : 1, "state" : 7, "stateStr" : "ARBITER", "uptime" : 311026, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") } > db.stats() { "raw" : { "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" : { "db" : "test", "collections" : 3, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } }, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } Here are the logs to support my research. ITS-M01 ----------- Apache logs:- reputation.aol.com 172.19.150.23 - - [20/Sep/2011:15:50:39 --0400] "GET /its/api/submit_claim?username=a052l508&user_id=a052l508&context=login&source=imagination&reason=evil&confidence=1&value=fraud&ip=172.19.150.23&locale=en-US HTTP/1.1" 500 2494 "-" "-" Tnjuj6wVzHQAAGK2M-gAAAIt "-" /20110920/20110920-1550/20110920-155039-Tnjuj6wVzHQAAGK2M-gAAAIt 0 997 md5:0474fc81293193ac2f05ed83a2200993 Tomcat Logs:- ---------------------- 2011-09-20 15:50:39,046 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Executing BasicDBObjectCommand { "username" : { "$in" : [ "a052l508"]}} for reputationDb.Karma cursor 2011-09-20 15:50:39,048 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - MongoDB query plan com.aol.trust.model.Karma$DirtyableDBObject $1@1cad1378 cursor = "null" nscanned = "0" n = "0" millis = "null" 2011-09-20 15:50:39,050 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Saving Karma identified by {_id=4e78ee8f6db5deac51214377, user name=a052l508} with document {createTime=Tue Sep 20 15:50:39 EDT 2011, __olv=1, username=a052l508, _id=4e78ee8f6db5deac51214377, lastUpdate=Tue Sep 20 15:50:39 EDT 2011, userId=a052l508, state=fraud, contexts={login=com.aol.trust.model.Context@26d939fd}, lastContext=login} 2011-09-20 15:50:39,052 [TP-Processor37] INFO com.aol.trust.timer.db.write.karma - ITS_DB_WRITE_KARMA_TIME - [a052l508] 2 millisecs 2011-09-20 15:50:39,052 [TP-Processor37] DEBUG com.aol.trust.model.dao.HistoryDAO$2 - Saving History identified by {_id=4e78ee8f6db5deac51214378, username=a052l508} with document {__olv=1, username=a052l508, time=Tue Sep 20 15:50:39 EDT 2011, scope=login, _id=4e78ee8f6db5deac51214378, source= imagination, reason=evil, userId=a052l508, xff=null, value=fraud, confidence=1, ip=172.19.150.23} 2011-09-20 15:50:39,117 [TP-Processor37] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/its].[ApiServlet] - Servlet.service () for servlet ApiServlet threw exception com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } at com.mongodb.CommandResult.getException(CommandResult.java:82) at com.mongodb.CommandResult.throwOnError(CommandResult.java:116) at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:126) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:148) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:132) at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:343) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.trueSave(MongoDBQueryModelDAO.java:588) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.save(MongoDBQueryModelDAO.java:257) at com.aol.trust.IdentityTrustServiceImplementation.saveToHistory(IdentityTrustServiceImplementation.java:310) at com.aol.trust.IdentityTrustServiceImplementation.submitClaim(IdentityTrustServiceImplementation.java:133) at com.aol.trust.SubmitClaimHandler.processRequest(SubmitClaimHandler.java:73) at com.aol.trust.ApiServlet.processApiCall(ApiServlet.java:131) at com.aol.trust.ApiServlet.doGet(ApiServlet.java:104) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:204) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) Mongos ---------- Tue Sep 20 15:50:39 [conn8] setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, asserti on: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Tue Sep 20 15:50:39 [conn8] Assertion: 10429:setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 200 0|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } 0x5204fa 0x6a15ed 0x6a1152 0x6a1152 0x6a1152 0x578ce6 0x57715a 0x6311ed 0x6395bc 0x63d433 0x6682f2 0x67d187 0x580b7c 0x6a50a0 0x321420673d 0x32136d44bd /data/servers/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a) [0x5204fa] /data/servers/mongodb/bin/mongos [0x6a15ed] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S 7_bi+0x16) [0x578ce6] /data/servers/mongodb/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0xba) [0x57715a] /data/servers/mongodb/bin/mongos(_ZN5mongo8Strategy7doWriteEiRNS_7RequestERKNS_5ShardEb+0xdd) [0x6311ed] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7_updateERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrINS_12ChunkManagerEEE+0xd1c) [0x6395bc] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x2b3) [0x63d433] /data/servers/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x172) [0x6682f2] /data/servers/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77) [0x67d187] /data/servers/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c) [0x580b7c] /data/servers/mongodb/bin/mongos(thread_proxy+0x80) [0x6a50a0] /lib64/libpthread.so.0 [0x321420673d] /lib64/libc.so.6(clone+0x6d) [0x32136d44bd] Tue Sep 20 15:50:39 [conn8] ~ScopedDBConnection: _conn != null Tue Sep 20 15:50:39 [conn8] AssertionException in process: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Mongod ------------ > db.Karma.find({"username" : "a052l508"}) { "_id" : ObjectId("4e78ee8f6db5deac51214377"), "createTime" : ISODate("2011-09-20T19:50:39.046Z"), "__olv" : 1, "username" : "a052l508", "lastUpdate" : ISODate("2011-09-20T19:50:39.046Z"), "userId" : "a052l508", "state" : "fraud", "contexts" : { "login" : { "time" : ISODate("2011-09-20T19:50:39.046Z"), "scope" : "login", "source" : "imagination", "reason" : "evil", "xff" : null, "value" : "fraud", "confidence" : "1", "ip" : "172.19.150.23" } }, "lastContext" : "login" }

    JIRA | 5 years ago | Ajay Batheja
    com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 }
  2. 0

    Hello, I got the following errors, when the application try to insert the data in DB. These error started appearing on initial load but after few minutes everything backs to normal. I found that there were three 500 application errors ( one on each server except its-d01) I looked at the tomcat logs and saw the same three errors which were complaining about mongos. The time of these error entries matches exactly with the error These issues that I noticed got fixed automatically without any changes/intervention from my side. However, what I am not able to comprehend is that if the errors show up in apache logs. tomcat logs and mongos logs, how did the data pertaining to these usernames(a17828, a052l508) still end up in the DB - in spite of these critical errors. Not sure why it happened and how to prevent it from happening in the future. Here is the current configuration:- Shards 2 Configuration Servers :- 3 Mongos :- 4 > db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard01", "host" : "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" } { "_id" : "shard02", "host" : "shard02/its-d02.db.aol.com:4308,its-m02.db.aol.com:4308" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard01" } { "_id" : "historyDb", "partitioned" : true, "primary" : "shard01" } historyDb.History chunks: shard02 29 shard01 29 too many chunksn to print, use verbose if you want to force print { "_id" : "reputationDb", "partitioned" : true, "primary" : "shard01" } reputationDb.Karma chunks: shard02 40 shard01 41 too many chunksn to print, use verbose if you want to force print shard01:PRIMARY> rs.status() { "set" : "shard01", "date" : ISODate("2011-09-22T17:58:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "its-d01.db.aol.com:4308", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "self" : true }, { "_id" : 1, "name" : "its-m01.db.aol.com:4308", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311026, "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") }, { "_id" : 2, "name" : "its-m02.db.aol.com:4309", "health" : 1, "state" : 7, "stateStr" : "ARBITER", "uptime" : 311026, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") } > db.stats() { "raw" : { "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" : { "db" : "test", "collections" : 3, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } }, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } Here are the logs to support my research. ITS-M01 ----------- Apache logs:- reputation.aol.com 172.19.150.23 - - [20/Sep/2011:15:50:39 --0400] "GET /its/api/submit_claim?username=a052l508&user_id=a052l508&context=login&source=imagination&reason=evil&confidence=1&value=fraud&ip=172.19.150.23&locale=en-US HTTP/1.1" 500 2494 "-" "-" Tnjuj6wVzHQAAGK2M-gAAAIt "-" /20110920/20110920-1550/20110920-155039-Tnjuj6wVzHQAAGK2M-gAAAIt 0 997 md5:0474fc81293193ac2f05ed83a2200993 Tomcat Logs:- ---------------------- 2011-09-20 15:50:39,046 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Executing BasicDBObjectCommand { "username" : { "$in" : [ "a052l508"]}} for reputationDb.Karma cursor 2011-09-20 15:50:39,048 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - MongoDB query plan com.aol.trust.model.Karma$DirtyableDBObject $1@1cad1378 cursor = "null" nscanned = "0" n = "0" millis = "null" 2011-09-20 15:50:39,050 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Saving Karma identified by {_id=4e78ee8f6db5deac51214377, user name=a052l508} with document {createTime=Tue Sep 20 15:50:39 EDT 2011, __olv=1, username=a052l508, _id=4e78ee8f6db5deac51214377, lastUpdate=Tue Sep 20 15:50:39 EDT 2011, userId=a052l508, state=fraud, contexts={login=com.aol.trust.model.Context@26d939fd}, lastContext=login} 2011-09-20 15:50:39,052 [TP-Processor37] INFO com.aol.trust.timer.db.write.karma - ITS_DB_WRITE_KARMA_TIME - [a052l508] 2 millisecs 2011-09-20 15:50:39,052 [TP-Processor37] DEBUG com.aol.trust.model.dao.HistoryDAO$2 - Saving History identified by {_id=4e78ee8f6db5deac51214378, username=a052l508} with document {__olv=1, username=a052l508, time=Tue Sep 20 15:50:39 EDT 2011, scope=login, _id=4e78ee8f6db5deac51214378, source= imagination, reason=evil, userId=a052l508, xff=null, value=fraud, confidence=1, ip=172.19.150.23} 2011-09-20 15:50:39,117 [TP-Processor37] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/its].[ApiServlet] - Servlet.service () for servlet ApiServlet threw exception com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } at com.mongodb.CommandResult.getException(CommandResult.java:82) at com.mongodb.CommandResult.throwOnError(CommandResult.java:116) at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:126) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:148) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:132) at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:343) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.trueSave(MongoDBQueryModelDAO.java:588) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.save(MongoDBQueryModelDAO.java:257) at com.aol.trust.IdentityTrustServiceImplementation.saveToHistory(IdentityTrustServiceImplementation.java:310) at com.aol.trust.IdentityTrustServiceImplementation.submitClaim(IdentityTrustServiceImplementation.java:133) at com.aol.trust.SubmitClaimHandler.processRequest(SubmitClaimHandler.java:73) at com.aol.trust.ApiServlet.processApiCall(ApiServlet.java:131) at com.aol.trust.ApiServlet.doGet(ApiServlet.java:104) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:204) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) Mongos ---------- Tue Sep 20 15:50:39 [conn8] setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, asserti on: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Tue Sep 20 15:50:39 [conn8] Assertion: 10429:setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 200 0|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } 0x5204fa 0x6a15ed 0x6a1152 0x6a1152 0x6a1152 0x578ce6 0x57715a 0x6311ed 0x6395bc 0x63d433 0x6682f2 0x67d187 0x580b7c 0x6a50a0 0x321420673d 0x32136d44bd /data/servers/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a) [0x5204fa] /data/servers/mongodb/bin/mongos [0x6a15ed] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S 7_bi+0x16) [0x578ce6] /data/servers/mongodb/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0xba) [0x57715a] /data/servers/mongodb/bin/mongos(_ZN5mongo8Strategy7doWriteEiRNS_7RequestERKNS_5ShardEb+0xdd) [0x6311ed] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7_updateERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrINS_12ChunkManagerEEE+0xd1c) [0x6395bc] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x2b3) [0x63d433] /data/servers/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x172) [0x6682f2] /data/servers/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77) [0x67d187] /data/servers/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c) [0x580b7c] /data/servers/mongodb/bin/mongos(thread_proxy+0x80) [0x6a50a0] /lib64/libpthread.so.0 [0x321420673d] /lib64/libc.so.6(clone+0x6d) [0x32136d44bd] Tue Sep 20 15:50:39 [conn8] ~ScopedDBConnection: _conn != null Tue Sep 20 15:50:39 [conn8] AssertionException in process: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Mongod ------------ > db.Karma.find({"username" : "a052l508"}) { "_id" : ObjectId("4e78ee8f6db5deac51214377"), "createTime" : ISODate("2011-09-20T19:50:39.046Z"), "__olv" : 1, "username" : "a052l508", "lastUpdate" : ISODate("2011-09-20T19:50:39.046Z"), "userId" : "a052l508", "state" : "fraud", "contexts" : { "login" : { "time" : ISODate("2011-09-20T19:50:39.046Z"), "scope" : "login", "source" : "imagination", "reason" : "evil", "xff" : null, "value" : "fraud", "confidence" : "1", "ip" : "172.19.150.23" } }, "lastContext" : "login" }

    JIRA | 5 years ago | Ajay Batheja
    com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 }
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Hello, I got the following errors, when the application try to insert the data in DB. These error started appearing on initial load but after few minutes everything backs to normal. I found that there were three 500 application errors ( one on each server except its-d01) I looked at the tomcat logs and saw the same three errors which were complaining about mongos. The time of these error entries matches exactly with the error These issues that I noticed got fixed automatically without any changes/intervention from my side. However, what I am not able to comprehend is that if the errors show up in apache logs. tomcat logs and mongos logs, how did the data pertaining to these usernames(a17828, a052l508) still end up in the DB - in spite of these critical errors. Not sure why it happened and how to prevent it from happening in the future. Here is the current configuration:- Shards 2 Configuration Servers :- 3 Mongos :- 4 > db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard01", "host" : "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" } { "_id" : "shard02", "host" : "shard02/its-d02.db.aol.com:4308,its-m02.db.aol.com:4308" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "shard01" } { "_id" : "historyDb", "partitioned" : true, "primary" : "shard01" } historyDb.History chunks: shard02 29 shard01 29 too many chunksn to print, use verbose if you want to force print { "_id" : "reputationDb", "partitioned" : true, "primary" : "shard01" } reputationDb.Karma chunks: shard02 40 shard01 41 too many chunksn to print, use verbose if you want to force print shard01:PRIMARY> rs.status() { "set" : "shard01", "date" : ISODate("2011-09-22T17:58:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "its-d01.db.aol.com:4308", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "self" : true }, { "_id" : 1, "name" : "its-m01.db.aol.com:4308", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311026, "optime" : { "t" : 1316714290000, "i" : 2 }, "optimeDate" : ISODate("2011-09-22T17:58:10Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") }, { "_id" : 2, "name" : "its-m02.db.aol.com:4309", "health" : 1, "state" : 7, "stateStr" : "ARBITER", "uptime" : 311026, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2011-09-22T17:58:12Z") } > db.stats() { "raw" : { "shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308" : { "db" : "test", "collections" : 3, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } }, "objects" : 5, "avgObjSize" : 40, "dataSize" : 200, "storageSize" : 7168, "numExtents" : 3, "indexes" : 1, "indexSize" : 8192, "fileSize" : 201326592, "ok" : 1 } Here are the logs to support my research. ITS-M01 ----------- Apache logs:- reputation.aol.com 172.19.150.23 - - [20/Sep/2011:15:50:39 --0400] "GET /its/api/submit_claim?username=a052l508&user_id=a052l508&context=login&source=imagination&reason=evil&confidence=1&value=fraud&ip=172.19.150.23&locale=en-US HTTP/1.1" 500 2494 "-" "-" Tnjuj6wVzHQAAGK2M-gAAAIt "-" /20110920/20110920-1550/20110920-155039-Tnjuj6wVzHQAAGK2M-gAAAIt 0 997 md5:0474fc81293193ac2f05ed83a2200993 Tomcat Logs:- ---------------------- 2011-09-20 15:50:39,046 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Executing BasicDBObjectCommand { "username" : { "$in" : [ "a052l508"]}} for reputationDb.Karma cursor 2011-09-20 15:50:39,048 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - MongoDB query plan com.aol.trust.model.Karma$DirtyableDBObject $1@1cad1378 cursor = "null" nscanned = "0" n = "0" millis = "null" 2011-09-20 15:50:39,050 [TP-Processor37] DEBUG com.aol.trust.model.dao.KarmaDAO$1 - Saving Karma identified by {_id=4e78ee8f6db5deac51214377, user name=a052l508} with document {createTime=Tue Sep 20 15:50:39 EDT 2011, __olv=1, username=a052l508, _id=4e78ee8f6db5deac51214377, lastUpdate=Tue Sep 20 15:50:39 EDT 2011, userId=a052l508, state=fraud, contexts={login=com.aol.trust.model.Context@26d939fd}, lastContext=login} 2011-09-20 15:50:39,052 [TP-Processor37] INFO com.aol.trust.timer.db.write.karma - ITS_DB_WRITE_KARMA_TIME - [a052l508] 2 millisecs 2011-09-20 15:50:39,052 [TP-Processor37] DEBUG com.aol.trust.model.dao.HistoryDAO$2 - Saving History identified by {_id=4e78ee8f6db5deac51214378, username=a052l508} with document {__olv=1, username=a052l508, time=Tue Sep 20 15:50:39 EDT 2011, scope=login, _id=4e78ee8f6db5deac51214378, source= imagination, reason=evil, userId=a052l508, xff=null, value=fraud, confidence=1, ip=172.19.150.23} 2011-09-20 15:50:39,117 [TP-Processor37] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/its].[ApiServlet] - Servlet.service () for servlet ApiServlet threw exception com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } at com.mongodb.CommandResult.getException(CommandResult.java:82) at com.mongodb.CommandResult.throwOnError(CommandResult.java:116) at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:126) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:148) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:132) at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:343) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.trueSave(MongoDBQueryModelDAO.java:588) at org.iternine.jeppetto.dao.mongodb.MongoDBQueryModelDAO.save(MongoDBQueryModelDAO.java:257) at com.aol.trust.IdentityTrustServiceImplementation.saveToHistory(IdentityTrustServiceImplementation.java:310) at com.aol.trust.IdentityTrustServiceImplementation.submitClaim(IdentityTrustServiceImplementation.java:133) at com.aol.trust.SubmitClaimHandler.processRequest(SubmitClaimHandler.java:73) at com.aol.trust.ApiServlet.processApiCall(ApiServlet.java:131) at com.aol.trust.ApiServlet.doGet(ApiServlet.java:104) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:204) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) Mongos ---------- Tue Sep 20 15:50:39 [conn8] setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, asserti on: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Tue Sep 20 15:50:39 [conn8] Assertion: 10429:setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 200 0|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } 0x5204fa 0x6a15ed 0x6a1152 0x6a1152 0x6a1152 0x578ce6 0x57715a 0x6311ed 0x6395bc 0x63d433 0x6682f2 0x67d187 0x580b7c 0x6a50a0 0x321420673d 0x32136d44bd /data/servers/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a) [0x5204fa] /data/servers/mongodb/bin/mongos [0x6a15ed] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos [0x6a1152] /data/servers/mongodb/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S 7_bi+0x16) [0x578ce6] /data/servers/mongodb/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0xba) [0x57715a] /data/servers/mongodb/bin/mongos(_ZN5mongo8Strategy7doWriteEiRNS_7RequestERKNS_5ShardEb+0xdd) [0x6311ed] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7_updateERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrINS_12ChunkManagerEEE+0xd1c) [0x6395bc] /data/servers/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x2b3) [0x63d433] /data/servers/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x172) [0x6682f2] /data/servers/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77) [0x67d187] /data/servers/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c) [0x580b7c] /data/servers/mongodb/bin/mongos(thread_proxy+0x80) [0x6a50a0] /lib64/libpthread.so.0 [0x321420673d] /lib64/libc.so.6(clone+0x6d) [0x32136d44bd] Tue Sep 20 15:50:39 [conn8] ~ScopedDBConnection: _conn != null Tue Sep 20 15:50:39 [conn8] AssertionException in process: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, assertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 } Mongod ------------ > db.Karma.find({"username" : "a052l508"}) { "_id" : ObjectId("4e78ee8f6db5deac51214377"), "createTime" : ISODate("2011-09-20T19:50:39.046Z"), "__olv" : 1, "username" : "a052l508", "lastUpdate" : ISODate("2011-09-20T19:50:39.046Z"), "userId" : "a052l508", "state" : "fraud", "contexts" : { "login" : { "time" : ISODate("2011-09-20T19:50:39.046Z"), "scope" : "login", "source" : "imagination", "reason" : "evil", "xff" : null, "value" : "fraud", "confidence" : "1", "ip" : "172.19.150.23" } }, "lastContext" : "login" }

    JIRA | 5 years ago | Ajay Batheja
    com.mongodb.MongoException: setShardVersion failed host[shard01/its-d01.db.aol.com:4308,its-m01.db.aol.com:4308] { oldVersion: Timestamp 2000|5, as sertion: "assertion s/d_state.cpp:501", errmsg: "db assertion failure", ok: 0.0 }
  5. 0

    invalid-sort-specified-for-tailable-cursor

    Google Groups | 3 months ago | Prasanna Ganesh
    com.mongodb.MongoException: Unable to execute query: error processing query: ns=local.oplog.rs limit=0 skip=0 Tree: $and Sort: { $natural: -1 } Proj: {} invalid sort specified for tailable cursor: { $natural: -1 }

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.mongodb.MongoException

      Invalid modifier specified$barf

      at com.mongodb.DBTCPConnector._checkWriteError()
    2. MongoDB Java Driver
      DBApiLayer$MyCollection.update
      1. com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:141)
      2. com.mongodb.DBTCPConnector.say(DBTCPConnector.java:162)
      3. com.mongodb.DBTCPConnector.say(DBTCPConnector.java:146)
      4. com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:335)
      4 frames