org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: oi slice: shard3

lucene-solr-user | Markus Jelsma | 2 years ago
  1. 0

    4.10.4 - nodes up, shard without leader

    lucene-solr-user | 2 years ago | Markus Jelsma
    org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: oi slice: shard3
  2. 0

    Zookeeper issue causing all nodes to fail

    apache.org | 1 year ago
    org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: XXXslice: shard9
  3. 0

    Re: How to recover a Shard

    lucene-solr-user | 2 years ago | Erick Erickson
    org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: kla_collection slice: shard6
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    When I setup my solr cloud without blockUnknown property it works as expected. When I want to block non authenticated requests I get following stacktrace during startup: {code} 2016-06-06 05:49:16.360 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2016-06-06 05:49:16.360 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2016-06-06 05:49:16.361 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.SyncStrategy Sync replicas to https://172.30.92.66:8983/solr/testowa_shard1_replica3/ 2016-06-06 05:49:16.364 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr START replicas=[https://172.30.182.43:8983/solr/testowa_shard1_replica1/, https://172.30.182.44:8983/solr/testowa_shard1_replica2/] nUpdates=100 2016-06-06 05:49:16.365 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={ "core":"retailers_shard1_replica1", "core_node_name":"core_node1", "roles":null, "base_url":"https://172.30.182.44:8983/solr", "node_name":"172.30.182.44:8983_solr", "numShards":"1", "state":"active", "shard":"shard1", "collection":"retailers", "operation":"state"} 2016-06-06 05:49:16.481 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/retailers/state.json version: 89 2016-06-06 05:49:17.758 WARN (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr exception talking to https://172.30.182.44:8983/solr/testowa_shard1_replica2/, failed org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://172.30.182.44:8983/solr/testowa_shard1_replica2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized request, Response code: 401</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /solr/testowa_shard1_replica2/get. Reason: <pre> Unauthorized request, Response code: 401</pre></p> </body> </html> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:198) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3.00000000C403EBB0.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:17.761 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr DONE. sync failed 2016-06-06 05:49:17.761 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2016-06-06 05:49:17.763 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery 2016-06-06 05:49:17.763 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ElectionContext Canceling election /collections/testowa/leader_elect/shard1/election/240112591792046141-core_node1-n_0000001279 2016-06-06 05:49:17.766 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2016-06-06 05:49:17.767 INFO (updateExecutor-2-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.DefaultSolrCoreState Running recovery 2016-06-06 05:49:17.769 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.LeaderElector Joined leadership election with path: /collections/testowa/leader_elect/shard1/election/240112591792046141-core_node1-n_0000001282 2016-06-06 05:49:17.769 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2016-06-06 05:49:17.773 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy ###### startupVersions=[[1536114347343872000, -1536114346456776704, -1536114210874851328, -1536114165133869056, -1536113890436317184, -1536112800387039232, -1536112795602386944, -1536112079575973888, 1535485004938739712, 1535485004934545409, 1535485004934545408, 1535485004930351104, 1535485004926156801, 1535485004926156800, 1535485004919865346, 1535485004919865345, 1535485004919865344, 1535485004914622464, 1535485004908331010, 1535485004908331009, 1535485004908331008, 1535485004902039552, 1535485004898893824, 1535485004894699521, 1535485004894699520, 1535485004891553792, 1535485004887359488, 1535485004883165185, 1535485004883165184, 1535485004878970880, 1535485004875825152, 1535485004871630849, 1535485004871630848, 1535485004867436544, 1535485004864290816, 1535485004860096513, 1535485004860096512, 1535485004855902208, 1535485004851707905, 1535485004851707904, 1535485004847513600, 1535485004843319297, 1535485004843319296, 1535485004837027841, 1535485004837027840, 1535485004832833538, 1535485004832833537, 1535485004832833536, 1535485004823396353, 1535485004823396352, 1535485004819202048, 1535485004816056321, 1535485004816056320, 1535485004811862016, 1535485004807667712, 1535485004803473409, 1535485004803473408, 1535485004799279104, 1535485004795084801, 1535485004795084800, 1535485004790890496, 1535485004787744768, 1535485004786696192, 1535485004783550464, 1535485004778307585, 1535485004778307584, 1535485004775161856, 1535485004770967552, 1535485004767821824, 1535485004766773248, 1535485004763627520, 1535485004759433217, 1535485004759433216, 1535485004754190337, 1535485004754190336, 1535485004748947456, 1535485004744753153, 1535485004744753152, 1535485004740558849, 1535485004740558848, 1535485004735315968, 1535485004731121664, 1535485004727975936, 1535485004726927360, 1535485004723781633, 1535485004723781632, 1535485004722733056, 1535485004714344448, 1535485004710150145, 1535485004710150144, 1535485004703858689, 1535485004703858688]] 2016-06-06 05:49:17.775 INFO (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.LeaderElector Watching path /collections/testowa/leader_elect/shard1/election/240112591792046143-core_node3-n_0000001281 to know if I could be the leader 2016-06-06 05:49:17.777 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { "operation":"leader", "shard":"shard1", "collection":"testowa"} current state version: 38 2016-06-06 05:49:17.886 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testowa/state.json version: 1966 2016-06-06 05:49:17.889 INFO (zkCallback-4-thread-2-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testowa/state.json] for collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-06 05:49:17.891 INFO (zkCallback-4-thread-2-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [testowa] from [1966] to [1967] 2016-06-06 05:49:20.450 ERROR (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://172.30.182.43:8983/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:107) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:202) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:155) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:20.462 ERROR (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.a(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:172) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:159) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:20.464 WARN (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Failed to decrypt header, trying after refreshing the key 2016-06-06 05:49:20.474 ERROR (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://172.30.182.43:8983/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:107) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:202) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:162) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:20.477 ERROR (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.a(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:172) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:163) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:20.479 ERROR (qtp-1181790263-26) [ ] o.a.s.s.PKIAuthenticationPlugin Could not decipher a header 172.30.182.43:8983_solr iIopWis85umHbUJihyqXMQ3RIYIr3xIf0PhPkRNgmd0r434n8EB+R0+wB7GPQDTDu1zCWoW2w/DFFE43cFZ5XQ== . No principal set 2016-06-06 05:49:20.485 INFO (qtp-1181790263-26) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.s.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@6e151bd5 2016-06-06 05:49:20.485 INFO (qtp-1181790263-26) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.s.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [READ], collections: [testowa,], Path: [/get] path : /get params :distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2 2016-06-06 05:49:20.506 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { "operation":"leader", "shard":"shard1", "collection":"testowa"} current state version: 38 2016-06-06 05:49:20.511 INFO (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_0000000150) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testowa/state.json version: 1967 2016-06-06 05:49:20.513 INFO (zkCallback-4-thread-2-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testowa/state.json] for collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-06 05:49:20.515 INFO (zkCallback-4-thread-2-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [testowa] from [1967] to [1968] 2016-06-06 05:49:21.782 ERROR (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Error while trying to recover. core=testowa_shard1_replica3:org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: testowa slice: shard1 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:609) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:595) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3.00000000C403EBB0.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:21.783 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates. 2016-06-06 05:49:21.783 ERROR (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (0) 2016-06-06 05:49:21.783 INFO (recoveryExecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.RecoveryStrategy Wait [2.0] seconds before trying to recover again (attempt=1) 2016-06-06 05:49:23.166 ERROR (qtp-1181790263-30) [ ] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from : https://172.30.182.44:8983/solr org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html> <head> <meta http-equiv="Content-' at org.noggit.JSONParser.err(JSONParser.java:356) at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:712) at org.noggit.JSONParser.next(JSONParser.java:886) at org.noggit.JSONParser.nextEvent(JSONParser.java:930) at org.noggit.ObjectBuilder.<init>(ObjectBuilder.java:44) at org.noggit.ObjectBuilder.getVal(ObjectBuilder.java:37) at org.apache.solr.common.util.Utils.fromJSON(Utils.java:107) at org.apache.solr.security.PKIAuthenticationPlugin.getRemotePublicKey(PKIAuthenticationPlugin.java:202) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:155) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) 2016-06-06 05:49:23.169 ERROR (qtp-1181790263-30) [ ] o.a.s.s.PKIAuthenticationPlugin Decryption failed , key must be wrong java.security.InvalidKeyException: No installed provider supports this key: (null) at javax.crypto.Cipher.a(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at javax.crypto.Cipher.init(Unknown Source) at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277) at org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:172) at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:159) at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118) at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:283) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:518) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:785) {code}

    Apache's JIRA Issue Tracker | 6 months ago | Piotr Tempes
    org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: testowa slice: shard1
  6. 0

    Lucene - Java Developer - [JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 64 - Failure

    nabble.com | 6 months ago
    org.apache.solr.common.SolrException: No registered leader was found after waiting for 30000ms , collection: halfdeletedcollection2 slice: shard1

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.solr.common.SolrException

      No registered leader was found after waiting for 4000ms , collection: oi slice: shard3

      at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry()
    2. Apache Solr Solrj
      ZkStateReader.getLeaderRetry
      1. org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:568)
      2. org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:551)
      2 frames
    3. Apache Solr Core
      RecoveryStrategy.run
      1. org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:330)
      2. org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
      2 frames