com.datastax.driver.core.exceptions.NoHostAvailableException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • I test tps. But running for a while, throw exception beblow: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /x.x.x.1:9042 (com.datastax.driver.core.TransportException: [/x.x.x.2:9042] Connection has been closed), /x.x.x.3:9042 (com.datastax.driver.core.TransportException: [/x.x.x.3:9042] Connection has been closed), /x.x.x.1:9042 (com.datastax.driver.core.TransportException: [/x.x.x.2:9042] Connection has been closed))        at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)        at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289)        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:205)        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)        at com.duowan.test.cassandra.TPSTest$2.run(TPSTest.java:107)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)        at java.util.concurrent.FutureTask.run(FutureTask.java:262)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:745) my code: {code:java} PoolingOptions poolingOptions = new PoolingOptions();   poolingOptions.setMaxSimultaneousRequestsPerHostThreshold(HostDistance.LOCAL, 1024).setMaxSimultaneousRequestsPerHostThreshold(HostDistance.REMOTE, 256);   poolingOptions.setPoolTimeoutMillis(60000);   ReconnectionPolicy reconnectionPolicy = new ExponentialReconnectionPolicy(1000,10000);   SocketOptions socketOptions = new SocketOptions();   socketOptions.setReadTimeoutMillis(30000);   cluster = Cluster.builder().addContactPoint(node)     .withPoolingOptions(poolingOptions)     .withReconnectionPolicy(reconnectionPolicy)     .withSocketOptions(socketOptions)     .withProtocolVersion(ProtocolVersion.V3)     .withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)     .build();   session = cluster.connect(); {code}
    via by chenwu,
  • I'm using CassandraConnector.withSessionDo inside RDD.mapPartitions to achieve LEFT OUTER JOIN which is currently not provided by the driver. Code to reproduce the issue: https://github.com/lustefaniak/spark-cassandra-connector-bug Unfortunatelly I got hit by exception which wasn't telling me too much as I'm using single local Cassandra node for testing: {code} [error] com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [error] at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) [error] at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:271) [error] at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:185) [error] at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55) [error] at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:47) [error] at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) [error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [error] at java.lang.reflect.Method.invoke(Method.java:483) [error] at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:33) [error] at com.sun.proxy.$Proxy4.execute(Unknown Source) [error] at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) [error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [error] at java.lang.reflect.Method.invoke(Method.java:483) [error] at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:33) [error] at com.sun.proxy.$Proxy4.execute(Unknown Source) [error] at buggy.Main$$anonfun$doMappingInPartitions$1$$anonfun$apply$1$$anonfun$apply$2.apply(Main.scala:77) [error] at buggy.Main$$anonfun$doMappingInPartitions$1$$anonfun$apply$1$$anonfun$apply$2.apply(Main.scala:76) [error] at scala.collection.Iterator$$anon$11.next(Iterator.scala:370) [error] at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1555) [error] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1121) [error] at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1121) [error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848) [error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848) [error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) [error] at org.apache.spark.scheduler.Task.run(Task.scala:88) [error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [error] at java.lang.Thread.run(Thread.java:745) [error] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [error] at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:220) [error] at com.datastax.driver.core.RequestHandler.access$900(RequestHandler.java:45) [error] at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:279) [error] at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:118) [error] at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:94) [error] at com.datastax.driver.core.SessionManager.execute(SessionManager.java:565) [error] at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:602) [error] at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:98) [error] ... 26 more {code} Just before exception there is: {code} [error] 15/11/19 17:05:59 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added [error] 15/11/19 17:05:59 INFO CassandraConnector: Connected to Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added [error] 15/11/19 17:05:59 INFO CassandraConnector: Connected to Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added [error] 15/11/19 17:05:59 INFO CassandraConnector: Connected to Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added [error] 15/11/19 17:05:59 INFO CassandraConnector: Connected to Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO CassandraConnector: Disconnected from Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO CassandraConnector: Disconnected from Cassandra cluster: Mac Cluster [error] 15/11/19 17:05:59 INFO CassandraConnector: Disconnected from Cassandra cluster: Mac Cluster {code} After lot of debugging, I noticed that probably there is a race condition with marking connection as being closed/closing. As a workaround I found that by setting `spark.cassandra.connection.keep_alive_ms` config to "very-big-number-longer-than-whole-computation" I was able to run whole processing without any other issues.
    via by Lukas Stefaniak,
  • {code}21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[NonPeriodicTasks:1,5,main] org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db: The process cannot access the file because it is being used by another process. at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) ~[cassandra-all-2.1.1.jar:2.1.1] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121) ~[cassandra-all-2.1.1.jar:2.1.1] at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113) ~[cassandra-all-2.1.1.jar:2.1.1] at org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94) ~[cassandra-all-2.1.1.jar:2.1.1] at org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664) ~[cassandra-all-2.1.1.jar:2.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_71] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] Caused by: java.nio.file.FileSystemException: E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) ~[na:1.7.0_71] at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.7.0_71] at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[na:1.7.0_71] at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) ~[na:1.7.0_71] at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) ~[na:1.7.0_71] at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) ~[cassandra-all-2.1.1.jar:2.1.1] ... 11 common frames omitted{code}
    via by Ilya Komolkin,
    • com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /x.x.x.1:9042 (com.datastax.driver.core.TransportException: [/x.x.x.2:9042] Connection has been closed), /x.x.x.3:9042 (com.datastax.driver.core.TransportException: [/x.x.x.3:9042] Connection has been closed), /x.x.x.1:9042 (com.datastax.driver.core.TransportException: [/x.x.x.2:9042] Connection has been closed)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:205) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.duowan.test.cassandra.TPSTest$2.run(TPSTest.java:107) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

    Users with the same issue

    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,
    2 more bugmates