java.lang.NullPointerException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Very simply, trying to load a RDD from a Cassandra table using the Spark-Cassandra connector. Here are my complete build.gradle dependencies: {quote}compile "org.scala-lang:scala-library:2.11.8" compile "org.apache.spark:spark-core_2.11:2.1.0" compile "org.apache.spark:spark-sql_2.10:2.1.0" compile "org.slf4j:slf4j-api:latest.release" compile "com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M3" {quote} Here is the scala code I'm trying to execute: {quote}val sc = new SparkContext(new SparkConf()) val rdd = sc.cassandraTable("keyspace", "T1") rdd.persist(StorageLevel.MEMORY_AND_DISK_SER) println("Count: "+rdd.count) println("First: "+rdd.first){quote} This code is submitted via spark-submit to a 9 node Spark cluster, managed with Mesos, with each Spark executor situated locally on a Cassandra node: {quote}./bin/spark-submit \ --name "TestApp01:" \ --class com.test.testdata.aggregator \ --master mesos://y.y.y.y:7077 \ --deploy-mode cluster \ --executor-memory 16G \ --executor-cores 1 \ --driver-cores 10 \ --driver-memory 48G \ --num-executors 1 \ --conf spark.cassandra.connection.host=x.x.x.80 \ file://mnt/test1/test/jars/test-driver.jar{quote} Total resources available to the job: 360 CPUs, 1.1TB RAM, plenty of disk space. SPARK_EXECUTOR_URI=http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.3.tgz The table consists of approximately 380,000,000 rows, and has the following structure: {quote}CREATE TABLE IF NOT EXISTS T1 ( F1 bigint, F2 int, F3 bigint, F4 bigint, primary key ((F1, F2, F3) );{quote} Upon execution, the job immediately fails. The master log is attached, as well as that of one of the executors. The executor is falling with {quote}java.lang.NullPointerException at com.datastax.driver.core.Cluster$Manager.close(Cluster.java:1585) . . .{quote} traced back to many exceptions following this same pattern: {quote}17/02/13 08:39:07 WARN BlockManager: Putting block rdd_0_26 failed due to an exception 17/02/13 08:39:07 WARN BlockManager: Block rdd_0_26 could not be removed as it was not found on disk or in memory 17/02/13 08:39:07 ERROR Executor: Exception in task 26.0 in stage 0.0 (TID 26) java.lang.NullPointerException at com.datastax.driver.core.Cluster$Manager.close(Cluster.java:1585) at com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:1293) at com.datastax.driver.core.Cluster.closeAsync(Cluster.java:564) at com.datastax.driver.core.Cluster.close(Cluster.java:576) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:82) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:326) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745){quote}
    via by Joe Olson,
    • java.lang.NullPointerException at com.datastax.driver.core.Cluster$Manager.close(Cluster.java:1585) at com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:1293) at com.datastax.driver.core.Cluster.closeAsync(Cluster.java:564) at com.datastax.driver.core.Cluster.close(Cluster.java:576) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:82) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:326) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Users with the same issue

    Unknown visitor
    Unknown visitor1 times, last one,