java.lang.IllegalStateException: Spark program execution failed.

Cask Community Issue Tracker | Rohit Sinha | 2 years ago
tip
Do you find the tips below useful? Click on the to mark them and say thanks to rp . Or join the community to write better ones.
  1. 0
    samebug tip
    Check if there is no nexus running. If not, delete sonatype-work/nexus/nexus.lock .
  2. 0

    When closing PathChildrenCache, and immediately afterwards closing CuratorFramework, some ERROR-level stack traces are logged. This was previously reported on the mailing list: http://curator.markmail.org/thread/bmfr62ekx5p2vv7f The cause is that the BackgroundCallback defined in PathChildrenCache.refresh() will, when triggered, perform some more ZooKeeper operations. Thus one can get in sequences such as: * operation with BackgroundCallback is submitted * processResult of the BackgroundCallback is called * PathChildrenCache is closed * CuratorFramework is closed * processResult, which is running on another thread, comes to the point it does operations on ZooKeeper, which fail because ZooKeeper is closed. There is no real impact on the application, it is just for log-esthetical reasons that I'd like to avoid it. In the more common case, the processResult will receive an IllegalStateException, which could be easily catched and ignored in PathChildrenCache if the PathChildrenCache is closed: {noformat} 14/10/30 11:24:51 ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up java.lang.IllegalStateException: instance must be started before calling this method at com.google.common.base.Preconditions.checkState(Preconditions.java:149) at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:360) at org.apache.curator.framework.recipes.cache.PathChildrenCache.getDataAndStat(PathChildrenCache.java:545) at org.apache.curator.framework.recipes.cache.PathChildrenCache.processChildren(PathChildrenCache.java:668) at org.apache.curator.framework.recipes.cache.PathChildrenCache.access$200(PathChildrenCache.java:68) at org.apache.curator.framework.recipes.cache.PathChildrenCache$4.processResult(PathChildrenCache.java:490) at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:715) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:502) at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) {noformat} But sometimes it also fails with other async operations deeper down: {noformat} 14/10/30 11:24:51 ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up java.lang.IllegalStateException: Client is not started at com.google.common.base.Preconditions.checkState(Preconditions.java:149) at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113) at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:474) at org.apache.curator.framework.imps.GetDataBuilderImpl.performBackgroundOperation(GetDataBuilderImpl.java:263) at org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:65) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:789) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:487) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:275) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.cache.PathChildrenCache.getDataAndStat(PathChildrenCache.java:545) at org.apache.curator.framework.recipes.cache.PathChildrenCache.processChildren(PathChildrenCache.java:668) at org.apache.curator.framework.recipes.cache.PathChildrenCache.access$200(PathChildrenCache.java:68) at org.apache.curator.framework.recipes.cache.PathChildrenCache$4.processResult(PathChildrenCache.java:490) at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:715) at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:502) at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) {noformat} Therefore I have created a patch where PathChildrenCache.close() will wait until the possibly running BackgroundCallback is finished. I will also attach a small class that illustrates the problem.

    Apache's JIRA Issue Tracker | 2 years ago | Bruno Dumon
    java.lang.IllegalStateException: instance must be started before calling this method
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  1. jf-ast 3 times, last 1 month ago
  2. SteVio1989 16 times, last 2 months ago
  3. Hronom 2 times, last 2 months ago
  4. raytest 1 times, last 2 months ago
  5. fervidnerd 2 times, last 4 months ago
9 more registered users
63 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.lang.IllegalStateException

    Spark program execution failed.

    at com.google.common.base.Preconditions.checkState()
  2. Guava
    Preconditions.checkState
    1. com.google.common.base.Preconditions.checkState(Preconditions.java:149)
    1 frame
  3. co.cask.cdap
    SparkRuntimeService.run
    1. co.cask.cdap.internal.app.runtime.spark.SparkRuntimeService.run(SparkRuntimeService.java:193)
    1 frame
  4. Guava
    AbstractExecutionThreadService$1$1.run
    1. com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
    1 frame
  5. co.cask.cdap
    SparkRuntimeService$1$1.run
    1. co.cask.cdap.internal.app.runtime.spark.SparkRuntimeService$1$1.run(SparkRuntimeService.java:247)
    1 frame
  6. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:744)
    1 frame