org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/j/status

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Job `undeploy` operation throws the following stacktrace: ``` http-nio-9393-exec-5 zookeeper.ZooKeeperJobRepository - Exception while transitioning job 'j' state to undeploying org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/j/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:260) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:256) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:252) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:239) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:177) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:199) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:1) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeploy(AbstractInstancePersistingDeployer.java:68) at org.springframework.xd.dirt.rest.XDController.undeploy(XDController.java:125) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ```
    via by Ilayaperumal Gopinathan,
  • Job `undeploy` operation throws the following stacktrace: ``` http-nio-9393-exec-5 zookeeper.ZooKeeperJobRepository - Exception while transitioning job 'j' state to undeploying org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/j/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:260) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:256) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:252) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:239) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:177) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:199) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:1) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeploy(AbstractInstancePersistingDeployer.java:68) at org.springframework.xd.dirt.rest.XDController.undeploy(XDController.java:125) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ```
    via by Ilayaperumal Gopinathan,
  • Sorry to set it "Blocker"; but the problem makes SpringXD unusable. We are getting this weird, NoNode exception on the status ZNode. Example and Log given below. Once this happens, both streams and jobs cannot be deployed. For whatever reason the "status" znode goes missing. The only way for us to get the cluster back to working state, is to clear the zk znode /xd tree and restart spring-xd. At which point, we have to recreate all our streams and jobs back again.. /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status NoNode Exception: 13 Oct 2014 14:16:16,044 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testDestroyStream1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:16,705 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:22,818 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:23,585 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:37,694 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED 13 Oct 2014 14:16:49,950 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:54,490 INFO Deployer server.StreamDeploymentListener - Deployment status for stream 'testCreateStream_SrcHttp_SinkFile1413234903170': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=sink, label='file'}' to container 'd03bccd6-524b-4ff8-84d2-88f3f6daac42' timed out after 30000 ms; Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=source, label='http'}' to container '52abf1c8-ba45-4994-8324-6079b03c670c' timed out after 30000 ms} 13 Oct 2014 14:16:54,493 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:185) at org.springframework.xd.dirt.server.StreamDeploymentListener.onChildAdded(StreamDeploymentListener.java:100) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:179) ... 7 more 13 Oct 2014 14:16:56,251 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_REMOVED 13 Oct 2014 14:17:08,179 INFO Deployer server.JobDeploymentListener - Deployment status for job 'filetsjob-newjob001': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='filetsjob-newjob001', type=job, label='filepolltimeseries'}' to container '244d5076-f69d-42a4-8110-3b046cea2667' timed out after 30000 ms} 13 Oct 2014 14:17:08,181 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:175) at org.springframework.xd.dirt.server.JobDeploymentListener.onChildAdded(JobDeploymentListener.java:99) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:165) ... 7 more 13 Oct 2014 14:17:10,553 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED
    via by Venkatesh Sivasubramanian,
  • SHA: 33de93797106c8dd413dfb08f2fdbbb4931b528c Deployment: 1 Admin, 1 Container DataStore: MySQL A SpringXDException is thrown when running the testJobDeployUndeployFlow test in Acceptance tests back to back. # Ran the test once. Success # Ran the test a second time. ## The following exception is thrown: SpringXDException: Batch Job with the name deployundeployjob already exists ## The XD_JOB_REGISTRY_STEP_NAMES & XD_JOB_REGISTRY still have job_name deployunderployjob still stored. ## The following Exception is seen on the admin server ### 20:45:59,214 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:04,741 INFO Deployer server.JobDeploymentListener - Deployment status for job 'deployundeployjob': DeploymentStatus{state=deployed} 20:46:05,436 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_REMOVED 20:46:12,471 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:16,330 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_REMOVED 20:46:17,819 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:17,832 INFO Deployer server.JobDeploymentListener - Deployment status for job 'deployundeployjob': DeploymentStatus{state=deployed} 20:46:17,834 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:175) at org.springframework.xd.dirt.server.JobDeploymentListener.onChildAdded(JobDeploymentListener.java:99) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:165) ... 7 more # Clear XD_JOB_REGISTRY_STEP_NAMES & XD_JOB_REGISTRY tables and run the test a third time # Ran the test a third time. ## The following exception is thrown: SpringXDException: The job named 'deployundeployjob' is already deployed ## The following Exception was reported by the admin server. ### 21:18:29,355 WARN http-nio-9393-exec-4 zookeeper.ZooKeeperJobRepository - Exception while transitioning job 'deployundeployjob' state to undeploying org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:174) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:196) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:54) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeploy(AbstractInstancePersistingDeployer.java:77) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeployAll(AbstractInstancePersistingDeployer.java:103) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.deleteAll(AbstractInstancePersistingDeployer.java:111) at org.springframework.xd.dirt.rest.XDController.deleteAll(XDController.java:114) at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:885) at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextFilterConfiguration$1.doFilterInternal(EndpointWebMvcAutoConfiguration.java:257) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:85) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) # Ran the test a fourth time. Success
    via by Glenn renfro,
  • Sorry to set it "Blocker"; but the problem makes SpringXD unusable. We are getting this weird, NoNode exception on the status ZNode. Example and Log given below. Once this happens, both streams and jobs cannot be deployed. For whatever reason the "status" znode goes missing. The only way for us to get the cluster back to working state, is to clear the zk znode /xd tree and restart spring-xd. At which point, we have to recreate all our streams and jobs back again.. /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status NoNode Exception: 13 Oct 2014 14:16:16,044 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testDestroyStream1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:16,705 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:22,818 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:23,585 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:37,694 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED 13 Oct 2014 14:16:49,950 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:54,490 INFO Deployer server.StreamDeploymentListener - Deployment status for stream 'testCreateStream_SrcHttp_SinkFile1413234903170': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=sink, label='file'}' to container 'd03bccd6-524b-4ff8-84d2-88f3f6daac42' timed out after 30000 ms; Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=source, label='http'}' to container '52abf1c8-ba45-4994-8324-6079b03c670c' timed out after 30000 ms} 13 Oct 2014 14:16:54,493 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:185) at org.springframework.xd.dirt.server.StreamDeploymentListener.onChildAdded(StreamDeploymentListener.java:100) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:179) ... 7 more 13 Oct 2014 14:16:56,251 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_REMOVED 13 Oct 2014 14:17:08,179 INFO Deployer server.JobDeploymentListener - Deployment status for job 'filetsjob-newjob001': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='filetsjob-newjob001', type=job, label='filepolltimeseries'}' to container '244d5076-f69d-42a4-8110-3b046cea2667' timed out after 30000 ms} 13 Oct 2014 14:17:08,181 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:175) at org.springframework.xd.dirt.server.JobDeploymentListener.onChildAdded(JobDeploymentListener.java:99) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:165) ... 7 more 13 Oct 2014 14:17:10,553 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED
    via by Venkatesh Sivasubramanian,
  • SHA: 33de93797106c8dd413dfb08f2fdbbb4931b528c Deployment: 1 Admin, 1 Container DataStore: MySQL A SpringXDException is thrown when running the testJobDeployUndeployFlow test in Acceptance tests back to back. # Ran the test once. Success # Ran the test a second time. ## The following exception is thrown: SpringXDException: Batch Job with the name deployundeployjob already exists ## The XD_JOB_REGISTRY_STEP_NAMES & XD_JOB_REGISTRY still have job_name deployunderployjob still stored. ## The following Exception is seen on the admin server ### 20:45:59,214 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:04,741 INFO Deployer server.JobDeploymentListener - Deployment status for job 'deployundeployjob': DeploymentStatus{state=deployed} 20:46:05,436 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_REMOVED 20:46:12,471 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:16,330 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_REMOVED 20:46:17,819 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/deployundeployjob, type: CHILD_ADDED 20:46:17,832 INFO Deployer server.JobDeploymentListener - Deployment status for job 'deployundeployjob': DeploymentStatus{state=deployed} 20:46:17,834 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:175) at org.springframework.xd.dirt.server.JobDeploymentListener.onChildAdded(JobDeploymentListener.java:99) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:165) ... 7 more # Clear XD_JOB_REGISTRY_STEP_NAMES & XD_JOB_REGISTRY tables and run the test a third time # Ran the test a third time. ## The following exception is thrown: SpringXDException: The job named 'deployundeployjob' is already deployed ## The following Exception was reported by the admin server. ### 21:18:29,355 WARN http-nio-9393-exec-4 zookeeper.ZooKeeperJobRepository - Exception while transitioning job 'deployundeployjob' state to undeploying org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/deployundeployjob/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:174) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:196) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:54) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeploy(AbstractInstancePersistingDeployer.java:77) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeployAll(AbstractInstancePersistingDeployer.java:103) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.deleteAll(AbstractInstancePersistingDeployer.java:111) at org.springframework.xd.dirt.rest.XDController.deleteAll(XDController.java:114) at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:885) at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextFilterConfiguration$1.doFilterInternal(EndpointWebMvcAutoConfiguration.java:257) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:85) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) # Ran the test a fourth time. Success
    via by Glenn renfro,
  • RS crash upon replication
    via by amit.m...@gmail.com,
  • Region server crashes when using replication
    via by Eran Kutner,
    • org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/j/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:260) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:256) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:252) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:239) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:177) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:199) at org.springframework.xd.dirt.stream.zookeeper.ZooKeeperJobRepository.delete(ZooKeeperJobRepository.java:1) at org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.undeploy(AbstractInstancePersistingDeployer.java:68) at org.springframework.xd.dirt.rest.XDController.undeploy(XDController.java:125) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    Users with the same issue

    nasimk
    nasimk1 times, last one,
    Nikolay Rybak
    Nikolay Rybak1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    RoiOtero
    RoiOtero12 times, last one,
    4 more bugmates