org.apache.zookeeper.KeeperException$NodeExistsException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • (the trace below occurs in the admin console but the Admin continues to handle subsequent deployment requests fine it seems) To reproduce: * start xd-admin and xd-container (just 1 of each) * deploy 'time | log' * kill both xd-admin and xd-container * start xd-container by itself * wait 10 seconds or so, then start xd-admin result: {code} 21:35:01,575 ERROR ContainersPathChildrenCache-0 cache.PathChildrenCache:550 - org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/deployments/modules/f89e5fdc-7dc1-49fb-93cc-d536ab853f12/s.source.time-0 at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:411) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:319) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:255) at org.springframework.xd.dirt.server.ContainerListener.onChildAdded(ContainerListener.java:272) at org.springframework.xd.dirt.server.ContainerListener.childEvent(ContainerListener.java:152) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:494) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:488) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:485) at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) at org.apache.curator.framework.recipes.cache.PathChildrenCache$11.run(PathChildrenCache.java:755) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {code}
    via by Mark Fisher,
  • (the trace below occurs in the admin console but the Admin continues to handle subsequent deployment requests fine it seems) To reproduce: * start xd-admin and xd-container (just 1 of each) * deploy 'time | log' * kill both xd-admin and xd-container * start xd-container by itself * wait 10 seconds or so, then start xd-admin result: {code} 21:35:01,575 ERROR ContainersPathChildrenCache-0 cache.PathChildrenCache:550 - org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/deployments/modules/f89e5fdc-7dc1-49fb-93cc-d536ab853f12/s.source.time-0 at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:411) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:319) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:255) at org.springframework.xd.dirt.server.ContainerListener.onChildAdded(ContainerListener.java:272) at org.springframework.xd.dirt.server.ContainerListener.childEvent(ContainerListener.java:152) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:494) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:488) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:485) at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) at org.apache.curator.framework.recipes.cache.PathChildrenCache$11.run(PathChildrenCache.java:755) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {code}
    via by Mark Fisher,
  • GitHub comment 11#162772842
    via GitHub by stantonk
    ,
  • coordinator node exception
    via by xiaoxiaoluo,
  • As reported by Matt Stine: After closing and reopening a laptop, the following stack trace appears in the container log: {noformat} 00:47:28,226 INFO main-EventThread state.ConnectionStateManager:194 - State change: RECONNECTED 00:47:28,226 INFO ConnectionStateManager-0 zookeeper.ZooKeeperConnection:255 - >>> Curator connected event: RECONNECTED 00:47:28,322 ERROR ConnectionStateManager-0 listen.ListenerContainer:96 - Listener (org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener@6abf4158) threw an exception java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:301) at org.springframework.xd.dirt.server.ContainerRegistrar.access$100(ContainerRegistrar.java:93) at org.springframework.xd.dirt.server.ContainerRegistrar$ContainerAttributesRegisteringZooKeeperConnectionListener.onConnect(ContainerRegistrar.java:316) at org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener.stateChanged(ZooKeeperConnection.java:257) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:222) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:218) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:215) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:75) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:42) at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:295) ... 15 more Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:69) ... 17 more {noformat} This can occur if ZK does not remove the ephemeral node before the container creates a new one. This can be fixed in the following ways: * Remove the existing ephemeral node if it already exists * Register containers with a new UUID upon every new connection For now I'll implement the first solution.
    via by Patrick Peralta,
  • When the container which has modules deployed disconnects/reconnects to the cluster while the admin leader isn't available, following exception is thrown: This is more likely to happen in single-node scenario as there is no admin leader re-election there. In distributed mode, we can always setup HA on admins so that the leadership re-election happens. 20:03:16,307 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache - org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/deployments/modules/allocated/53f41042-8abd-443b-abfb-ba42a24fb9fb/foo.sink.log.1/metadata at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.springframework.xd.dirt.server.ContainerRegistrar.writeModuleMetadata(ContainerRegistrar.java:486) at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:461) at org.springframework.xd.dirt.server.ContainerRegistrar.access$8(ContainerRegistrar.java:426) at org.springframework.xd.dirt.server.ContainerRegistrar$DeploymentListener.childEvent(ContainerRegistrar.java:807) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
    via by Ilayaperumal Gopinathan,
  • As reported by Matt Stine: After closing and reopening a laptop, the following stack trace appears in the container log: {noformat} 00:47:28,226 INFO main-EventThread state.ConnectionStateManager:194 - State change: RECONNECTED 00:47:28,226 INFO ConnectionStateManager-0 zookeeper.ZooKeeperConnection:255 - >>> Curator connected event: RECONNECTED 00:47:28,322 ERROR ConnectionStateManager-0 listen.ListenerContainer:96 - Listener (org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener@6abf4158) threw an exception java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:301) at org.springframework.xd.dirt.server.ContainerRegistrar.access$100(ContainerRegistrar.java:93) at org.springframework.xd.dirt.server.ContainerRegistrar$ContainerAttributesRegisteringZooKeeperConnectionListener.onConnect(ContainerRegistrar.java:316) at org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener.stateChanged(ZooKeeperConnection.java:257) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:222) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:218) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:215) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:75) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:42) at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:295) ... 15 more Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:69) ... 17 more {noformat} This can occur if ZK does not remove the ephemeral node before the container creates a new one. This can be fixed in the following ways: * Remove the existing ephemeral node if it already exists * Register containers with a new UUID upon every new connection For now I'll implement the first solution.
    via by Patrick Peralta,
  • When the container which has modules deployed disconnects/reconnects to the cluster while the admin leader isn't available, following exception is thrown: This is more likely to happen in single-node scenario as there is no admin leader re-election there. In distributed mode, we can always setup HA on admins so that the leadership re-election happens. 20:03:16,307 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache - org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/deployments/modules/allocated/53f41042-8abd-443b-abfb-ba42a24fb9fb/foo.sink.log.1/metadata at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.springframework.xd.dirt.server.ContainerRegistrar.writeModuleMetadata(ContainerRegistrar.java:486) at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:461) at org.springframework.xd.dirt.server.ContainerRegistrar.access$8(ContainerRegistrar.java:426) at org.springframework.xd.dirt.server.ContainerRegistrar$DeploymentListener.childEvent(ContainerRegistrar.java:807) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
    via by Ilayaperumal Gopinathan,
    • org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/deployments/modules/f89e5fdc-7dc1-49fb-93cc-d536ab853f12/s.source.time-0 at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:411) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:319) at org.apache.curator.framework.imps.CreateBuilderImpl$4.forPath(CreateBuilderImpl.java:255) at org.springframework.xd.dirt.server.ContainerListener.onChildAdded(ContainerListener.java:272) at org.springframework.xd.dirt.server.ContainerListener.childEvent(ContainerListener.java:152) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:494) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:488) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:485) at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) at org.apache.curator.framework.recipes.cache.PathChildrenCache$11.run(PathChildrenCache.java:755) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

    Users with the same issue

    nasimknasimk
    2 times, last one,
    Unknown visitor
    Unknown visitor1 times, last one,