com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!

Atlassian JIRA | Michael Heemskerk | 11 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    h2. Symptoms * 1 node is shutdown in a Data Center installation * partition migration on shutdown fails: {noformat} 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Failed migration from Member [<ip2>]:5701 for com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Add ress[<ip1>]:5701, valid=true, processing=false} com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster! at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66] at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1826) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1795) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1931) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1907) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1884) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] ... 1 frame trimmed 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Migration failed: com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Address[<ip1>]:5701, valid=true , processing=false} {noformat} <23 hours pass> * Error 500 while error the projects page * Errors in the logs with the following stacktrace: {noformat} ... Caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.serial.PROJECT at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:278) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:107) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorateUrl(VersionAwareAvatarUrlDecorator.java:62) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorate(VersionAwareAvatarUrlDecorator.java:38) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:429) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:418) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.project.InternalNormalProject.accept(InternalNormalProject.java:37) ~[bitbucket-model-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:219) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.bitbucket.util.RequestLocalCache.get(RequestLocalCache.java:62) ~[bitbucket-util-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService.getUrlForProject(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) ~[atlassian-plugins-core-4.1.1.jar:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.getUrl(AvatarUrlFunction.java:78) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:67) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:23) ~[classes/:na] at com.atlassian.soy.impl.modules.SoyJavaFunctionAdapter.computeForJava(SoyJavaFunctionAdapter.java:34) ~[na:na] at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38) ~[na:na] at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670) ~[na:na] ... 132 common frames omitted {noformat}

    Atlassian JIRA | 11 months ago | Michael Heemskerk
    com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!
  2. 0

    h2. Symptoms * 1 node is shutdown in a Data Center installation * partition migration on shutdown fails: {noformat} 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Failed migration from Member [<ip2>]:5701 for com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Add ress[<ip1>]:5701, valid=true, processing=false} com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster! at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66] at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1826) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1795) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1931) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1907) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1884) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] ... 1 frame trimmed 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Migration failed: com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Address[<ip1>]:5701, valid=true , processing=false} {noformat} <23 hours pass> * Error 500 while error the projects page * Errors in the logs with the following stacktrace: {noformat} ... Caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.serial.PROJECT at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:278) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:107) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorateUrl(VersionAwareAvatarUrlDecorator.java:62) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorate(VersionAwareAvatarUrlDecorator.java:38) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:429) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:418) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.project.InternalNormalProject.accept(InternalNormalProject.java:37) ~[bitbucket-model-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:219) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.bitbucket.util.RequestLocalCache.get(RequestLocalCache.java:62) ~[bitbucket-util-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService.getUrlForProject(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) ~[atlassian-plugins-core-4.1.1.jar:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.getUrl(AvatarUrlFunction.java:78) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:67) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:23) ~[classes/:na] at com.atlassian.soy.impl.modules.SoyJavaFunctionAdapter.computeForJava(SoyJavaFunctionAdapter.java:34) ~[na:na] at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38) ~[na:na] at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670) ~[na:na] ... 132 common frames omitted {noformat}

    Atlassian JIRA | 11 months ago | Michael Heemskerk
    com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!
  3. 0

    Many problems with Cluster / Hazelcast plugin | Ignite Realtime

    igniterealtime.org | 9 months ago
    com.hazelcast.core.MemberLeftException: Member [172.25.3.62]:5701 has left cluster!
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.hazelcast.core.MemberLeftException

      Member [<ip2>]:5701 has left cluster!

      at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run()
    2. com.hazelcast.spi
      InvocationRegistry$OnMemberLeftTask.run
      1. com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398)[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
      1 frame
    3. Hazelcast
      CachedExecutorServiceDelegate$Worker.run
      1. com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209)[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
      1 frame
    4. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[na:1.8.0_66]
      2. java.lang.Thread.run(Thread.java:745)[na:1.8.0_66]
      2 frames
    5. Hazelcast
      HazelcastManagedThread.run
      1. com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
      2. com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
      2 frames