com.hazelcast.core.MemberLeftException: Member [172.22.23.139]:5701 has left cluster!

Google Groups | Jeff Long | 3 months ago
  1. 0

    EventBus Messages Getting Lost on AWS Cluster

    Google Groups | 3 months ago | Jeff Long
    com.hazelcast.core.MemberLeftException: Member [172.22.23.139]:5701 has left cluster!
  2. 0

    MemberLeftException on getQueue

    Google Groups | 2 months ago | Abdülhakim Ünlü
    com.hazelcast.core.MemberLeftException: Member [10.133.203.31]:5701 has left cluster!
  3. 0

    Many problems with Cluster / Hazelcast plugin | Ignite Realtime

    igniterealtime.org | 6 months ago
    com.hazelcast.core.MemberLeftException: Member [172.25.3.62]:5701 has left cluster!
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    h2. Symptoms * 1 node is shutdown in a Data Center installation * partition migration on shutdown fails: {noformat} 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Failed migration from Member [<ip2>]:5701 for com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Add ress[<ip1>]:5701, valid=true, processing=false} com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster! at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66] at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1826) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1795) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1931) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1907) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1884) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] ... 1 frame trimmed 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Migration failed: com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Address[<ip1>]:5701, valid=true , processing=false} {noformat} <23 hours pass> * Error 500 while error the projects page * Errors in the logs with the following stacktrace: {noformat} ... Caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.serial.PROJECT at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:278) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:107) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorateUrl(VersionAwareAvatarUrlDecorator.java:62) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorate(VersionAwareAvatarUrlDecorator.java:38) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:429) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:418) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.project.InternalNormalProject.accept(InternalNormalProject.java:37) ~[bitbucket-model-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:219) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.bitbucket.util.RequestLocalCache.get(RequestLocalCache.java:62) ~[bitbucket-util-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService.getUrlForProject(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) ~[atlassian-plugins-core-4.1.1.jar:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.getUrl(AvatarUrlFunction.java:78) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:67) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:23) ~[classes/:na] at com.atlassian.soy.impl.modules.SoyJavaFunctionAdapter.computeForJava(SoyJavaFunctionAdapter.java:34) ~[na:na] at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38) ~[na:na] at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670) ~[na:na] ... 132 common frames omitted {noformat}

    Atlassian JIRA | 8 months ago | Michael Heemskerk
    com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!
  6. 0

    h2. Symptoms * 1 node is shutdown in a Data Center installation * partition migration on shutdown fails: {noformat} 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Failed migration from Member [<ip2>]:5701 for com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Add ress[<ip1>]:5701, valid=true, processing=false} com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster! at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry$OnMemberLeftTask.run(InvocationRegistry.java:398) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66] at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1826) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1795) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1931) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1907) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1884) [hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24] ... 1 frame trimmed 2016-04-14 18:20:33,324 WARN [hz.hazelcast.migration] c.h.p.InternalPartitionService [<ip1>]:5701 [admin] [3.5.2-atlassian-24] Migration failed: com.hazelcast.partition.MigrationInfo{partitionId=125, source=Address[<ip2>]:5701, destination=Address[<ip1>]:5701, master=Address[<ip1>]:5701, valid=true , processing=false} {noformat} <23 hours pass> * Error 500 while error the projects page * Errors in the logs with the following stacktrace: {noformat} ... Caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.serial.PROJECT at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:278) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:107) ~[atlassian-cache-hazelcast-2.11.0.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorateUrl(VersionAwareAvatarUrlDecorator.java:62) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.VersionAwareAvatarUrlDecorator.decorate(VersionAwareAvatarUrlDecorator.java:38) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:429) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$AvatarProjectVisitor.visit(ConfigurableAvatarService.java:418) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.project.InternalNormalProject.accept(InternalNormalProject.java:37) ~[bitbucket-model-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:219) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService$2.get(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.bitbucket.util.RequestLocalCache.get(RequestLocalCache.java:62) ~[bitbucket-util-4.5.1.jar:na] at com.atlassian.stash.internal.avatar.ConfigurableAvatarService.getUrlForProject(ConfigurableAvatarService.java:216) ~[bitbucket-service-impl-4.5.1.jar:na] at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) ~[atlassian-plugins-core-4.1.1.jar:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.getUrl(AvatarUrlFunction.java:78) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:67) ~[classes/:na] at com.atlassian.stash.internal.web.soy.functions.AvatarUrlFunction.apply(AvatarUrlFunction.java:23) ~[classes/:na] at com.atlassian.soy.impl.modules.SoyJavaFunctionAdapter.computeForJava(SoyJavaFunctionAdapter.java:34) ~[na:na] at com.atlassian.soy.impl.modules.CompositeFunctionAdaptor.computeForJava(CompositeFunctionAdaptor.java:38) ~[na:na] at com.google.template.soy.sharedpasses.render.EvalVisitor.computeFunctionHelper(EvalVisitor.java:670) ~[na:na] ... 132 common frames omitted {noformat}

    Atlassian JIRA | 8 months ago | Michael Heemskerk
    com.hazelcast.core.MemberLeftException: Member [<ip2>]:5701 has left cluster!

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.hazelcast.core.MemberLeftException

      Member [172.22.23.139]:5701 has left cluster!

      at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$OnMemberLeftTask.run()
    2. com.hazelcast.spi
      InvocationMonitor$OnMemberLeftTask.run
      1. com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$OnMemberLeftTask.run(InvocationMonitor.java:284)
      1 frame
    3. Hazelcast
      CachedExecutorServiceDelegate$Worker.run
      1. com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212)
      1 frame
    4. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      3. java.lang.Thread.run(Thread.java:745)
      3 frames
    5. Hazelcast
      HazelcastManagedThread.run
      1. com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
      2. com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
      2 frames