java.util.ConcurrentModificationException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • In my cluster I have set elasticity mode as Auto and then I set manual elasticity mode with "targetComputeNodeNum" as max ( 4 ) Then onwards I can not set manual elasticity mode. Steps followed : 1. serengeti>cluster setParam --name hadoop1 --elasticityMode auto 2. serengeti>cluster setParam --name hadoop1 --elasticityMode manual --targetComputeNodeNum 3 From no onwards everytime setting up manual elasticity mode fails with following message {noformat} cluster hadoop1 setParam failed: task execution failed: No error message from VHM. you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan* {noformat} and I get following exception in serengeti.log. {noformat} 2013 May 18 05:11:45,339+0000 ERROR Thread-92| com.vmware.bdd.utils.RabbitMQConsumer: stop receiving messages without normal termination 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: cleanup execution: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: execution succeed: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,345+0000 ERROR pool-3-thread-2| com.vmware.bdd.command.MessageTask: No error message from VHM. 2013 May 18 05:11:45,347+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,348+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: setManualElasticityStep 2013 May 18 05:11:45,351+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 May 18 05:11:45,354+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 30 2013 May 18 05:11:45,355+0000 INFO SimpleAsyncTaskExecutor-30| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=setManualElasticityJob]] completed with the following parameters: [{activeComputeNodeNumber=4, clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=["compute"], hadoopJobTrackerIP=10.140.109.215, timeStamp=1368853602938}] and the following status: [FAILED] {noformat} The exception found in vhm.log {noformat} 2013 May 18 05:34:53.125 [Stats_Thread_352-AbstractStatsProducer] StatsProducer com.vmware.vhadoop.vhm.stats.VCStatsProducer threw uncaught and unexpected exception java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1171) at com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115) at com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) {noformat}
    via by Rajit Saha,
  • In my cluster I have set elasticity mode as Auto and then I set manual elasticity mode with "targetComputeNodeNum" as max ( 4 ) Then onwards I can not set manual elasticity mode. Steps followed : 1. serengeti>cluster setParam --name hadoop1 --elasticityMode auto 2. serengeti>cluster setParam --name hadoop1 --elasticityMode manual --targetComputeNodeNum 3 From no onwards everytime setting up manual elasticity mode fails with following message {noformat} cluster hadoop1 setParam failed: task execution failed: No error message from VHM. you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan* {noformat} and I get following exception in serengeti.log. {noformat} 2013 May 18 05:11:45,339+0000 ERROR Thread-92| com.vmware.bdd.utils.RabbitMQConsumer: stop receiving messages without normal termination 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: cleanup execution: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: execution succeed: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,345+0000 ERROR pool-3-thread-2| com.vmware.bdd.command.MessageTask: No error message from VHM. 2013 May 18 05:11:45,347+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,348+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: setManualElasticityStep 2013 May 18 05:11:45,351+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 May 18 05:11:45,354+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 30 2013 May 18 05:11:45,355+0000 INFO SimpleAsyncTaskExecutor-30| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=setManualElasticityJob]] completed with the following parameters: [{activeComputeNodeNumber=4, clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=["compute"], hadoopJobTrackerIP=10.140.109.215, timeStamp=1368853602938}] and the following status: [FAILED] {noformat} The exception found in vhm.log {noformat} 2013 May 18 05:34:53.125 [Stats_Thread_352-AbstractStatsProducer] StatsProducer com.vmware.vhadoop.vhm.stats.VCStatsProducer threw uncaught and unexpected exception java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1171) at com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115) at com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) {noformat}
    via by Rajit Saha,
  • Concurrent deploy failures?
    via GitHub by realityforge
    ,
  • Hadoop 2.4.1 Release Notes
    via by Unknown author,
    • java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1171) at com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115) at com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,