java.util.ConcurrentModificationException

SpringSource Issue Tracker | Rajit Saha | 4 years ago
  1. 0

    In my cluster I have set elasticity mode as Auto and then I set manual elasticity mode with "targetComputeNodeNum" as max ( 4 ) Then onwards I can not set manual elasticity mode. Steps followed : 1. serengeti>cluster setParam --name hadoop1 --elasticityMode auto 2. serengeti>cluster setParam --name hadoop1 --elasticityMode manual --targetComputeNodeNum 3 From no onwards everytime setting up manual elasticity mode fails with following message {noformat} cluster hadoop1 setParam failed: task execution failed: No error message from VHM. you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan* {noformat} and I get following exception in serengeti.log. {noformat} 2013 May 18 05:11:45,339+0000 ERROR Thread-92| com.vmware.bdd.utils.RabbitMQConsumer: stop receiving messages without normal termination 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: cleanup execution: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: execution succeed: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,345+0000 ERROR pool-3-thread-2| com.vmware.bdd.command.MessageTask: No error message from VHM. 2013 May 18 05:11:45,347+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,348+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: setManualElasticityStep 2013 May 18 05:11:45,351+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 May 18 05:11:45,354+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 30 2013 May 18 05:11:45,355+0000 INFO SimpleAsyncTaskExecutor-30| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=setManualElasticityJob]] completed with the following parameters: [{activeComputeNodeNumber=4, clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=["compute"], hadoopJobTrackerIP=10.140.109.215, timeStamp=1368853602938}] and the following status: [FAILED] {noformat} The exception found in vhm.log {noformat} 2013 May 18 05:34:53.125 [Stats_Thread_352-AbstractStatsProducer] StatsProducer com.vmware.vhadoop.vhm.stats.VCStatsProducer threw uncaught and unexpected exception java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1171) at com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115) at com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) {noformat}

    SpringSource Issue Tracker | 4 years ago | Rajit Saha
    java.util.ConcurrentModificationException
  2. 0

    In my cluster I have set elasticity mode as Auto and then I set manual elasticity mode with "targetComputeNodeNum" as max ( 4 ) Then onwards I can not set manual elasticity mode. Steps followed : 1. serengeti>cluster setParam --name hadoop1 --elasticityMode auto 2. serengeti>cluster setParam --name hadoop1 --elasticityMode manual --targetComputeNodeNum 3 From no onwards everytime setting up manual elasticity mode fails with following message {noformat} cluster hadoop1 setParam failed: task execution failed: No error message from VHM. you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan* {noformat} and I get following exception in serengeti.log. {noformat} 2013 May 18 05:11:45,339+0000 ERROR Thread-92| com.vmware.bdd.utils.RabbitMQConsumer: stop receiving messages without normal termination 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: cleanup execution: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,344+0000 INFO Thread-92| com.vmware.bdd.utils.TracedRunnable: execution succeed: com.vmware.bdd.command.MessageProcessor 2013 May 18 05:11:45,345+0000 ERROR pool-3-thread-2| com.vmware.bdd.command.MessageTask: No error message from VHM. 2013 May 18 05:11:45,347+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,347+0000 ERROR SimpleAsyncTaskExecutor-30| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: No error message from VHM. at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.SetManualElasticityStep.executeStep(SetManualElasticityStep.java:76) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1727.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy152.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 May 18 05:11:45,348+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: setManualElasticityStep 2013 May 18 05:11:45,351+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 May 18 05:11:45,354+0000 INFO SimpleAsyncTaskExecutor-30| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 30 2013 May 18 05:11:45,355+0000 INFO SimpleAsyncTaskExecutor-30| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=setManualElasticityJob]] completed with the following parameters: [{activeComputeNodeNumber=4, clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=["compute"], hadoopJobTrackerIP=10.140.109.215, timeStamp=1368853602938}] and the following status: [FAILED] {noformat} The exception found in vhm.log {noformat} 2013 May 18 05:34:53.125 [Stats_Thread_352-AbstractStatsProducer] StatsProducer com.vmware.vhadoop.vhm.stats.VCStatsProducer threw uncaught and unexpected exception java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1171) at com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115) at com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) {noformat}

    SpringSource Issue Tracker | 4 years ago | Rajit Saha
    java.util.ConcurrentModificationException
  3. 0

    ConcurrentModificationException in Metamodel

    GitHub | 3 years ago | lucaswerkmeister
    java.util.ConcurrentModificationException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Combining Multimaps with respect to Functional Dependencies

    Stack Overflow | 2 years ago | cagirici
    java.util.ConcurrentModificationException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.ConcurrentModificationException

      No message provided

      at java.util.TreeMap$PrivateEntryIterator.nextEntry()
    2. Java RT
      TreeMap$KeyIterator.next
      1. java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117)
      2. java.util.TreeMap$KeyIterator.next(TreeMap.java:1171)
      2 frames
    3. com.vmware.vhadoop
      AbstractStatsProducer.run
      1. com.vmware.vhadoop.vhm.stats.StatsCollectorImpl$StatsProducerCallback.addSnapshots(StatsCollectorImpl.java:115)
      2. com.vmware.vhadoop.vhm.stats.AbstractStatsProducer.run(AbstractStatsProducer.java:34)
      2 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      2. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      3. java.util.concurrent.FutureTask.run(FutureTask.java:166)
      4. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      5. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      6. java.lang.Thread.run(Thread.java:636)
      6 frames