com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster test100

SpringSource Issue Tracker | Wenwu Peng | 4 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Try to deploy 100 Nodes, it is failed at 95% serengit log: 2013 Jun 04 10:11:33,452+0000 ERROR SimpleAsyncTaskExecutor-3| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: operation : CREATE failed on cluster: test100 com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster test100 at com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_UNKNOWN_ERROR(SoftwareManagementException.java:55) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:83) at com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask.call(IronfanSoftwareManagementTask.java:78) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:83) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor2148.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy153.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) Caused by: org.apache.thrift.transport.TTransportException Terry and Jared think maybe it is Jetty Issue of a request contains headers that exceed the buffer size (4096), the request fails with the following exception: WARN log handle failed java.io.IOException: FULL at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:368) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) http://jira.codehaus.org/browse/JETTY-336

    SpringSource Issue Tracker | 4 years ago | Wenwu Peng
    com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster test100
  2. 0

    Try to deploy 100 Nodes, it is failed at 95% serengit log: 2013 Jun 04 10:11:33,452+0000 ERROR SimpleAsyncTaskExecutor-3| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: operation : CREATE failed on cluster: test100 com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster test100 at com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_UNKNOWN_ERROR(SoftwareManagementException.java:55) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:83) at com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask.call(IronfanSoftwareManagementTask.java:78) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:83) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor2148.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy153.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) Caused by: org.apache.thrift.transport.TTransportException Terry and Jared think maybe it is Jetty Issue of a request contains headers that exceed the buffer size (4096), the request fails with the following exception: WARN log handle failed java.io.IOException: FULL at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:368) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) http://jira.codehaus.org/browse/JETTY-336

    SpringSource Issue Tracker | 4 years ago | Wenwu Peng
    com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster test100
  3. 0

    Scenario: ========= I had a data/compute separate cluster with 5 Compute nodes. I set elasticity as Manual with --targetComputeNodeNum 2. Then I tried to scale out the cluster to 6 compute nodes with cluster resize command. The command fails with exceptions in Serengeti.log. The problems are 1. Command Fails {noformat} FAILED 92% node group: master, instance number: 1 roles:[hadoop_namenode, hadoop_jobtracker] NAME IP STATUS TASK ------------------------------------------------ hadoop1-master-0 xx.yyy.zzz.150 VM Ready node group: data, instance number: 1 roles:[hadoop_datanode] NAME IP STATUS TASK ---------------------------------------------- hadoop1-data-0 xx.yyy.zzz.228 VM Ready node group: compute, instance number: 6 roles:[hadoop_tasktracker] NAME IP STATUS TASK ------------------------------------------------------------------ hadoop1-compute-5 xx.yyy.zzz.135 VM Ready Formatting data disks hadoop1-compute-4 xx.yyy.zzz.112 VM Ready hadoop1-compute-3 xx.yyy.zzz.145 VM Ready hadoop1-compute-2 xx.yyy.zzz.195 VM Ready Bootstrapping VM hadoop1-compute-1 xx.yyy.zzz.239 VM Ready Bootstrapping VM hadoop1-compute-0 xx.yyy.zzz.221 VM Ready Bootstrapping VM node group: client, instance number: 1 roles:[hadoop_client, pig, hive, hive_server] NAME IP STATUS TASK ------------------------------------------------ hadoop1-client-0 xx.yyy.zzz.116 VM Ready cluster hadoop1 resize failed: task execution failed: you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan*,/opt/serengeti/logs/task/17/1 {noformat} 2. Found some exception in serengeti.log {noformat} 2013 Jun 01 08:12:50,264+0000 INFO ProgressMonitor-hadoop1| com.vmware.bdd.service.job.software.ProgressMonitor: operation has not finished. wait again 2013 Jun 01 08:12:53,443+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient: Failed run cluseter operation for cluster: hadoop1 2013 Jun 01 08:12:53,444+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: operation : CREATE failed on cluster: hadoop1 com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster hadoop1. Error is: Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2) at com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_FAILURE(SoftwareManagementException.java:49) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:80) at com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask.call(IronfanSoftwareManagementTask.java:83) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:83) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) Caused by: ClusterOperationException(message:Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2)) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result$runClusterOperation_resultStandardScheme.read(SoftwareManagement.java:1105) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result$runClusterOperation_resultStandardScheme.read(SoftwareManagement.java:1083) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result.read(SoftwareManagement.java:1027) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$Client.recv_runClusterOperation(SoftwareManagement.java:106) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$Client.runClusterOperation(SoftwareManagement.java:93) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:76) ... 33 more 2013 Jun 01 08:12:53,445+0000 INFO ProgressMonitor-hadoop1| com.vmware.bdd.service.job.software.ProgressMonitor: Monitor thread was stopped ... 2013 Jun 01 08:12:53,716+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.entity.NodeEntity: node hadoop1-master-0 action changed to 2013 Jun 01 08:12:53,716+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: updated progress. finished? false 2013 Jun 01 08:12:53,716+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: command execution failed, error message is 2013 Jun 01 08:12:53,718+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 Jun 01 08:12:53,718+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:89) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 Jun 01 08:12:53,719+0000 ERROR SimpleAsyncTaskExecutor-17| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:89) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 Jun 01 08:12:53,720+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: softwareResizeClusterStep 2013 Jun 01 08:12:53,725+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 Jun 01 08:12:53,729+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 17 2013 Jun 01 08:12:53,731+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.ResizeClusterJobExecutionListener: Set cluster hadoop1 group compute instance number to 5 2013 Jun 01 08:12:53,733+0000 INFO SimpleAsyncTaskExecutor-17| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=resizeClusterJob]] completed with the following parameters: [{clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=compute, newInstanceNumber=6, oldInstanceNumber=5, timeStamp=1370074136179, verifyNodeScope=group}] and the following status: [FAILED] 2013 Jun 01 08:12:55,450+0000 ERROR http-8080-1| com.vmware.bdd.manager.JobManager: mark task as failed: task execution failed: 2013 Jun 01 08:14:19,511+0000 INFO VC_TASK_THREAD| com.vmware.bdd.entity.NodeEntity: node hadoop1-compute-2 status changed to Powered Off 2013 Jun 01 08:14:19,512+0000 INFO VC_TASK_THREAD| com.vmware.bdd.entity.NodeEntity: node hadoop1-compute-1 status changed to Powered Off ... {noformat} 3. Instead of 2 , 3 Compute VMs were switched ON and as well as Jobtrackers shows 3 TTs are online, whereas Manual elasticity modes shows to have 2 TT only should be UP

    SpringSource Issue Tracker | 4 years ago | Rajit Saha
    com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster hadoop1. Error is: Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Scenario: ========= I had a data/compute separate cluster with 5 Compute nodes. I set elasticity as Manual with --targetComputeNodeNum 2. Then I tried to scale out the cluster to 6 compute nodes with cluster resize command. The command fails with exceptions in Serengeti.log. The problems are 1. Command Fails {noformat} FAILED 92% node group: master, instance number: 1 roles:[hadoop_namenode, hadoop_jobtracker] NAME IP STATUS TASK ------------------------------------------------ hadoop1-master-0 xx.yyy.zzz.150 VM Ready node group: data, instance number: 1 roles:[hadoop_datanode] NAME IP STATUS TASK ---------------------------------------------- hadoop1-data-0 xx.yyy.zzz.228 VM Ready node group: compute, instance number: 6 roles:[hadoop_tasktracker] NAME IP STATUS TASK ------------------------------------------------------------------ hadoop1-compute-5 xx.yyy.zzz.135 VM Ready Formatting data disks hadoop1-compute-4 xx.yyy.zzz.112 VM Ready hadoop1-compute-3 xx.yyy.zzz.145 VM Ready hadoop1-compute-2 xx.yyy.zzz.195 VM Ready Bootstrapping VM hadoop1-compute-1 xx.yyy.zzz.239 VM Ready Bootstrapping VM hadoop1-compute-0 xx.yyy.zzz.221 VM Ready Bootstrapping VM node group: client, instance number: 1 roles:[hadoop_client, pig, hive, hive_server] NAME IP STATUS TASK ------------------------------------------------ hadoop1-client-0 xx.yyy.zzz.116 VM Ready cluster hadoop1 resize failed: task execution failed: you can get task failure details from serengeti server log at: /opt/serengeti/logs/serengeti*,/opt/serengeti/logs/ironfan*,/opt/serengeti/logs/task/17/1 {noformat} 2. Found some exception in serengeti.log {noformat} 2013 Jun 01 08:12:50,264+0000 INFO ProgressMonitor-hadoop1| com.vmware.bdd.service.job.software.ProgressMonitor: operation has not finished. wait again 2013 Jun 01 08:12:53,443+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient: Failed run cluseter operation for cluster: hadoop1 2013 Jun 01 08:12:53,444+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: operation : CREATE failed on cluster: hadoop1 com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster hadoop1. Error is: Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2) at com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_FAILURE(SoftwareManagementException.java:49) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:80) at com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask.call(IronfanSoftwareManagementTask.java:83) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:83) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) Caused by: ClusterOperationException(message:Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2)) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result$runClusterOperation_resultStandardScheme.read(SoftwareManagement.java:1105) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result$runClusterOperation_resultStandardScheme.read(SoftwareManagement.java:1083) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$runClusterOperation_result.read(SoftwareManagement.java:1027) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$Client.recv_runClusterOperation(SoftwareManagement.java:106) at com.vmware.bdd.software.mgmt.thrift.SoftwareManagement$Client.runClusterOperation(SoftwareManagement.java:93) at com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:76) ... 33 more 2013 Jun 01 08:12:53,445+0000 INFO ProgressMonitor-hadoop1| com.vmware.bdd.service.job.software.ProgressMonitor: Monitor thread was stopped ... 2013 Jun 01 08:12:53,716+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.entity.NodeEntity: node hadoop1-master-0 action changed to 2013 Jun 01 08:12:53,716+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: updated progress. finished? false 2013 Jun 01 08:12:53,716+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask: command execution failed, error message is 2013 Jun 01 08:12:53,718+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Aspect for exception handling 2013 Jun 01 08:12:53,718+0000 ERROR SimpleAsyncTaskExecutor-17| com.vmware.bdd.aop.logging.ExceptionHandlerAspect: Service error com.vmware.bdd.exception.TaskException: task execution failed: at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:89) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 Jun 01 08:12:53,719+0000 ERROR SimpleAsyncTaskExecutor-17| org.springframework.batch.core.step.AbstractStep: Encountered an error executing the step com.vmware.bdd.exception.TaskException: task execution failed: at com.vmware.bdd.exception.TaskException.EXECUTION_FAILED(TaskException.java:27) at com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:89) at com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48) at sun.reflect.GeneratedMethodAccessor1870.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy160.execute(Unknown Source) at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264) at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76) at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367) at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214) at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143) at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120) at java.lang.Thread.run(Thread.java:636) 2013 Jun 01 08:12:53,720+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.SimpleStepExecutionListener: step finished: softwareResizeClusterStep 2013 Jun 01 08:12:53,725+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.ClusterJobExecutionListener: set cluster hadoop1 status to RUNNING 2013 Jun 01 08:12:53,729+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.InMemoryJobExecutionStatusHolder: unregistering job execution: 17 2013 Jun 01 08:12:53,731+0000 INFO SimpleAsyncTaskExecutor-17| com.vmware.bdd.service.job.ResizeClusterJobExecutionListener: Set cluster hadoop1 group compute instance number to 5 2013 Jun 01 08:12:53,733+0000 INFO SimpleAsyncTaskExecutor-17| org.springframework.batch.core.launch.support.SimpleJobLauncher: Job: [FlowJob: [name=resizeClusterJob]] completed with the following parameters: [{clusterFailureStatus=RUNNING, clusterName=hadoop1, clusterSuccessStatus=RUNNING, groupName=compute, newInstanceNumber=6, oldInstanceNumber=5, timeStamp=1370074136179, verifyNodeScope=group}] and the following status: [FAILED] 2013 Jun 01 08:12:55,450+0000 ERROR http-8080-1| com.vmware.bdd.manager.JobManager: mark task as failed: task execution failed: 2013 Jun 01 08:14:19,511+0000 INFO VC_TASK_THREAD| com.vmware.bdd.entity.NodeEntity: node hadoop1-compute-2 status changed to Powered Off 2013 Jun 01 08:14:19,512+0000 INFO VC_TASK_THREAD| com.vmware.bdd.entity.NodeEntity: node hadoop1-compute-1 status changed to Powered Off ... {noformat} 3. Instead of 2 , 3 Compute VMs were switched ON and as well as Jobtrackers shows 3 TTs are online, whereas Manual elasticity modes shows to have 2 TT only should be UP

    SpringSource Issue Tracker | 4 years ago | Rajit Saha
    com.vmware.bdd.software.mgmt.exception.SoftwareManagementException: failed run operation CREATE for cluster hadoop1. Error is: Exception was thrown during calling ironfan cluster APIs. Ironfan error message: No route to host - connect(2)

    Root Cause Analysis

    1. com.vmware.bdd.software.mgmt.exception.SoftwareManagementException

      failed run operation CREATE for cluster test100

      at com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_UNKNOWN_ERROR()
    2. com.vmware.bdd
      TrackableTasklet.execute
      1. com.vmware.bdd.software.mgmt.exception.SoftwareManagementException.CLUSTER_OPERATIOIN_UNKNOWN_ERROR(SoftwareManagementException.java:55)
      2. com.vmware.bdd.software.mgmt.impl.SoftwareManagementClient.runClusterOperation(SoftwareManagementClient.java:83)
      3. com.vmware.bdd.service.job.software.ironfan.IronfanSoftwareManagementTask.call(IronfanSoftwareManagementTask.java:78)
      4. com.vmware.bdd.service.job.software.SoftwareManagementStep.executeStep(SoftwareManagementStep.java:83)
      5. com.vmware.bdd.service.job.TrackableTasklet.execute(TrackableTasklet.java:48)
      5 frames
    3. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor2148.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:616)
      3 frames
    4. Spring AOP
      JdkDynamicAopProxy.invoke
      1. org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
      2. org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      3. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      4. org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55)
      5. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
      6. org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      7. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      8. org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      8 frames
    5. Unknown
      $Proxy153.execute
      1. $Proxy153.execute(Unknown Source)
      1 frame
    6. Spring Batch Core
      TaskletStep$ChunkTransactionCallback.doInTransaction
      1. org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
      1 frame
    7. Spring Tx
      TransactionTemplate.execute
      1. org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
      1 frame
    8. Spring Batch Core
      StepContextRepeatCallback.doInIteration
      1. org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
      2. org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
      2 frames
    9. Spring Batch Infrastructure
      RepeatTemplate.iterate
      1. org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
      2. org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
      3. org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
      3 frames
    10. Spring Batch Core
      SimpleJobLauncher$1.run
      1. org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
      2. org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
      3. org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
      4. org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
      5. org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
      6. org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
      7. org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
      8. org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
      9. org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
      10. org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
      10 frames
    11. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:636)
      1 frame