org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly

Spring JIRA | Chad Wilson | 3 years ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    We recently migrated to Spring Batch 3.x a system that has been running fine in production for a number of years on older Spring Batch 2.x releases. While this has been running fine in production for a couple of months now, yesterday a batch failed with what appears to be some kind of race condition / multithreading bug. Re-running the batch completely (which would operate on exactly the same data in a re-run) worked fine so this is not easily reproducible. The error was {noformat} 08:25:21,432 [main] AbstractJob ERROR <execute> - Encountered fatal error executing job org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:140) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:362) at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:590) at com.ml.elt.automarking.util.AutomarkBatchCommandLineRunner.main(AutomarkBatchCommandLineRunner.java:14) Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=ECIJob at state=ECIJob.loadXMLMaster with exception at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:171) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:141) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134) ... 7 more Caused by: java.lang.NullPointerException at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.decrement(SynchronizationManagerSupport.java:149) at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.close(SynchronizationManagerSupport.java:143) at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.release(SynchronizationManagerSupport.java:193) at org.springframework.batch.core.scope.context.StepSynchronizationManager.release(StepSynchronizationManager.java:112) at org.springframework.batch.core.step.AbstractStep.doExecutionRelease(AbstractStep.java:284) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:274) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:162) ... 9 more {noformat} In the step that failed, the batch has been split into partitions with gridSize=2 running across 5 threads. it looks like the failure happened at the end of the step. Having a look inside the code the NPE line is below. Could this be due to inconsistent synchronization or possibly an assumption that the count hasn't already been removed that is not true? Without being highly familiar with the code it looks a bit odd to me to synchronize on one field (counts) during increment, but another (contexts) during decrement. {code:java} private void decrement() { E current = getCurrent().pop(); if (current != null) { int remaining = counts.get(current).decrementAndGet(); // <--- PROBLEMATIC LINE if (remaining <= 0) { synchronized (contexts) { contexts.remove(current); counts.remove(current); } } } } public void increment() { E current = getCurrent().peek(); if (current != null) { AtomicInteger count; synchronized (counts) { count = counts.get(current); if (count == null) { count = new AtomicInteger(); counts.put(current, count); } } count.incrementAndGet(); } } {code}

    Spring JIRA | 3 years ago | Chad Wilson
    org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
  2. 0

    We recently migrated to Spring Batch 3.x a system that has been running fine in production for a number of years on older Spring Batch 2.x releases. While this has been running fine in production for a couple of months now, yesterday a batch failed with what appears to be some kind of race condition / multithreading bug. Re-running the batch completely (which would operate on exactly the same data in a re-run) worked fine so this is not easily reproducible. The error was {noformat} 08:25:21,432 [main] AbstractJob ERROR <execute> - Encountered fatal error executing job org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:140) at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:362) at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:590) at com.ml.elt.automarking.util.AutomarkBatchCommandLineRunner.main(AutomarkBatchCommandLineRunner.java:14) Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=ECIJob at state=ECIJob.loadXMLMaster with exception at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:171) at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:141) at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134) ... 7 more Caused by: java.lang.NullPointerException at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.decrement(SynchronizationManagerSupport.java:149) at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.close(SynchronizationManagerSupport.java:143) at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.release(SynchronizationManagerSupport.java:193) at org.springframework.batch.core.scope.context.StepSynchronizationManager.release(StepSynchronizationManager.java:112) at org.springframework.batch.core.step.AbstractStep.doExecutionRelease(AbstractStep.java:284) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:274) at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:162) ... 9 more {noformat} In the step that failed, the batch has been split into partitions with gridSize=2 running across 5 threads. it looks like the failure happened at the end of the step. Having a look inside the code the NPE line is below. Could this be due to inconsistent synchronization or possibly an assumption that the count hasn't already been removed that is not true? Without being highly familiar with the code it looks a bit odd to me to synchronize on one field (counts) during increment, but another (contexts) during decrement. {code:java} private void decrement() { E current = getCurrent().pop(); if (current != null) { int remaining = counts.get(current).decrementAndGet(); // <--- PROBLEMATIC LINE if (remaining <= 0) { synchronized (contexts) { contexts.remove(current); counts.remove(current); } } } } public void increment() { E current = getCurrent().peek(); if (current != null) { AtomicInteger count; synchronized (counts) { count = counts.get(current); if (count == null) { count = new AtomicInteger(); counts.put(current, count); } } count.incrementAndGet(); } } {code}

    Spring JIRA | 3 years ago | Chad Wilson
    org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
  3. 0

    Observed in the field for a 1.1.0 deployment, but I suspect it affects 1.0.0 as well... Include an empty {{.csv}} file in a batch and the entire job will fail after completing the 'raw' validation step. E.g. if {{external_course_tag.csv}} is completely empty: {noformat} 11:38:30.499 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvWriter - File created for writing file name: /Users/dmccallum/dev1/ssp-data-importer/src/ssp-data-importer/ssp-data-importer-assembly/target/ssp-data-importer/bin/../importjob/upsert/external_course_tag.csv 11:38:30.499 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvWriter - Start Raw Write Step for file file:/Users/dmccallum/dev1/ssp-data-importer/src/ssp-data-importer/ssp-data-importer-assembly/target/ssp-data-importer/bin/../importjob/process/external_course_tag.csv 11:38:30.500 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvReader - Start Raw Read Step for external_course_tag.csv 11:38:30.504 [taskExecutor-1] ERROR o.s.batch.core.step.AbstractStep - Encountered an error executing the step org.springframework.batch.item.ItemStreamException: Failed to initialize the reader at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:142) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE] at com.sun.proxy.$Proxy4.open(Unknown Source) ~[na:na] at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.step.item.ChunkMonitor.open(ChunkMonitor.java:115) ~[spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:306) ~[spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:192) ~[spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:139) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:136) [spring-batch-core-2.2.2.RELEASE.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.NullPointerException: null at org.jasig.ssp.util.importer.job.csv.RawItemCsvReader.getTokenizer(RawItemCsvReader.java:79) ~[ssp-data-importer-impl-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at org.jasig.ssp.util.importer.job.csv.RawItemCsvReader.handleLine(RawItemCsvReader.java:74) ~[ssp-data-importer-impl-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:270) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na] at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:139) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na] ... 23 common frames omitted 11:38:30.505 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvReader - End Raw Read Step for external_course_tag.csv lines read: 0 lines skipped: 0 11:38:30.505 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvWriter - End Raw Write Step for file:/Users/dmccallum/dev1/ssp-data-importer/src/ssp-data-importer/ssp-data-importer-assembly/target/ssp-data-importer/bin/../importjob/process/external_course_tag.csv lines read: 0 lines skipped: 0 11:38:30.506 [taskExecutor-1] INFO o.j.s.u.i.job.csv.RawItemCsvWriter - StepExecution: id=110, version=1, name=filterRawItemsSlave:partition1, status=FAILED, exitStatus=FAILED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0 {noformat} Then the end of the job has this: {noformat} 11:38:30.547 [main] ERROR o.s.batch.core.step.AbstractStep - Encountered an error executing the step org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:111) ~[spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195) ~[spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:137) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:152) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:131) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:301) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:134) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) [spring-core-3.2.2.RELEASE.jar:3.2.2.RELEASE] at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:127) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:351) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:577) [spring-batch-core-2.2.2.RELEASE.jar:na] at org.jasig.ssp.util.importer.job.Main.main(Main.java:29) [ssp-data-importer-impl-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] {noformat} And the report indicates that the 'upsert' step basically never happened, but no errors are listed: {noformat} 11:38:30.571 [main] INFO o.j.s.u.i.job.report.ReportGenerator - Start Time: 10-06-2014 11:38:30 End Time: 10-06-2014 11:38:30 Duration: 0 seconds (0:00:00.280 ) Job Id: 18 Job Paramters: {run.id=17, random=983432} Job Status: FAILED Job Details: Table Name: external_catalog_year Parsed: 2 Skipped on Parse: 0 Upserted: null Skipped on Write: null Table Name: external_term Parsed: 2 Skipped on Parse: 0 Upserted: null Skipped on Write: null Table Name: external_course_tag Parsed: 0 Skipped on Parse: 0 Upserted: null Skipped on Write: null Errors: No Errors Found. {noformat}

    Apereo Issues | 3 years ago | Dan McCallum
    org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [ExitStatus] works only with COMPLETED WITH SKIPS - Spring Forum

    spring.io | 1 year ago
    org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
  6. 0

    Memory leak because of HibernateSessionImpl in Spring batch

    Stack Overflow | 2 years ago | Chakrapani Kulkarni
    org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step

    4 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.NullPointerException

      No message provided

      at org.springframework.batch.core.scope.context.SynchronizationManagerSupport.decrement()
    2. Spring Batch Core
      SimpleJobLauncher$1.run
      1. org.springframework.batch.core.scope.context.SynchronizationManagerSupport.decrement(SynchronizationManagerSupport.java:149)
      2. org.springframework.batch.core.scope.context.SynchronizationManagerSupport.close(SynchronizationManagerSupport.java:143)
      3. org.springframework.batch.core.scope.context.SynchronizationManagerSupport.release(SynchronizationManagerSupport.java:193)
      4. org.springframework.batch.core.scope.context.StepSynchronizationManager.release(StepSynchronizationManager.java:112)
      5. org.springframework.batch.core.step.AbstractStep.doExecutionRelease(AbstractStep.java:284)
      6. org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:274)
      7. org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
      8. org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
      9. org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
      10. org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:162)
      11. org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:141)
      12. org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
      13. org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
      14. org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
      14 frames
    3. Spring Core
      SyncTaskExecutor.execute
      1. org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
      1 frame
    4. Spring Batch Core
      CommandLineJobRunner.main
      1. org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
      2. org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:362)
      3. org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:590)
      3 frames
    5. com.ml.elt
      AutomarkBatchCommandLineRunner.main
      1. com.ml.elt.automarking.util.AutomarkBatchCommandLineRunner.main(AutomarkBatchCommandLineRunner.java:14)
      1 frame