java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor did not accept within 500 MILLISECONDS

MuleSoft JIRA | Jeroen Verellen | 5 years ago
  1. 0

    With the included test cases you can reproduce the issue for Services and Flows. At startup all is fine. New messages are sent using mule client. Once the seda queue is full and all threads are still busy you can face this issue. When the threadWaitTimeout expires the message is refused. On the exception handling that follows you get the below exception which crashes the service or flow. Under test/resources there are 2 thread dumps, one before and one after the crash. In the second dump you can see that the dispatcher thread "sedaFlowCrash.stage1.01" has effectively died. #### - ERROR 2011-12-21 17:32:34,366 [sedaFlowCrash.stage1.01] org.mule.exception.DefaultMessagingExceptionStrategy: Caught exception in Exception Strategy: ThreadPoolExecutor did not accept within 500 MILLISECONDS java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor did not accept within 500 MILLISECONDS at org.mule.util.concurrent.WaitPolicy.rejectedExecution(WaitPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.mule.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:41) at org.mule.work.MuleWorkManager.executeWork(MuleWorkManager.java:250) at org.mule.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:218) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:233) at org.mule.work.WorkerContext.run(WorkerContext.java:310) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) #### - WARN 2011-12-21 17:32:34,378 [sedaFlowCrash.stage1.01] org.mule.DefaultMuleMessage: Message access violation java.lang.IllegalStateException: Only owner thread can write to message: Thread[connector.VM.mule.default.receiver.03,5,main]/Thread[sedaFlowCrash.stage1.01,5,main] at org.mule.DefaultMuleMessage.newException(DefaultMuleMessage.java:1574) at org.mule.DefaultMuleMessage.checkMutable(DefaultMuleMessage.java:1560) at org.mule.DefaultMuleMessage.assertAccess(DefaultMuleMessage.java:1489) at org.mule.DefaultMuleMessage.setExceptionPayload(DefaultMuleMessage.java:993) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:60) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:66) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:239) at org.mule.work.WorkerContext.run(WorkerContext.java:310) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) #### - ERROR 2011-12-21 17:32:34,383 [sedaFlowCrash.stage1.01] org.mule.processor.AsyncWorkListener: Work caused exception on 'workCompleted'. Work being executed was: org.mule.processor.SedaStageInterceptingMessageProcessor Exception in thread "sedaFlowCrash.stage1.01" org.mule.api.MuleRuntimeException: An exception occurred while invoking message processor "DefaultMessageProcessorChain '(inner iterating chain) of 'sedaFlowCrash' processor chain' [ DefaultJavaComponent{sedaFlowCrash.commponent.1821889520}, DefaultOutboundEndpoint{endpointUri=vm://flow.out, connector=VMConnector { name=connector.VM.mule.default lifecycle=start this=63fb050c numberOfConcurrentTransactedReceivers=4 createMultipleTransactedReceivers=true connected=true supportedProtocols=[vm] serviceOverrides=<none> } , name='endpoint.vm.flow.out', mep=ONE_WAY, properties={}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}, org.mule.routing.requestreply.AsyncReplyToPropertyRequestReplyReplier ]" for asynchronously. at org.mule.processor.AsyncWorkListener.handleWorkException(AsyncWorkListener.java:74) at org.mule.processor.AsyncWorkListener.workCompleted(AsyncWorkListener.java:50) at org.mule.work.WorkerContext.run(WorkerContext.java:369) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalStateException: Only owner thread can write to message: Thread[connector.VM.mule.default.receiver.03,5,main]/Thread[sedaFlowCrash.stage1.01,5,main] at org.mule.DefaultMuleMessage.newException(DefaultMuleMessage.java:1574) at org.mule.DefaultMuleMessage.checkMutable(DefaultMuleMessage.java:1560) at org.mule.DefaultMuleMessage.assertAccess(DefaultMuleMessage.java:1489) at org.mule.DefaultMuleMessage.setExceptionPayload(DefaultMuleMessage.java:993) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:60) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:66) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:239) at org.mule.work.WorkerContext.run(WorkerContext.java:310) ... 3 more

    MuleSoft JIRA | 5 years ago | Jeroen Verellen
    java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor did not accept within 500 MILLISECONDS
  2. 0

    With the included test cases you can reproduce the issue for Services and Flows. At startup all is fine. New messages are sent using mule client. Once the seda queue is full and all threads are still busy you can face this issue. When the threadWaitTimeout expires the message is refused. On the exception handling that follows you get the below exception which crashes the service or flow. Under test/resources there are 2 thread dumps, one before and one after the crash. In the second dump you can see that the dispatcher thread "sedaFlowCrash.stage1.01" has effectively died. #### - ERROR 2011-12-21 17:32:34,366 [sedaFlowCrash.stage1.01] org.mule.exception.DefaultMessagingExceptionStrategy: Caught exception in Exception Strategy: ThreadPoolExecutor did not accept within 500 MILLISECONDS java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor did not accept within 500 MILLISECONDS at org.mule.util.concurrent.WaitPolicy.rejectedExecution(WaitPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.mule.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:41) at org.mule.work.MuleWorkManager.executeWork(MuleWorkManager.java:250) at org.mule.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:218) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:233) at org.mule.work.WorkerContext.run(WorkerContext.java:310) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) #### - WARN 2011-12-21 17:32:34,378 [sedaFlowCrash.stage1.01] org.mule.DefaultMuleMessage: Message access violation java.lang.IllegalStateException: Only owner thread can write to message: Thread[connector.VM.mule.default.receiver.03,5,main]/Thread[sedaFlowCrash.stage1.01,5,main] at org.mule.DefaultMuleMessage.newException(DefaultMuleMessage.java:1574) at org.mule.DefaultMuleMessage.checkMutable(DefaultMuleMessage.java:1560) at org.mule.DefaultMuleMessage.assertAccess(DefaultMuleMessage.java:1489) at org.mule.DefaultMuleMessage.setExceptionPayload(DefaultMuleMessage.java:993) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:60) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:66) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:239) at org.mule.work.WorkerContext.run(WorkerContext.java:310) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) #### - ERROR 2011-12-21 17:32:34,383 [sedaFlowCrash.stage1.01] org.mule.processor.AsyncWorkListener: Work caused exception on 'workCompleted'. Work being executed was: org.mule.processor.SedaStageInterceptingMessageProcessor Exception in thread "sedaFlowCrash.stage1.01" org.mule.api.MuleRuntimeException: An exception occurred while invoking message processor "DefaultMessageProcessorChain '(inner iterating chain) of 'sedaFlowCrash' processor chain' [ DefaultJavaComponent{sedaFlowCrash.commponent.1821889520}, DefaultOutboundEndpoint{endpointUri=vm://flow.out, connector=VMConnector { name=connector.VM.mule.default lifecycle=start this=63fb050c numberOfConcurrentTransactedReceivers=4 createMultipleTransactedReceivers=true connected=true supportedProtocols=[vm] serviceOverrides=<none> } , name='endpoint.vm.flow.out', mep=ONE_WAY, properties={}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}, org.mule.routing.requestreply.AsyncReplyToPropertyRequestReplyReplier ]" for asynchronously. at org.mule.processor.AsyncWorkListener.handleWorkException(AsyncWorkListener.java:74) at org.mule.processor.AsyncWorkListener.workCompleted(AsyncWorkListener.java:50) at org.mule.work.WorkerContext.run(WorkerContext.java:369) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalStateException: Only owner thread can write to message: Thread[connector.VM.mule.default.receiver.03,5,main]/Thread[sedaFlowCrash.stage1.01,5,main] at org.mule.DefaultMuleMessage.newException(DefaultMuleMessage.java:1574) at org.mule.DefaultMuleMessage.checkMutable(DefaultMuleMessage.java:1560) at org.mule.DefaultMuleMessage.assertAccess(DefaultMuleMessage.java:1489) at org.mule.DefaultMuleMessage.setExceptionPayload(DefaultMuleMessage.java:993) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:60) at org.mule.exception.AbstractMessagingExceptionStrategy.handleException(AbstractMessagingExceptionStrategy.java:66) at org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:239) at org.mule.work.WorkerContext.run(WorkerContext.java:310) ... 3 more

    MuleSoft JIRA | 5 years ago | Jeroen Verellen
    java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor did not accept within 500 MILLISECONDS
  3. 0

    [Cassandra-commits] [jira] [Commented] (CASSANDRA-6565) New node refuses to join the ring. - Grokbase

    grokbase.com | 3 months ago
    java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassadra 1.1.1 crashes while inserting heavy data using Hector 1.0.5

    Stack Overflow | 4 years ago | Kayalvizhi NoorulAmeen
    java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
  6. 0

    0.7.9 RejectedExecutionException

    Google Groups | 5 years ago | Ashley Martens
    java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.RejectedExecutionException

      ThreadPoolExecutor did not accept within 500 MILLISECONDS

      at org.mule.util.concurrent.WaitPolicy.rejectedExecution()
    2. Mule Core
      WaitPolicy.rejectedExecution
      1. org.mule.util.concurrent.WaitPolicy.rejectedExecution(WaitPolicy.java:62)
      1 frame
    3. Java RT
      ThreadPoolExecutor.execute
      1. java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      2. java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      2 frames
    4. Mule Core
      WorkerContext.run
      1. org.mule.work.ScheduleWorkExecutor.doExecute(ScheduleWorkExecutor.java:41)
      2. org.mule.work.MuleWorkManager.executeWork(MuleWorkManager.java:250)
      3. org.mule.work.MuleWorkManager.scheduleWork(MuleWorkManager.java:218)
      4. org.mule.processor.SedaStageInterceptingMessageProcessor.run(SedaStageInterceptingMessageProcessor.java:233)
      5. org.mule.work.WorkerContext.run(WorkerContext.java:310)
      5 frames
    5. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      3. java.lang.Thread.run(Thread.java:662)
      3 frames