java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52]

JBoss Issue Tracker | Ladislav Thon | 1 year ago
  1. 0

    In one of our tests, we've seen this exception during server shutdown: {code} 2015-06-05 10:34:44,387 ERROR [org.xnio.listener] (XNIO-1 I/O-2) XNIO001007: A channel event listener threw an exception: java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at org.xnio.XnioWorker.execute(XnioWorker.java:741) at org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54) ... {code} This is a very basic test that only starts the server and then shuts it down using {{:shutdown}}. After looking into this for a while, I believe that this is caused by the core management server ({{UndertowHttpManagementService}}) shutting down an XNIO worker (= thread pool) while the {{:shutdown}} management operation is still running (or, in fact, finishing, trying to close the network connection). I have a Byteman-based reproducer that inserts artifical pauses to certain well-defined places. I'm not sure if this has some connection to the graceful shutdown system, but I believe that even if it does, something like this shouldn't happen. Steps to reproduce: # {{./bin/standalone.sh -c standalone-full-ha.xml}} and wait until it starts completely # {{jps -v | grep "\-D\\[Standalone\\]"}} to figure out the PID of the newly started server # {{bminstall.sh -b -Dorg.jboss.byteman.transform.all $PID}} # {{bmsubmit.sh reproducer.btm}}, where {{reproducer.btm}} is a Byteman script reproduced below # {{./bin/jboss-cli.sh -c}} # {{:read-resource}} repeat few times # {{:shutdown(timeout=1)}} (or plain {{:shutdown}}) The Byteman script: {code} RULE XnioWorker.TaskPool/ThreadPoolExecutor shutdown CLASS java.util.concurrent.ThreadPoolExecutor METHOD shutdown() AFTER INVOKE advanceRunState IF TRUE DO Thread.sleep(10000) ENDRULE RULE Remoting onClose handler CLASS org.jboss.remoting3.remote.RemoteReadListener$1 METHOD handleEvent(java.nio.channels.Channel) AT ENTRY IF TRUE DO Thread.sleep(5000) ENDRULE {code}

    JBoss Issue Tracker | 1 year ago | Ladislav Thon
    java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52]
  2. 0

    GitHub comment 7985#137879121

    GitHub | 1 year ago | wildfly-ci
    java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@891d00 rejected from org.xnio.XnioWorker$TaskPool@6bc63d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11]
  3. 0

    JsseSslStreamSourceConduit returns -1 before underlying socket returns -1 Resulting in {code} java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@13e2ee8 rejected from org.xnio.XnioWorker$TaskPool@1f9d2ff[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 482] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at org.xnio.XnioWorker.execute(XnioWorker.java:584) at org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72) at org.xnio.channels.TranslatingSuspendableChannel.close(TranslatingSuspendableChannel.java:813) at org.xnio.IoUtils.safeClose(IoUtils.java:137) at org.xnio.channels.TranslatingSuspendableChannel$3.handleEvent(TranslatingSuspendableChannel.java:123) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72) at org.xnio.channels.TranslatingSuspendableChannel.close(TranslatingSuspendableChannel.java:813) at org.xnio.IoUtils.safeClose(IoUtils.java:137) at org.xnio.channels.TranslatingSuspendableChannel$3.handleEvent(TranslatingSuspendableChannel.java:123) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72) at org.xnio.nio.AbstractNioStreamChannel.invokeCloseHandler(AbstractNioStreamChannel.java:384) at org.xnio.nio.NioTcpChannel.close(NioTcpChannel.java:163) at org.xnio.IoUtils.safeClose(IoUtils.java:137) at org.xnio.nio.WorkerThread.run(WorkerThread.java:154) {code}

    JBoss Issue Tracker | 1 year ago | Brad Maxwell
    java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@13e2ee8 rejected from org.xnio.XnioWorker$TaskPool@1f9d2ff[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 482]
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Exception {code} ERROR [org.xnio.listener] (XNIO-2 I/O-1) XNIO001007: A channel event listener threw an exception: java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@2d696007 rejected from org.xnio.XnioWorker$TaskPool@74a153ee[Shutting down, pool size = 9, active threads = 0, queued tasks = 0, completed tasks = 50] {code}, details at [1] could be hit when stopping server. This happens intermediately. I run transaction integration tests on WildFly/EAP7 DR builds. My tests works in way of doing some settings via CLI, deploy application, do the test, undeploy application, stop the server. During the stopping the server this exception is shown in log. Adding server log from my test for possible more details. [1] {code} INFO [org.jboss.as.server] (Thread-3) WFLYSRV0220: Server shutdown has been requested. INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = module_ojdbc7.jar INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS] INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-1) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory INFO [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA] INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 77) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0019: Stopped Driver service with driver-name = h2 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0019: Host default-host stopping ERROR [org.xnio.listener] (XNIO-2 I/O-1) XNIO001007: A channel event listener threw an exception: java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@2d696007 rejected from org.xnio.XnioWorker$TaskPool@74a153ee[Shutting down, pool size = 9, active threads = 0, queued tasks = 0, completed tasks = 50] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) at org.xnio.XnioWorker.execute(XnioWorker.java:741) at org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.channels.TranslatingSuspendableChannel.close(TranslatingSuspendableChannel.java:906) at org.xnio.IoUtils.safeClose(IoUtils.java:134) at org.xnio.channels.TranslatingSuspendableChannel$3.handleEvent(TranslatingSuspendableChannel.java:133) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:80) at org.xnio.Connection.writeClosed(Connection.java:117) at org.xnio.nio.AbstractNioStreamConnection.writeClosed(AbstractNioStreamConnection.java:47) at org.xnio.nio.NioSocketConduit.terminateWrites(NioSocketConduit.java:181) at org.xnio.nio.NioSocketConduit.truncateWrites(NioSocketConduit.java:191) at org.xnio.conduits.ConduitStreamSinkChannel.close(ConduitStreamSinkChannel.java:186) at org.xnio.IoUtils.safeClose(IoUtils.java:134) at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.forceTermination(WriteReadyHandler.java:57) at org.xnio.nio.NioSocketConduit.forceTermination(NioSocketConduit.java:107) at org.xnio.nio.WorkerThread.run(WorkerThread.java:490) INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 71) AMQ151003: resource adaptor stopped INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 71) AMQ221002: Apache ActiveMQ Artemis Message Broker version 1.0.0 [8e39e2e6-1fc3-11e5-b5ab-adb038d9447d] stopped INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTP listener default suspending INFO [com.arjuna.ats.jbossatx] (MSC service thread 1-3) ARJUNA032018: Destroying TransactionManagerService INFO [com.arjuna.ats.jbossatx] (MSC service thread 1-3) ARJUNA032014: Stopping transaction recovery manager DEBUG [com.arjuna.ats.arjuna] (Listener:4712) Recovery listener existing com.arjuna.ats.internal.arjuna.recovery.WorkerService DEBUG [com.arjuna.ats.arjuna] (MSC service thread 1-3) PeriodicRecovery: Mode <== TERMINATED DEBUG [com.arjuna.ats.arjuna] (MSC service thread 1-3) PeriodicRecovery: shutdown scan wait complete DEBUG [com.arjuna.ats.arjuna] (Periodic Recovery) PeriodicRecovery: background thread exiting INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTP listener default stopped, was bound to localhost/127.0.0.1:8080 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0004: Undertow 1.3.0.Beta2 stopping INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Alpha5) stopped in 237ms {code}

    JBoss Issue Tracker | 1 year ago | Ondřej Chaloupka
    java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@2d696007 rejected from org.xnio.XnioWorker$TaskPool@74a153ee[Shutting down, pool size = 9, active threads = 0, queued tasks = 0, completed tasks = 50]
  6. 0

    [GSS](6.4.z) RejectedExecutionException when closing connection on channel close

    https://bugzilla.redhat.com/bugzilla/ | 1 year ago | Brad Maxwell
    java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@13e2ee8 rejected from org.xnio.XnioWorker$TaskPool@1f9d2ff[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 482]

  1. tyson925 16 times, last 1 week ago
  2. Nikolay Rybak 1 times, last 1 month ago
  3. rlovtangen 109 times, last 2 months ago
  4. rp 2 times, last 2 months ago
  5. rp 1 times, last 5 months ago
4 more registered users
32 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.util.concurrent.RejectedExecutionException

    Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52]

    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution()
  2. Java RT
    ThreadPoolExecutor.execute
    1. java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
    2. java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
    3. java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
    3 frames
  3. XNIO API
    XnioWorker.execute
    1. org.xnio.XnioWorker.execute(XnioWorker.java:741)
    1 frame
  4. JBoss Remoting
    RemoteReadListener$1.handleEvent
    1. org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54)
    1 frame