java.util.concurrent.RejectedExecutionException: Endpoint is not open

JBoss Issue Tracker | Ladislav Thon | 2 years ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    In one of our tests, we've seen this exception during server shutdown: {code} 2015-06-05 10:34:44,387 ERROR [org.xnio.listener] (XNIO-1 I/O-2) XNIO001007: A channel event listener threw an exception: java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at org.xnio.XnioWorker.execute(XnioWorker.java:741) at org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54) ... {code} This is a very basic test that only starts the server and then shuts it down using {{:shutdown}}. After looking into this for a while, I believe that this is caused by the core management server ({{UndertowHttpManagementService}}) shutting down an XNIO worker (= thread pool) while the {{:shutdown}} management operation is still running (or, in fact, finishing, trying to close the network connection). I have a Byteman-based reproducer that inserts artifical pauses to certain well-defined places. I'm not sure if this has some connection to the graceful shutdown system, but I believe that even if it does, something like this shouldn't happen. Steps to reproduce: # {{./bin/standalone.sh -c standalone-full-ha.xml}} and wait until it starts completely # {{jps -v | grep "\-D\\[Standalone\\]"}} to figure out the PID of the newly started server # {{bminstall.sh -b -Dorg.jboss.byteman.transform.all $PID}} # {{bmsubmit.sh reproducer.btm}}, where {{reproducer.btm}} is a Byteman script reproduced below # {{./bin/jboss-cli.sh -c}} # {{:read-resource}} repeat few times # {{:shutdown(timeout=1)}} (or plain {{:shutdown}}) The Byteman script: {code} RULE XnioWorker.TaskPool/ThreadPoolExecutor shutdown CLASS java.util.concurrent.ThreadPoolExecutor METHOD shutdown() AFTER INVOKE advanceRunState IF TRUE DO Thread.sleep(10000) ENDRULE RULE Remoting onClose handler CLASS org.jboss.remoting3.remote.RemoteReadListener$1 METHOD handleEvent(java.nio.channels.Channel) AT ENTRY IF TRUE DO Thread.sleep(5000) ENDRULE {code}

    JBoss Issue Tracker | 2 years ago | Ladislav Thon
    java.util.concurrent.RejectedExecutionException: Endpoint is not open

    Root Cause Analysis

    1. java.util.concurrent.RejectedExecutionException

      Endpoint is not open

      at org.jboss.remoting3.EndpointImpl.executorUntick()
    2. JBoss Remoting
      RemoteReadListener$1.handleEvent
      1. org.jboss.remoting3.EndpointImpl.executorUntick(EndpointImpl.java:200)
      2. org.jboss.remoting3.EndpointImpl$TrackingExecutor.execute(EndpointImpl.java:700)
      3. org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54)
      3 frames
    3. XNIO API
      Connection.writeClosed
      1. org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
      2. org.xnio.channels.TranslatingSuspendableChannel.close(TranslatingSuspendableChannel.java:906)
      3. org.xnio.IoUtils.safeClose(IoUtils.java:134)
      4. org.xnio.channels.TranslatingSuspendableChannel$3.handleEvent(TranslatingSuspendableChannel.java:133)
      5. org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
      6. org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
      7. org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
      8. org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:80)
      9. org.xnio.Connection.writeClosed(Connection.java:117)
      9 frames
    4. XNIO NIO Implementation
      NioSocketConduit.truncateWrites
      1. org.xnio.nio.AbstractNioStreamConnection.writeClosed(AbstractNioStreamConnection.java:47)
      2. org.xnio.nio.NioSocketConduit.terminateWrites(NioSocketConduit.java:181)
      3. org.xnio.nio.NioSocketConduit.truncateWrites(NioSocketConduit.java:191)
      3 frames
    5. org.xnio.conduits
      ConduitStreamSinkChannel.close
      1. org.xnio.conduits.ConduitStreamSinkChannel.close(ConduitStreamSinkChannel.java:186)
      1 frame
    6. XNIO API
      IoUtils.safeClose
      1. org.xnio.IoUtils.safeClose(IoUtils.java:134)
      1 frame
    7. org.xnio.conduits
      WriteReadyHandler$ChannelListenerHandler.forceTermination
      1. org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.forceTermination(WriteReadyHandler.java:57)
      1 frame
    8. XNIO NIO Implementation
      WorkerThread.run
      1. org.xnio.nio.NioSocketConduit.forceTermination(NioSocketConduit.java:107)
      2. org.xnio.nio.WorkerThread.run(WorkerThread.java:490)
      2 frames