java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue

DataStax JIRA | Gábor AUTH | 1 year ago
  1. 0

    It is a VPS instance with 1024MB memory, it runs a Jenkins. The Jenkins starts integration tests, the issue appears exactly after the 16th connect-disconnect in same JVM (possible connection leak?), the 16th connect to the cluster always throw the exception above. I've tested the all version of the Datastax driver (successful test cycles / test cycles): 2.1.3: ok (4/4) 2.1.4: ok (4/4) 2.1.5: ok (4/4) 2.1.6: failed (0/4) 2.1.7: failed (0/4) 2.1.8: failed (0/4) 2.2.0-rc1: failed (0/4) 2.2.0-rc2: failed (0/4) 2.2.0-rc3: failed (0/4) 3.0.0-alpha5: failed (0/4) The stack trace in the getErrors: {code} com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Error writing at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:554) at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:538) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:621) at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) at com.datastax.driver.core.Connection$Flusher.run(Connection.java:875) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at java.lang.Thread.run(Thread.java:745) Caused by: io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:622) at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:939) at io.netty.channel.AbstractChannel.write(AbstractChannel.java:234) ... 5 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:645) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:229) at io.netty.buffer.PoolArena.allocate(PoolArena.java:205) at io.netty.buffer.PoolArena.allocate(PoolArena.java:133) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:83) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:288) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:257) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ... 15 more {code} I've tried to set log level, and found a possible root cause of the issue, it is thrown after every disconnect from Cassandra cluster: {code} 2015-11-22 21:08:59,954 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] Caused by: java.lang.ClassNotFoundException: io.netty.util.Recycler$WeakOrderQueue from [Module "deployment.gacivs-backend-dao-services-0.0.27-SNAPSHOT.war:main" from Service Module Loader] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] ... 15 more 2015-11-22 21:08:59,961 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] {code} P.S.: the "Netty upgrade to 4.x" (https://datastax-oss.atlassian.net/browse/JAVA-622) issue resolved in the 2.1.6 version and the OutOfMemoryError: Direct buffer memory" appeared in the 2.1.6 version...

    DataStax JIRA | 1 year ago | Gábor AUTH
    java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue
  2. 0

    netty-socketio under Tomcat running,press ctrl C stop Tomcat, Console print error

    GitHub | 2 years ago | 617726909
    java.lang.NoClassDefFoundError: io/n etty/util/concurrent/DefaultPromise$2
  3. 0

    GitHub comment 228#207362973

    GitHub | 8 months ago | thheller
    java.lang.NoClassDefFoundError: io/netty/util/concurrent/DefaultPromise$1
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    java.lang.NoClassDefFoundError: io/netty/util/concurrent/DefaultPromise$2 Error

    Stack Overflow | 1 month ago | Thala
    java.lang.NoClassDefFoundError: io/netty/util/concurrent/DefaultPromise$2
  6. 0

    JMS Receiver Application Down (Wildfly)

    Stack Overflow | 2 days ago | Karim Ahmed
    java.lang.NoClassDefFoundError: io/netty/buffer/PoolArena$1

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.NoClassDefFoundError

      io/netty/util/Recycler $WeakOrderQueue

      at io.netty.util.Recycler$DefaultHandle.recycle()
    2. Netty
      DefaultThreadFactory$DefaultRunnableDecorator.run
      1. io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150)[netty-common-4.0.33.Final.jar:4.0.33.Final]
      2. io.netty.util.Recycler.recycle(Recycler.java:111)[netty-common-4.0.33.Final.jar:4.0.33.Final]
      3. io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      4. io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      5. io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      6. io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      7. io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      8. io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      9. io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      10. io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      11. io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
      12. io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195)[netty-common-4.0.33.Final.jar:4.0.33.Final]
      13. io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130)[netty-common-4.0.33.Final.jar:4.0.33.Final]
      14. io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[netty-common-4.0.33.Final.jar:4.0.33.Final]
      14 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:744)[rt.jar:1.8.0]
      1 frame