java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue

DataStax JIRA | Gábor AUTH | 1 year ago
  1. 0

    It is a VPS instance with 1024MB memory, it runs a Jenkins. The Jenkins starts integration tests, the issue appears exactly after the 16th connect-disconnect in same JVM (possible connection leak?), the 16th connect to the cluster always throw the exception above. I've tested the all version of the Datastax driver (successful test cycles / test cycles): 2.1.3: ok (4/4) 2.1.4: ok (4/4) 2.1.5: ok (4/4) 2.1.6: failed (0/4) 2.1.7: failed (0/4) 2.1.8: failed (0/4) 2.2.0-rc1: failed (0/4) 2.2.0-rc2: failed (0/4) 2.2.0-rc3: failed (0/4) 3.0.0-alpha5: failed (0/4) The stack trace in the getErrors: {code} com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Error writing at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:554) at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:538) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:621) at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) at com.datastax.driver.core.Connection$Flusher.run(Connection.java:875) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at java.lang.Thread.run(Thread.java:745) Caused by: io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:622) at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:939) at io.netty.channel.AbstractChannel.write(AbstractChannel.java:234) ... 5 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:645) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:229) at io.netty.buffer.PoolArena.allocate(PoolArena.java:205) at io.netty.buffer.PoolArena.allocate(PoolArena.java:133) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:83) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:288) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:257) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ... 15 more {code} I've tried to set log level, and found a possible root cause of the issue, it is thrown after every disconnect from Cassandra cluster: {code} 2015-11-22 21:08:59,954 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] Caused by: java.lang.ClassNotFoundException: io.netty.util.Recycler$WeakOrderQueue from [Module "deployment.gacivs-backend-dao-services-0.0.27-SNAPSHOT.war:main" from Service Module Loader] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] ... 15 more 2015-11-22 21:08:59,961 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] {code} P.S.: the "Netty upgrade to 4.x" (https://datastax-oss.atlassian.net/browse/JAVA-622) issue resolved in the 2.1.6 version and the OutOfMemoryError: Direct buffer memory" appeared in the 2.1.6 version...

    DataStax JIRA | 1 year ago | Gábor AUTH
    java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue
  2. 0

    JMS Receiver Application Down (Wildfly)

    Stack Overflow | 1 week ago | Karim Ahmed
    java.lang.NoClassDefFoundError: io/netty/buffer/PoolArena$1
  3. 0

    JBoss JPA trying to load non-entity class when setting up persistence context

    Stack Overflow | 2 years ago | fansonly
    java.lang.NoClassDefFoundError: javax/ws/rs/container/ContainerResponseFilter
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Wildfly: ClassNotFoundException: org.ajax4jsf.Filter

    Stack Overflow | 2 years ago | Socrates
    abc.UndertowDeploymentInfoService: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./abc.UndertowDeploymentInfoService: java.lang.ClassNotFoundException: org.ajax4jsf.Filter from [Module "deployment.abc.war:main" from Service Module Loader]
  6. 0

    Deployment fails with wildfly gradle and hibernate

    Stack Overflow | 2 years ago | mostart
    java.lang.ClassNotFoundException: org.hibernate.Session from [Module "deployment.myshop-1.0.war:main" from Service Module Loader]

  1. Andreas Häber 9 times, last 2 weeks ago
  2. Reich 8 times, last 3 weeks ago
  3. andyglick 1 times, last 2 months ago
  4. Cafecanudo 3 times, last 2 months ago
  5. pbabcsany 2 times, last 2 months ago
3 more registered users
75 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.lang.ClassNotFoundException

    io.netty.util.Recycler$WeakOrderQueue from [Module "deployment.gacivs-backend-dao-services-0.0.27-SNAPSHOT.war:main" from Service Module Loader]

    at org.jboss.modules.ModuleClassLoader.findClass()
  2. JBoss Modules
    ConcurrentClassLoader.loadClass
    1. org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213)[jboss-modules-1.3.3.Final.jar:1.3.3.Final]
    2. org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)[jboss-modules-1.3.3.Final.jar:1.3.3.Final]
    3. org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)[jboss-modules-1.3.3.Final.jar:1.3.3.Final]
    4. org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)[jboss-modules-1.3.3.Final.jar:1.3.3.Final]
    5. org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)[jboss-modules-1.3.3.Final.jar:1.3.3.Final]
    5 frames
  3. Netty
    DefaultThreadFactory$DefaultRunnableDecorator.run
    1. io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150)[netty-common-4.0.33.Final.jar:4.0.33.Final]
    2. io.netty.util.Recycler.recycle(Recycler.java:111)[netty-common-4.0.33.Final.jar:4.0.33.Final]
    3. io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    4. io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    5. io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    6. io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    7. io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    8. io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    9. io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    10. io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    11. io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64)[netty-buffer-4.0.33.Final.jar:4.0.33.Final]
    12. io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195)[netty-common-4.0.33.Final.jar:4.0.33.Final]
    13. io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130)[netty-common-4.0.33.Final.jar:4.0.33.Final]
    14. io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[netty-common-4.0.33.Final.jar:4.0.33.Final]
    14 frames
  4. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:744)[rt.jar:1.8.0]
    1 frame