java.lang.OutOfMemoryError: Direct buffer memory

jetbrains.com | 5 months ago
  1. 0

    [Apache Cassandra Error] java.lang.OutOfMemoryError: Direct buffer memory – Upsource

    jetbrains.com | 5 months ago
    java.lang.OutOfMemoryError: Direct buffer memory
  2. 0

    It is a VPS instance with 1024MB memory, it runs a Jenkins. The Jenkins starts integration tests, the issue appears exactly after the 16th connect-disconnect in same JVM (possible connection leak?), the 16th connect to the cluster always throw the exception above. I've tested the all version of the Datastax driver (successful test cycles / test cycles): 2.1.3: ok (4/4) 2.1.4: ok (4/4) 2.1.5: ok (4/4) 2.1.6: failed (0/4) 2.1.7: failed (0/4) 2.1.8: failed (0/4) 2.2.0-rc1: failed (0/4) 2.2.0-rc2: failed (0/4) 2.2.0-rc3: failed (0/4) 3.0.0-alpha5: failed (0/4) The stack trace in the getErrors: {code} com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Error writing at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:554) at com.datastax.driver.core.Connection$10.operationComplete(Connection.java:538) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:621) at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) at com.datastax.driver.core.Connection$Flusher.run(Connection.java:875) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at java.lang.Thread.run(Thread.java:745) Caused by: io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:643) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:700) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:636) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:622) at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:939) at io.netty.channel.AbstractChannel.write(AbstractChannel.java:234) ... 5 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:645) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:229) at io.netty.buffer.PoolArena.allocate(PoolArena.java:205) at io.netty.buffer.PoolArena.allocate(PoolArena.java:133) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:83) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:288) at com.datastax.driver.core.Message$ProtocolEncoder.encode(Message.java:257) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ... 15 more {code} I've tried to set log level, and found a possible root cause of the issue, it is thrown after every disconnect from Cassandra cluster: {code} 2015-11-22 21:08:59,954 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] Caused by: java.lang.ClassNotFoundException: io.netty.util.Recycler$WeakOrderQueue from [Module "deployment.gacivs-backend-dao-services-0.0.27-SNAPSHOT.war:main" from Service Module Loader] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules-1.3.3.Final.jar:1.3.3.Final] ... 15 more 2015-11-22 21:08:59,961 WARN [io.netty.util.ThreadDeathWatcher] (threadDeathWatcher-2-1) Thread death watcher task raised an exception:: java.lang.NoClassDefFoundError: io/netty/util/Recycler $WeakOrderQueue at io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:150) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.Recycler.recycle(Recycler.java:111) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry.recycle(PoolThreadCache.java:459) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.freeEntry(PoolThreadCache.java:442) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:414) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$MemoryRegionCache.free(PoolThreadCache.java:406) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:263) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:254) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.free0(PoolThreadCache.java:235) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache.access$000(PoolThreadCache.java:38) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.buffer.PoolThreadCache$1.run(PoolThreadCache.java:64) [netty-buffer-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.notifyWatchees(ThreadDeathWatcher.java:195) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:130) [netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.33.Final.jar:4.0.33.Final] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0] {code} P.S.: the "Netty upgrade to 4.x" (https://datastax-oss.atlassian.net/browse/JAVA-622) issue resolved in the 2.1.6 version and the OutOfMemoryError: Direct buffer memory" appeared in the 2.1.6 version...

    DataStax JIRA | 1 year ago | Gábor AUTH
    com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Error writing
  3. 0

    OutOfMemory: Direct buffer memory

    GitHub | 3 years ago | defabricated
    io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct buffer memory
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    sshandle still have memory leak, tested with 4.0.24, 4.0.25 and 4.0.17.

    GitHub | 2 years ago | devsunny
    java.lang.OutOfMemoryError: Direct buffer memory

    7 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      Direct buffer memory

      at java.nio.Bits.reserveMemory()
    2. Java RT
      ByteBuffer.allocateDirect
      1. java.nio.Bits.reserveMemory(Bits.java:658)[na:1.8.0_45]
      2. java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[na:1.8.0_45]
      3. java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)[na:1.8.0_45]
      3 frames
    3. org.apache.cassandra
      SSTableReader$4.run
      1. org.apache.cassandra.io.compress.BufferType$2.allocate(BufferType.java:35)[apache-cassandra-2.2.4.jar:2.2.4]
      2. org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:69)[apache-cassandra-2.2.4.jar:2.2.4]
      3. org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:88)[apache-cassandra-2.2.4.jar:2.2.4]
      4. org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:58)[apache-cassandra-2.2.4.jar:2.2.4]
      5. org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createPooledReader(CompressedPoolingSegmentedFile.java:124)[apache-cassandra-2.2.4.jar:2.2.4]
      6. org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:63)[apache-cassandra-2.2.4.jar:2.2.4]
      7. org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1743)[apache-cassandra-2.2.4.jar:2.2.4]
      8. org.apache.cassandra.io.sstable.format.big.SimpleSliceReader.<init>(SimpleSliceReader.java:57)[apache-cassandra-2.2.4.jar:2.2.4]
      9. org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.createReader(SSTableSliceIterator.java:66)[apache-cassandra-2.2.4.jar:2.2.4]
      10. org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.<init>(SSTableSliceIterator.java:43)[apache-cassandra-2.2.4.jar:2.2.4]
      11. org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:75)[apache-cassandra-2.2.4.jar:2.2.4]
      12. org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246)[apache-cassandra-2.2.4.jar:2.2.4]
      13. org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)[apache-cassandra-2.2.4.jar:2.2.4]
      14. org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270)[apache-cassandra-2.2.4.jar:2.2.4]
      15. org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)[apache-cassandra-2.2.4.jar:2.2.4]
      16. org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2009)[apache-cassandra-2.2.4.jar:2.2.4]
      17. org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1813)[apache-cassandra-2.2.4.jar:2.2.4]
      18. org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360)[apache-cassandra-2.2.4.jar:2.2.4]
      19. org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)[apache-cassandra-2.2.4.jar:2.2.4]
      20. org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:288)[apache-cassandra-2.2.4.jar:2.2.4]
      21. org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:304)[apache-cassandra-2.2.4.jar:2.2.4]
      22. org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:76)[apache-cassandra-2.2.4.jar:2.2.4]
      23. org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:310)[apache-cassandra-2.2.4.jar:2.2.4]
      24. org.apache.cassandra.db.SystemKeyspace.getSSTableReadMeter(SystemKeyspace.java:956)[apache-cassandra-2.2.4.jar:2.2.4]
      25. org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.ensureReadMeter(SSTableReader.java:2234)[apache-cassandra-2.2.4.jar:2.2.4]
      26. org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:2045)[apache-cassandra-2.2.4.jar:2.2.4]
      27. org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:1986)[apache-cassandra-2.2.4.jar:2.2.4]
      28. org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:461)[apache-cassandra-2.2.4.jar:2.2.4]
      29. org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:363)[apache-cassandra-2.2.4.jar:2.2.4]
      30. org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:501)[apache-cassandra-2.2.4.jar:2.2.4]
      30 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[na:1.8.0_45]
      2. java.util.concurrent.FutureTask.run(FutureTask.java:266)[na:1.8.0_45]
      3. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[na:1.8.0_45]
      4. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[na:1.8.0_45]
      5. java.lang.Thread.run(Thread.java:745)[na:1.8.0_45]
      5 frames