java.lang.OutOfMemoryError: Direct buffer memory

kafka-users | Asaf Mesika | 8 months ago
  1. 0

    Re: New consumer: OutOfMemoryError: Direct buffer memory

    kafka-users | 8 months ago | Asaf Mesika
    java.lang.OutOfMemoryError: Direct buffer memory
  2. 0

    I'm running Kafka's new consumer with message handlers that can sometimes take a lot of time to return, and combining that with manual offset management (to get at-least-once semantics). Since poll() is the only way to heartbeat with the consumer, I have a thread that runs every 500 milliseconds that does the following: 1) Pause all partitions 2) Call poll(0) 3) Resume all partitions For the record, all accesses to KafkaConsumer are protected by synchronized blocks. This generally works, but I'm occasionally seeing messages like this: {code} java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) at sun.nio.ch.IOUtil.read(IOUtil.java:195) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108) at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) at org.apache.kafka.common.network.Selector.poll(Selector.java:286) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) {code} In addition, when I'm reporting offsets, I'm seeing: {code} java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) at sun.nio.ch.IOUtil.read(IOUtil.java:195) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108) at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) at org.apache.kafka.common.network.Selector.poll(Selector.java:286) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:358) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:968) {code} Given that I'm just calling the library, this behavior is unexpected.

    Apache's JIRA Issue Tracker | 8 months ago | Kanak Biscuitwala
    java.lang.OutOfMemoryError: Direct buffer memory
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Crash after 4 days of working, repeatly

    GitHub | 3 years ago | jpicerno
    java.lang.OutOfMemoryError: Direct buffer memory

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      Direct buffer memory

      at java.nio.Bits.reserveMemory()
    2. Java RT
      SocketChannelImpl.read
      1. java.nio.Bits.reserveMemory(Bits.java:658)
      2. java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
      3. java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
      4. sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
      5. sun.nio.ch.IOUtil.read(IOUtil.java:195)
      6. sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
      6 frames
    3. Apache Kafka
      Selector.poll
      1. org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
      2. org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
      3. org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
      4. org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
      5. org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
      6. org.apache.kafka.common.network.Selector.poll(Selector.java:286)
      6 frames