org.apache.hadoop.hbase.DroppedSnapshotException: region: IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.

Apache's JIRA Issue Tracker | Enis Soztutar | 4 years ago
  1. 0

    We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some time, this causes OOM for the RSs. Upon further investigation, I've found out that we end up with 200 regions, each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal allocates DirectBuffers, which is unlike HDFS 1 where there is no direct buffer allocation. It seems that there is no guards against the memory used by local buffers in hdfs 2, and having a large number of open files causes multiple GB of memory to be consumed from the RS process. This issue is to further investigate what is going on. Whether we can limit the memory usage in HDFS, or HBase, and/or document the setup. Possible mitigation scenarios are: - Turn off SSR for Hadoop 2 - Ensure that there is enough unallocated memory for the RS based on expected # of store files - Ensure that there is lower number of regions per region server (hence number of open files) Stack trace: {code} org.apache.hadoop.hbase.DroppedSnapshotException: region: IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache( at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache( at org.apache.hadoop.hbase.regionserver.HRegion.flushcache( at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion( at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure( at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500( at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$ at Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory( at java.nio.DirectByteBuffer.<init>( at java.nio.ByteBuffer.allocateDirect( at org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer( at org.apache.hadoop.hdfs.BlockReaderLocal.<init>( at org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader( at org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader( at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader( at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo( at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy( at at at at at at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>( at at org.apache.hadoop.hbase.regionserver.StoreFile.createReader( at org.apache.hadoop.hbase.regionserver.Store.validateStoreFile( at org.apache.hadoop.hbase.regionserver.Store.commitFile( at org.apache.hadoop.hbase.regionserver.Store.access$500( at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit( at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache( {code}

    Apache's JIRA Issue Tracker | 4 years ago | Enis Soztutar
    org.apache.hadoop.hbase.DroppedSnapshotException: region: IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
  2. 0

    OutOfMemoryError: Direct buffer memory (in while(true)-loop)

    GitHub | 3 years ago | Teots
    java.lang.OutOfMemoryError: Direct buffer memory
  3. 0

    java.lang.OutOfMemoryError: Direct buffer memory

    GitHub | 3 years ago | kevinherron
    java.lang.OutOfMemoryError: Direct buffer memory
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    problems when running sbt test (on spark master)

    Google Groups | 4 years ago | Shane Huang
    java.lang.OutOfMemoryError: Direct buffer memory
  6. 0

    OutOfMemoryError: Direct buffer memory

    Google Groups | 3 years ago | Daniel Guo
    java.lang.OutOfMemoryError: Direct buffer memory

    7 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      Direct buffer memory

      at java.nio.Bits.reserveMemory()
    2. Java RT
      1. java.nio.Bits.reserveMemory(
      2. java.nio.DirectByteBuffer.<init>(
      3. java.nio.ByteBuffer.allocateDirect(
      3 frames
    3. Apache Hadoop HDFS
      1. org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(
      2. org.apache.hadoop.hdfs.BlockReaderLocal.<init>(
      3. org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(
      4. org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(
      5. org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(
      6. org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(
      7. org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(
      8 frames
    4. Java RT
      1 frame
    5. HBase
      3 frames
    6. HBase - Client
      1. org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(
      3. org.apache.hadoop.hbase.regionserver.StoreFile.createReader(
      4. org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(
      5. org.apache.hadoop.hbase.regionserver.Store.commitFile(
      6. org.apache.hadoop.hbase.regionserver.Store.access$500(
      7. org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(
      8. org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(
      8 frames