org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location

Apache's JIRA Issue Tracker | Josh Elser | 9 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Saw an error in some $dayjob testing where, while a RegionServer was going down to due to an exception, there was a scary looking exception about being unable to write to the stats table because an hconnection was closed. Pardon the mis-matched line numbers: {noformat} 2016-07-17 07:52:13,229 ERROR [phoenix-update-statistics-0] stats.StatisticsScanner: Failed to update statistics table! org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794) at org.apache.hadoop.hbase.client.HTableWrapper.getScanner(HTableWrapper.java:215) at org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:136) at org.apache.phoenix.schema.stats.StatisticsWriter.deleteStats(StatisticsWriter.java:230) at org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:117) at org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:102) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x5314972b closed at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153) at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1133) at org.apache.hadoop.hbase.client.CoprocessorHConnection.relocateRegion(CoprocessorHConnection.java:41) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1338) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162) at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) ... 17 more {noformat} Looking into this some more, this async task to update the stats was still running after a RegionServer already was in the process of shutting down. The RegionServer already closed all of the "userRegions", but, because this task is async, the task is still running and using the RegionServer's CoprocessorHConnection. So, the RegionServer thinks all of the user regions are closed and it is safe to close the HConnection. In reality, there is still code tied to those user regions that might be running (as we can see with the above stacktrace). The next time the StatisticsScannerCallable tries to use the HConnection, it will then error. I think the simple fix is to just use the CoprocessorEnvironment to access the RegionServerServices and use the {{isClosing()}} and {{isClosed()}} methods. This is all pretty minor because the RegionServer is already shutting down, but it is likely misleading to less-experienced users who would think that the last exception in the log is the problem. Will put up a patch shortly.

    Apache's JIRA Issue Tracker | 9 months ago | Josh Elser
    org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location
  2. 0

    ADB Connection Error

    Stack Overflow | 4 years ago | Raveesh Lawrance
    java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(Unknown Source) at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at sun.nio.ch.IOUtil.read(Unknown Source) at sun.nio.ch.SocketChannelImpl.read(Unknown Source) at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:395) at com.android.ddmlib.Device.executeShellCommand(Device.java:462) at com.android.ddmuilib.logcat.LogCatReceiver$1.run(LogCatReceiver.java:110)
  3. 0

    java.lang.IllegalArgumentException: URI scheme is not "file"

    java-forums.org | 9 months ago
    java.io.IOException: Server returned HTTP response code: 403 for URL: at sun.net. at slideshowapplet.SlideshowApplet.doListofImagefiles (SlideshowApplet.java:461) at slideshowapplet.SlideshowApplet.init(SlideshowAppl et.java:180) at sun.plugin2.applet.Plugin2Manager$AppletExecutionR unnable.run(Plugin2Manager.java:1658)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    java.lang.IllegalArgumentException: URI scheme is not "file"

    java-forums.org | 9 months ago
    java.io.IOException: Server returned HTTP response code: 403 for URL: at sun.net. at slideshowapplet.SlideshowApplet.doListofImagefiles (SlideshowApplet.java:484) at slideshowapplet.SlideshowApplet.init(SlideshowAppl et.java:182) at sun.plugin2.applet.Plugin2Manager$AppletExecutionR unnable.run(Plugin2Manager.java:1658)
  6. 0

    data nodes evicted randomly and cluster marks node... - Cloudera Community

    cloudera.com | 2 years ago
    java.io.IOException: Broken pipe         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)         at sun.nio.ch.SocketDispatcher.write(SocketDispatcher .java:47)         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.jav a:93)         at sun.nio.ch.IOUtil.write(IOUtil.java:65)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      hconnection-0x5314972b closed

      at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion()
    2. HBase - Client
      HTableWrapper.getScanner
      1. org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
      2. org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
      3. org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1133)
      4. org.apache.hadoop.hbase.client.CoprocessorHConnection.relocateRegion(CoprocessorHConnection.java:41)
      5. org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1338)
      6. org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
      7. org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
      8. org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
      9. org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152)
      10. org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
      11. org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      12. org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
      13. org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
      14. org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
      15. org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)
      16. org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
      17. org.apache.hadoop.hbase.client.HTableWrapper.getScanner(HTableWrapper.java:215)
      17 frames
    3. Phoenix Core
      StatisticsScanner$StatisticsScannerCallable.call
      1. org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:136)
      2. org.apache.phoenix.schema.stats.StatisticsWriter.deleteStats(StatisticsWriter.java:230)
      3. org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:117)
      4. org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:102)
      4 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      3. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      4. java.lang.Thread.run(Thread.java:745)
      4 frames