org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)

SpringSource Issue Tracker | Wei Hu (QE) | 4 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)
  2. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException

      org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)

      at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta()
    2. HBase - Client
      CatalogTracker.waitForMeta
      1. org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486)
      1 frame
    3. HBase
      CreateTableHandler.<init>
      1. org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80)
      1 frame
    4. HBase - Client
      HMaster.createTable
      1. org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059)
      1 frame
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    6. HBase - Client
      HBaseServer$Handler.run
      1. org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      2. org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
      2 frames