org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)

SpringSource Issue Tracker | Wei Hu (QE) | 4 years ago
  1. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)
  2. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    about TestRollingRestart

    Google Groups | 6 years ago | Gaojinchao
    org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (2147483647ms)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException

      org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms)

      at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta()
    2. HBase - Client
      CatalogTracker.waitForMeta
      1. org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486)
      1 frame
    3. HBase
      CreateTableHandler.<init>
      1. org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80)
      1 frame
    4. HBase - Client
      HMaster.createTable
      1. org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059)
      1 frame
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    6. HBase - Client
      HBaseServer$Handler.run
      1. org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      2. org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
      2 frames