org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
  1. 0

    Reproduce steps: 1. Create a cluster with attachment spec file successfully. 2. Execute hbase command in client vm. {code} [serengeti@10 ~]$ hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10000 randomWrite 5 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.5--1, built on 06/05/2012 08:32 GMT 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:host.name=10.111.89.1 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_31 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk/jre 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hbase/conf:/usr/local/jdk/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.92.1-gphd-1.2.0.0-tests.jar:/usr/lib/hbase/bin/../hbase-0.92.1-gphd-1.2.0.0.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.3.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.3.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/lib/hbase/bin/../lib/commons-digester-1.8.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-math-2.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r09.jar:/usr/lib/hbase/bin/../lib/hadoop-core-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/lib/hbase/bin/../lib/httpclient-4.0.1.jar:/usr/lib/hbase/bin/../lib/httpcore-4.0.1.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.5.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/libthrift-0.7.0.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.4.0a.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/slf4j-api-1.5.8.jar:/usr/lib/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/velocity-1.7.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper-3.3.5-gphd-1.2.0.0.jar::/etc/hadoop/conf:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../conf:/usr/local/jdk/lib/tools.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/..:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../hadoop-core-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/asm-3.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-cli-1.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-codec-1.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-configuration-1.6.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-digester-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-el-1.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-io-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-lang-2.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-math-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-net-1.4.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/core-3.1.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-capacity-scheduler-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-fairscheduler-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-thriftfs-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jdeb-0.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-core-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-json-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-server-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jets3t-0.6.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jetty-6.1.26.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsch-0.1.42.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/junit-4.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/kfs-0.2.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/log4j-1.2.15.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/oro-2.0.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/xmlenc-0.52.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsp-2.1/jsp-api-2.1.jar 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-238.19.1.el5 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.name=serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.111.89.89:2181,10.111.88.210:2181,10.111.88.7:2181 sessionTimeout=180000 watcher=hconnection 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.111.89.89:2181 13/03/12 01:16:40 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 6545@10.111.89.1 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Socket connection established to 10.111.89.89/10.111.89.89:2181, initiating session 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Session establishment complete on server 10.111.89.89/10.111.89.89:2181, sessionid = 0x13d5c134cbe0003, negotiated timeout = 40000 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.111.89.89:2181,10.111.88.210:2181,10.111.88.7:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.111.88.210:2181 13/03/12 01:16:40 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 6545@10.111.89.1 13/03/12 01:16:40 DEBUG catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2a0ab444 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Socket connection established to 10.111.88.210/10.111.88.210:2181, initiating session 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Session establishment complete on server 10.111.88.210/10.111.88.210:2181, sessionid = 0x3d5c134a470004, negotiated timeout = 40000 13/03/12 01:16:40 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:40 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=0 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:41 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:41 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=1 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=2 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:43 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:43 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=3 of 10 failed; retrying after sleep of 2000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:45 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:45 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=4 of 10 failed; retrying after sleep of 2000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:47 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:47 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=5 of 10 failed; retrying after sleep of 4000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:51 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:51 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=6 of 10 failed; retrying after sleep of 4000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:55 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:55 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=7 of 10 failed; retrying after sleep of 8000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:17:03 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:17:03 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=8 of 10 failed; retrying after sleep of 16000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:17:19 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:17:19 DEBUG catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2a0ab444 13/03/12 01:17:19 INFO zookeeper.ZooKeeper: Session: 0x3d5c134a470004 closed org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:202) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:216) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:468) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:488) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1158) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1266) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1291) {code} HBase shell also don't work {code} hbase(main):001:0> create 'test', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Cluster info: {code} serengeti>cluster list --name hbase_mapred --detail cluster name: hbase_mapred, distro: gphd, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 1024 SHARED 10 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- hbase_mapred-master-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.118 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- hbaseMaster [hbase_master] 2 1 1024 SHARED 10 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ hbase_mapred-hbaseMaster-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.128 Service Ready hbase_mapred-hbaseMaster-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- hbase_mapred-zookeeper-2 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.7 Service Ready hbase_mapred-zookeeper-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.89 Service Ready hbase_mapred-zookeeper-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.210 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------- HRegionServerWithHDFS [hadoop_datanode, hbase_regionserver] 3 1 1024 LOCAL 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------------------- hbase_mapred-HRegionServerWithHDFS-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.221 Service Ready hbase_mapred-HRegionServerWithHDFS-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.23 Service Ready hbase_mapred-HRegionServerWithHDFS-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.100 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------- compute [hadoop_tasktracker] 3 1 1024 LOCAL 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- hbase_mapred-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.248 Service Ready hbase_mapred-compute-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.63 Service Ready hbase_mapred-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.2 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------------------------ hbaseClient [hadoop_client, pig, hive, hive_server, hbase_client] 1 1 1024 SHARED 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- hbase_mapred-hbaseClient-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.1 Service Ready {code} From HMaster webui, all regionserver is registed to HMaster. But everything works well after half an hour. Expected result: Cluster creation should end after all region server service address is connectable. The issue is intermittent.

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192
  2. 0

    Reproduce steps: 1. Create a cluster with attachment spec file successfully. 2. Execute hbase command in client vm. {code} [serengeti@10 ~]$ hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10000 randomWrite 5 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.5--1, built on 06/05/2012 08:32 GMT 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:host.name=10.111.89.1 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_31 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk/jre 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hbase/conf:/usr/local/jdk/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.92.1-gphd-1.2.0.0-tests.jar:/usr/lib/hbase/bin/../hbase-0.92.1-gphd-1.2.0.0.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.3.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.3.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/lib/hbase/bin/../lib/commons-digester-1.8.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-math-2.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r09.jar:/usr/lib/hbase/bin/../lib/hadoop-core-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/lib/hbase/bin/../lib/httpclient-4.0.1.jar:/usr/lib/hbase/bin/../lib/httpcore-4.0.1.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.5.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/libthrift-0.7.0.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.4.0a.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/slf4j-api-1.5.8.jar:/usr/lib/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/velocity-1.7.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper-3.3.5-gphd-1.2.0.0.jar::/etc/hadoop/conf:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../conf:/usr/local/jdk/lib/tools.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/..:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../hadoop-core-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/asm-3.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-cli-1.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-codec-1.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-configuration-1.6.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-digester-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-el-1.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-io-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-lang-2.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-math-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/commons-net-1.4.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/core-3.1.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-capacity-scheduler-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-fairscheduler-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hadoop-thriftfs-1.0.3-gphd-1.2.0.0.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jdeb-0.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-core-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-json-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jersey-server-1.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jets3t-0.6.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jetty-6.1.26.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsch-0.1.42.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/junit-4.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/kfs-0.2.2.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/log4j-1.2.15.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/oro-2.0.8.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/xmlenc-0.52.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/jsp-2.1/jsp-api-2.1.jar 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop-1.0.3-gphd-1.2.0.0/libexec/../lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-238.19.1.el5 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.name=serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/serengeti 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.111.89.89:2181,10.111.88.210:2181,10.111.88.7:2181 sessionTimeout=180000 watcher=hconnection 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.111.89.89:2181 13/03/12 01:16:40 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 6545@10.111.89.1 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Socket connection established to 10.111.89.89/10.111.89.89:2181, initiating session 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Session establishment complete on server 10.111.89.89/10.111.89.89:2181, sessionid = 0x13d5c134cbe0003, negotiated timeout = 40000 13/03/12 01:16:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.111.89.89:2181,10.111.88.210:2181,10.111.88.7:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.111.88.210:2181 13/03/12 01:16:40 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 6545@10.111.89.1 13/03/12 01:16:40 DEBUG catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2a0ab444 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Socket connection established to 10.111.88.210/10.111.88.210:2181, initiating session 13/03/12 01:16:40 INFO zookeeper.ClientCnxn: Session establishment complete on server 10.111.88.210/10.111.88.210:2181, sessionid = 0x3d5c134a470004, negotiated timeout = 40000 13/03/12 01:16:40 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:40 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=0 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:41 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:41 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=1 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=2 of 10 failed; retrying after sleep of 1000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:43 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:43 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=3 of 10 failed; retrying after sleep of 2000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:45 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:45 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=4 of 10 failed; retrying after sleep of 2000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:47 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:47 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=5 of 10 failed; retrying after sleep of 4000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:51 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:51 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=6 of 10 failed; retrying after sleep of 4000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:16:55 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:16:55 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=7 of 10 failed; retrying after sleep of 8000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:17:03 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:17:03 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=10.111.89.23, port=60020}, attempt=8 of 10 failed; retrying after sleep of 16000 because: No server address listed in -ROOT- for region .META.,,1.1028785192 13/03/12 01:17:19 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@286e4365; serverName=10.111.89.23,60020,1363049637425 13/03/12 01:17:19 DEBUG catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2a0ab444 13/03/12 01:17:19 INFO zookeeper.ZooKeeper: Session: 0x3d5c134a470004 closed org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:202) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:216) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:468) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:488) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1158) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1266) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1291) {code} HBase shell also don't work {code} hbase(main):001:0> create 'test', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Cluster info: {code} serengeti>cluster list --name hbase_mapred --detail cluster name: hbase_mapred, distro: gphd, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ master [hadoop_namenode, hadoop_jobtracker] 1 1 1024 SHARED 10 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- hbase_mapred-master-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.118 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------- hbaseMaster [hbase_master] 2 1 1024 SHARED 10 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ hbase_mapred-hbaseMaster-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.128 Service Ready hbase_mapred-hbaseMaster-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.142 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 1024 SHARED 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- hbase_mapred-zookeeper-2 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.88.7 Service Ready hbase_mapred-zookeeper-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.89 Service Ready hbase_mapred-zookeeper-0 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.210 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------------------------- HRegionServerWithHDFS [hadoop_datanode, hbase_regionserver] 3 1 1024 LOCAL 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------------------- hbase_mapred-HRegionServerWithHDFS-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.88.221 Service Ready hbase_mapred-HRegionServerWithHDFS-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.89.23 Service Ready hbase_mapred-HRegionServerWithHDFS-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.100 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------- compute [hadoop_tasktracker] 3 1 1024 LOCAL 10 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- hbase_mapred-compute-2 sin2-pekaurora-bdcqe014.eng.vmware.com 10.111.88.248 Service Ready hbase_mapred-compute-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.63 Service Ready hbase_mapred-compute-1 sin2-pekaurora-bdcqe013.eng.vmware.com 10.111.89.2 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------------------------ hbaseClient [hadoop_client, pig, hive, hive_server, hbase_client] 1 1 1024 SHARED 10 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- hbase_mapred-hbaseClient-0 sin2-pekaurora-bdcqe012.eng.vmware.com 10.111.89.1 Service Ready {code} From HMaster webui, all regionserver is registed to HMaster. But everything works well after half an hour. Expected result: Cluster creation should end after all region server service address is connectable. The issue is intermittent.

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192
  3. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Reproduce steps: 1.Create a default cdh4 cluster. 2.Run performance: hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=10000 randomWrite 20 {code} org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:215) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:204) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:218) at org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:484) at org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:524) at org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1194) at org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1309) at org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1334) {code} 3. Run HBase shell command: {code} hbase(main):001:0> create 'testtb', 'cf' ERROR: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (10000ms) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:486) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:80) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345) Here is some help for this command: Create table; pass table name, a dictionary of specifications per column family, and optionally a dictionary of table configuration. Dictionaries are described below in the GENERAL NOTES section. Examples: hbase> create 't1', {NAME => 'f1', VERSIONS => 5} hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']} hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'} {code} Expected result: Performance job run successfully. HBase shell works well. Cluster information: {code} HBaseDefaultde, distro: cdh4, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------- master [hadoop_namenode, hbase_master] 1 2 7500 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-master-0 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.50 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------- zookeeper [zookeeper] 3 1 3748 SHARED 20 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------------ HBaseDefaultde-zookeeper-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.103 Service Ready HBaseDefaultde-zookeeper-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.51 Service Ready HBaseDefaultde-zookeeper-1 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.93 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------ worker [hadoop_datanode, hbase_regionserver] 3 1 3748 LOCAL 50 NODE NAME HOST IP STATUS --------------------------------------------------------------------------------------------- HBaseDefaultde-worker-0 sin2-pekaurora-bdcqe004.eng.vmware.com 10.111.57.100 Service Ready HBaseDefaultde-worker-1 sin2-pekaurora-bdcqe003.eng.vmware.com 10.111.57.37 Service Ready HBaseDefaultde-worker-2 sin2-pekaurora-bdcqe006.eng.vmware.com 10.111.57.197 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ----------------------------------------------------------------------------------- client [hadoop_client, hbase_client] 1 1 3748 SHARED 50 NODE NAME HOST IP STATUS -------------------------------------------------------------------------------------------- HBaseDefaultde-client-0 sin2-pekaurora-bdcqe005.eng.vmware.com 10.111.57.47 Service Ready {code} Serengeti server: 10.111.57.25

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.hbase.client.NoServerForRegionException

      No server address listed in -ROOT- for region .META.,,1.1028785192

      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta()
    2. HBase - Client
      HBaseAdmin.tableExists
      1. org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988)
      2. org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814)
      3. org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
      4. org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
      5. org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
      6. org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
      7. org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
      8. org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:700)
      9. org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
      10. org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
      11. org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:202)
      12. org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:216)
      12 frames
    3. HBase
      PerformanceEvaluation.main
      1. org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:468)
      2. org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:488)
      3. org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1158)
      4. org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1266)
      5. org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1291)
      5 frames