org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/METASTORE/keys

  1. 0

    Hive does not start: Error creating path /hive/cluster/delegation/METASTORE/keys

    Stack Overflow | 3 months ago | Dennis Jaheruddin
    org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/METASTORE/keys
  2. 0

    Secure Zookeeper for CDH4.1.1 - Grokbase

    grokbase.com | 6 months ago
    java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
  3. 0

    In our product cluster, because of the net problem to kerberos server, the ZooKeeperWatcher in active hmaster fails to Auth , gets a connection Event of AUTH_FAILED and loose the master lock. But the zookeeper watcher ignores the event, so the old active hmaster keeps to be active. After the net problem is fixed, the backup hmaster gets the master lock and becomes active. There are two two active hmasters in the cluster. 2013-05-30 09:44:21,004 ERROR org.apache.zookeeper.client.ZooKeeperSaslClient: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: krb1.xiaomi.net)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. 2013-05-30 09:54:07,755 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection-0x3e10d98be405bc Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:123) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:231) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:595) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:850) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:825) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:286) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:201) at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200) at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:168) at org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123) at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134) at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92) at org.apache.hadoop.hbase.Chore.run(Chore.java:67) at java.lang.Thread.run(Thread.java:662) I want to just abort the hmaster server if AuthFailed or SaslAuthenticated. Any better idea about this issue? For ZookeeperWatcher is used in many classes, will the aborting will bring more problems? Any more problems we need consider?

    Apache's JIRA Issue Tracker | 4 years ago | Liu Shaohui
    org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase/master
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.zookeeper.KeeperException$AuthFailedException

      KeeperErrorCode = AuthFailed for /hive/cluster/delegation/METASTORE/keys

      at org.apache.zookeeper.KeeperException.create()
    2. Zookeeper
      ZooKeeper.create
      1. org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
      2. org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      3. org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
      3 frames
    3. Curator Framework
      CreateBuilderImpl$11.call
      1. org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
      2. org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
      2 frames
    4. Fabric8
      RetryLoop.callWithRetry
      1. org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
      1 frame
    5. Curator Framework
      CreateBuilderImpl$3.forPath
      1. org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
      2. org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
      3. org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
      4. org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
      5. org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
      6. org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
      6 frames
    6. Hive Shims
      HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager
      1. org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
      2. org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
      3. org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
      4. org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444)
      4 frames
    7. Hive Metastore
      HiveMetaStore.main
      1. org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6015)
      2. org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5930)
      2 frames
    8. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:498)
      4 frames
    9. Hadoop
      RunJar.main
      1. org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      2. org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      2 frames