org.ietf.jgss.GSSException: Channel binding mismatch (Mechanism level: ChannelBinding not provided!)

switch.ch | 4 months ago
  1. 0

    Troubleshooting - Shibboleth IdP V3 Kerberos Authentication Flow - SWITCH Forge

    switch.ch | 4 months ago
    org.ietf.jgss.GSSException: Channel binding mismatch (Mechanism level: ChannelBinding not provided!)
  2. 0

    DSE Graph authentication error

    Stack Overflow | 4 months ago | peleitor
    org.ietf.jgss.GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)
  3. 0

    After enabling HDFS Data at Rest Encryption using Cloudera Manager and a Java KMS, Mapreduce programs appear to fail. The "cdap" user is the KMS admin, as are users in the "sysadmin" group. {code} 2016-03-21 23:48:54,979 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@289] - Submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 2016-03-21 23:48:54,996 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.i.LocalClientProtocolProvider@42] - Using framework: yarn 2016-03-21 23:48:55,659 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-03-21 23:48:55,670 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:java.io.IOException: java.lang.reflect.UndeclaredThrowableException 2016-03-21 23:48:55,688 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@300] - Exception when submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,847 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,879 - ERROR [pcontroller-program:default.PurchaseHistory.mapreduce.PurchaseHistoryBuilder-5424fb96-efbf-11e5-b900-42010af00026:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@311] - Program runner error out. java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,912 - ERROR [TwillContainerService:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@323] - Program PurchaseHistoryBuilder execution failed. java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:317) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at org.apache.twill.internal.container.TwillContainerService.doRun(TwillContainerService.java:130) [org.apache.twill.twill-yarn-0.6.0-incubating.jar:0.6.0-incubating] at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179) [org.apache.twill.twill-core-0.6.0-incubating.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] ... 1 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted {code}

    Cask Community Issue Tracker | 9 months ago | Chris Gianelloni
    java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    After enabling HDFS Data at Rest Encryption using Cloudera Manager and a Java KMS, Mapreduce programs appear to fail. The "cdap" user is the KMS admin, as are users in the "sysadmin" group. {code} 2016-03-21 23:48:54,979 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@289] - Submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 2016-03-21 23:48:54,996 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.i.LocalClientProtocolProvider@42] - Using framework: yarn 2016-03-21 23:48:55,659 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-03-21 23:48:55,670 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:java.io.IOException: java.lang.reflect.UndeclaredThrowableException 2016-03-21 23:48:55,688 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@300] - Exception when submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,847 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,879 - ERROR [pcontroller-program:default.PurchaseHistory.mapreduce.PurchaseHistoryBuilder-5424fb96-efbf-11e5-b900-42010af00026:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@311] - Program runner error out. java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,912 - ERROR [TwillContainerService:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@323] - Program PurchaseHistoryBuilder execution failed. java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:317) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at org.apache.twill.internal.container.TwillContainerService.doRun(TwillContainerService.java:130) [org.apache.twill.twill-yarn-0.6.0-incubating.jar:0.6.0-incubating] at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179) [org.apache.twill.twill-core-0.6.0-incubating.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] ... 1 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted {code}

    Cask Community Issue Tracker | 9 months ago | Chris Gianelloni
    java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
  6. 0

    After enabling HDFS Data at Rest Encryption using Cloudera Manager and a Java KMS, Mapreduce programs appear to fail. The "cdap" user is the KMS admin, as are users in the "sysadmin" group. {code} 2016-03-21 23:48:54,979 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@289] - Submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 2016-03-21 23:48:54,996 - INFO [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.i.LocalClientProtocolProvider@42] - Using framework: yarn 2016-03-21 23:48:55,659 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-03-21 23:48:55,670 - WARN [MapReduceRunner-PurchaseHistoryBuilder:o.a.h.s.UserGroupInformation@1674] - PriviledgedActionException as:cdap (auth:KERBEROS) cause:java.io.IOException: java.lang.reflect.UndeclaredThrowableException 2016-03-21 23:48:55,688 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.b.MapReduceRuntimeService@300] - Exception when submitting MapReduce Job: job=PurchaseHistoryBuilder,=namespaceId=default, applicationId=PurchaseHistory, program=PurchaseHistoryBuilder, runid=5424fb96-efbf-11e5-b900-42010af00026 java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,847 - ERROR [MapReduceRunner-PurchaseHistoryBuilder:c.c.c.i.a.r.ProgramControllerServiceAdapter@81] - Program terminated with exception java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) [co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,879 - ERROR [pcontroller-program:default.PurchaseHistory.mapreduce.PurchaseHistoryBuilder-5424fb96-efbf-11e5-b900-42010af00026:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@311] - Program runner error out. java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted 2016-03-21 23:48:55,912 - ERROR [TwillContainerService:c.c.c.i.a.r.d.AbstractProgramTwillRunnable@323] - Program PurchaseHistoryBuilder execution failed. java.util.concurrent.ExecutionException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:281) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.distributed.AbstractProgramTwillRunnable.run(AbstractProgramTwillRunnable.java:317) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at org.apache.twill.internal.container.TwillContainerService.doRun(TwillContainerService.java:130) [org.apache.twill.twill-yarn-0.6.0-incubating.jar:0.6.0-incubating] at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179) [org.apache.twill.twill-core-0.6.0-incubating.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:881) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2142) ~[hadoop-hdfs-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:169) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.2.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:291) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) [com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$1$1.run(MapReduceRuntimeService.java:387) ~[co.cask.cdap.cdap-app-fabric-3.3.1.jar:na] ... 1 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 16 common frames omitted Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:868) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:863) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) ~[hadoop-common-2.6.0-cdh5.5.2.jar:na] ... 17 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_67] at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_67] at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_67] at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ~[hadoop-auth-2.6.0-cdh5.5.2.jar:na] ... 28 common frames omitted {code}

    Cask Community Issue Tracker | 9 months ago | Chris Gianelloni
    java.io.IOException: java.lang.reflect.UndeclaredThrowableException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.ietf.jgss.GSSException

      Channel binding mismatch (Mechanism level: ChannelBinding not provided!)

      at sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>()
    2. Java RT
      InitSecContextToken.<init>
      1. sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>(InitialToken.java:263)[na:1.6.0_17]
      2. sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:126)[na:1.6.0_17]
      2 frames