java.io.IOException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • This might not be a bug. Here is the description. Any workarounds are appreciated. I am only able to execute hadoop commands using principals which are in the default realm. hadoop.security.auth_to_local seems to be ignored. Attached is a log of everything done. Here is overview of the configuration and some troubleshooting tests: # created and tested a principal using the KDC instead of AD and confirmed all OK hadoop org.apache.hadoop.security.HadoopKerberosName george@EC2.INTERNAL Name: george@EC2.INTERNAL to george # fails to use with principal from AD, seems to ignore rules in hadoop.security.auth_to_local hadoop org.apache.hadoop.security.HadoopKerberosName george@CLOUDSECURE.LOCAL Exception in thread "main" org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to george@CLOUDSECURE.LOCAL at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:378) at org.apache.hadoop.security.HadoopKerberosName.main(HadoopKerberosName.java:74) # note: ip-10-151-51-135.ec2.internal has Win 2008 R2 + AD DS with 1 forest, and defines all user accounts used for authentication /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] default_realm = EC2.INTERNAL dns_lookup_realm = false dns_lookup_kdc = false max_life = 1d max_renewable_life = 7d ticket_lifetime = 24h renew_lifetime = 7d forwardable = true default_tgs_enctypes = aes256-cts aes128-cts arcfour-hmac des3-hmac-sha1 des-hmac-sha1 des-cbc-md5 des-cbc-crc default_tkt_enctypes = aes256-cts aes128-cts arcfour-hmac des3-hmac-sha1 des-hmac-sha1 des-cbc-md5 des-cbc-crc [realms] EC2.INTERNAL = { kdc = ip-10-191-70-81.ec2.internal admin_server = ip-10-191-70-81.ec2.internal default_domain = EC2.INTERNAL } CLOUDSECURE.LOCAL = { kdc = ip-10-151-51-135.ec2.internal:88 admin_server = ip-10-151-51-135.ec2.internal:749 default_domain = EC2.INTERNAL } [domain_realm] .ec2.internal = EC2.INTERNAL ec2.internal = EC2.INTERNAL cat /etc/hadoop/conf.cloudera.hdfs1/core-site.xml <?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera CM on 2013-10-06T10:16:50.792Z--> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://ip-10-191-70-81.ec2.internal:8020</value> </property> <property> <name>fs.trash.interval</name> <value>1</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hadoop.rpc.protection</name> <value>authentication</value> </property> <property> <name>hadoop.security.auth_to_local</name> <value>RULE:[1:$1@$0](.*@\QEC2.INTERNAL\E$)s/@\QEC2.INTERNAL\E$// RULE:[2:$1@$0](.*@\QEC2.INTERNAL\E$)s/@\QEC2.INTERNAL\E$// RULE:[1:$1@$0](.*@\QCLOUDSECURE.LOCAL\E$)s/@\QCLOUDSECURE.LOCAL\E$// RULE:[2:$1@$0](.*@\QCLOUDSECURE.LOCAL\E$)s/@\QCLOUDSECURE.LOCAL\E$// DEFAULT</value> </property> </configuration>
    via by Daniel Rule,
  • Here's what I'm observing on a fully distributed cluster deployed via Bigtop from the RC0 2.0.3-alpha tarball: {noformat} 528077-oozie-tucu-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to yarn/localhost@LOCALREALM at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier.<init>(AbstractDelegationTokenIdentifier.java:68) at org.apache.hadoop.mapreduce.v2.api.MRDelegationTokenIdentifier.<init>(MRDelegationTokenIdentifier.java:51) at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getDelegationToken(HistoryClientService.java:336) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getDelegationToken(MRClientProtocolPBServiceImpl.java:210) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:240) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729) Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to yarn/localhost@LOCALREALM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:378) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier.<init>(AbstractDelegationTokenIdentifier.java:66) ... 12 more ] {noformat} This is submitting a mapreduce job via Oozie 3.3.1. The reason I think this is a Hadoop issue rather than the oozie one is because when I hack /etc/krb5.conf to be: {noformat} [libdefaults] ticket_lifetime = 600 default_realm = LOCALHOST default_tkt_enctypes = des3-hmac-sha1 des-cbc-crc default_tgs_enctypes = des3-hmac-sha1 des-cbc-crc [realms] LOCALHOST = { kdc = localhost:88 default_domain = .local } [domain_realm] .local = LOCALHOST [logging] kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmin.log default = FILE:/var/log/krb5lib.log {noformat} The issue goes away. Now, once again -- the kerberos auth is NOT configured for Hadoop, hence it should NOT pay attention to /etc/krb5.conf to begin with.
    via by Roman Shaposhnik,
  • SOLR security configured accordingly [this document|http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh5sg_search_security.html] User from primary realm (used by Hadoop cluster itself) can access the console, but user from trusted realm can't. {code} Sep 24, 2014 9:30:13 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet LoadAdminUI threw exception org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to admin@TRUSTED.REALM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:359) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:329) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:329) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:349) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:148) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) {code} Required kerberos auth_to_local rules are defined in hadoop/core-site.xml file and was added to /etc/krb5.conf as well. Another CDH components (for example, Impala) use these rules and allow access for users from trusted domain.
    via by Andrejs Dubovskis,
    • java.io.IOException: Login failure for name@XX.XX.COM from keytab \\NASdrive\name.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name name@XX.XX.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to name@XX.XX.COM at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) at Appname.ldapLookupLoop(Appname.java:111) at Appname.main(Appname.java:70) Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name name@XX.XX.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to name@XX.XX.COM at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at javax.security.auth.login.LoginContext.invoke(Unknown Source) at javax.security.auth.login.LoginContext.access$000(Unknown Source) at javax.security.auth.login.LoginContext$4.run(Unknown Source) at javax.security.auth.login.LoginContext$4.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(Unknown Source) at javax.security.auth.login.LoginContext.login(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953) ... 2 more Caused by: java.lang.IllegalArgumentException: Illegal principal name name@XX.XX.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to name@XX.XX.COM at org.apache.hadoop.security.User.<init>(User.java:51) at org.apache.hadoop.security.User.<init>(User.java:43) at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197) ... 14 more Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to name@XX.XX.COM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.User.<init>(User.java:48) ... 16 more

    Users with the same issue

    Unknown visitor1 times, last one,