org.apache.hadoop.yarn.exceptions.YarnRuntimeException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state - ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0 at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] ------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------- But digging into logs based on application id and run id on port 8088 tells me that the error is: 2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461) ... 8 more Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959) ... 10 more
    via by Rohan Nahata,
  • Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state - ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0 at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] ------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------- But digging into logs based on application id and run id on port 8088 tells me that the error is: 2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461) ... 8 more Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959) ... 10 more
    via by Rohan Nahata,
  • With the job classloader enabled, the MR AM throws ClassNotFoundException if a custom output format class is specified. {noformat} org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469) ... 8 more Caused by: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 10 more {noformat}
    via by Sangjin Lee,
  • batch ingestion error at version(0.9.2)
    via by Zhenyuan Gao,
  • HCatOutputFormat ClassNotFoundException
    via Stack Overflow by robin abishek
    ,
    • org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461) ... 8 more Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959) ... 10 more

    Users with the same issue

    Unknown visitor3 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Nikolay Rybak
    2 times, last one,
    Unknown visitor1 times, last one,
    10 more bugmates