org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found

Cask Community Issue Tracker | Rohan Nahata | 1 year ago
  1. 0

    Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state - ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0 at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] ------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------- But digging into logs based on application id and run id on port 8088 tells me that the error is: 2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461) ... 8 more Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959) ... 10 more

    Cask Community Issue Tracker | 1 year ago | Rohan Nahata
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
  2. 0

    Create an adapter on a cluster(cdap insecure singlenode) that takes a stream as input and writes to a tpfs avro sink and scheduled to run every minute. The MR job fails. The logs on the UI state - ERROR [MapReduceRunner-ETLMapReduce:c.c.c.i.a.r.ProgramControllerServiceAdapter$1@73] - Program terminated with exception java.lang.IllegalStateException: MapReduce execution failure: job-id : job_1436232970321_1499uber-mode : falsemap-progress : 0.0reduce-progress : 0.0cleanup-progress : 0.0setup-progress : 0.0runstate : FAILEDstart-time : 1436312495340user-name : yarnpriority : NORMALscheduling-info : NAnum-used-slots0num-reserved-slots0used-mem0reserved-mem0needed-mem0 at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:344) ~[co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:402) [co.cask.cdap.cdap-app-fabric-3.1.0-SNAPSHOT.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] ------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------- But digging into logs based on application id and run id on port 8088 tells me that the error is: 2015-07-07 22:58:41,536 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1961) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461) ... 8 more Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959) ... 10 more

    Cask Community Issue Tracker | 1 year ago | Rohan Nahata
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found
  3. 0

    Taiwan Hadoop Forum • 檢視主題 - HBase 的 ImportFromFile Failed

    hadoop.tw | 1 year ago
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [MAPREDUCE-5957] AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used - ASF JIRA

    apache.org | 1 year ago
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found
  6. 0

    hcatalog读取hive数据并写入hive--niko的文章--CFANZ社区--IT技术分享网站

    cfanz.cn | 1 year ago
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hcatalog.mapreduce.HCatOutputFormat not found

    13 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.ClassNotFoundException

      Class org.apache.avro.mapreduce.AvroKeyOutputFormat not found

      at org.apache.hadoop.conf.Configuration.getClassByName()
    2. Hadoop
      Configuration.getClass
      1. org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1867)
      2. org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1959)
      2 frames
    3. Hadoop
      JobContextImpl.getOutputFormatClass
      1. org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
      1 frame
    4. hadoop-mapreduce-client-app
      MRAppMaster.serviceInit
      1. org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:461)
      2. org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:366)
      2 frames
    5. Hadoop
      AbstractService.init
      1. org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      1 frame
    6. hadoop-mapreduce-client-app
      MRAppMaster$1.run
      1. org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1451)
      1 frame
    7. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:415)
      2 frames
    8. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
      1 frame
    9. hadoop-mapreduce-client-app
      MRAppMaster.main
      1. org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448)
      2. org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381)
      2 frames