java.lang.RuntimeException: Failed to apply upload policy

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • GitHub comment 232#235795657
    via GitHub by llamahunter
    ,
  • GitHub comment 232#235802943
    via GitHub by HenryCaiHaiying
    ,
  • GitHub comment 160#197607132
    via GitHub by skanda83
    ,
  • While trying to do a distcp from our native HDFS cluster to S3, we get the following error/stacktrace : We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors seen in HADOOP-12420 + other authentication errors). Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333) at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237) at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153) I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem created a ThreadPoolExecutor (which implements AbstractExecutorService which implements ExecutorService). I also checked on the classpath to make sure the version of the jars being picked up is what I expected. Help would be much appreciated.
    via by Jenny Dong,
  • spark - redshift - s3: class path conflict
    via Stack Overflow by Karol Sudol
    ,
    • java.lang.RuntimeException: Failed to apply upload policy at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:116) at com.pinterest.secor.consumer.Consumer.run(Consumer.java:106) Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34) at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:104) at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:195) at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:227) at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:114) ... 1 more Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at com.pinterest.secor.util.FileUtil.getFileSystem(FileUtil.java:87) at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:206) at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:66) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Nikolay Rybak
    Nikolay Rybak1 times, last one,
    Unknown visitor1 times, last one,