org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request.

JIRA | Michal Malohlava | 3 years ago
  1. 0

    1. step - launch 16 nodes (m2.2xlarge) 2. load via HDFS: s3n://h2o-airlines-unpacked/allyears.csv 3. run GLM on allyears with corresponding columns ignored 4. load s3n://h2o-airlines-unpacked/year2012.csv UI reports: Got Exception DistributedException, with msg from /10.159.2.113:54321; java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. while mapping key $00000000401700000000$s3n://h2o-airlines-unpacked/year2012.csv Logs reports: 04:38:36.223 FJ-9-18 ERRR WATER: + org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. + at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieve(Jets3tNativeFileSystemStore.java:154) + at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) + at java.lang.reflect.Method.invoke(Method.java:597) + at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) + at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) + at org.apache.hadoop.fs.s3native.$Proxy5.retrieve(Unknown Source) + at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.seek(NativeS3FileSystem.java:111) + at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:76) + at org.apache.hadoop.fs.BufferedFSInputStream.skip(BufferedFSInputStream.java:56) + at java.io.FilterInputStream.skip(FilterInputStream.java:125) + at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:683) + at water.persist.PersistHdfs$2.call(PersistHdfs.java:153) + at water.persist.PersistHdfs.run(PersistHdfs.java:239) + at water.persist.PersistHdfs.load(PersistHdfs.java:143) + at water.Value.loadPersist(Value.java:190) + at water.Value.memOrLoad(Value.java:88) + at water.parser.DParseTask$VAChunkDataIn.getChunkData(DParseTask.java:138) + at water.parser.CsvParser.parallelParse(CsvParser.java:416) + at water.parser.DParseTask.map(DParseTask.java:545) + at water.MRTask.lcompute(MRTask.java:66) + at water.DRemoteTask.compute2(DRemoteTask.java:91) + at water.H2O$H2OCountedCompleter.compute(H2O.java:683) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) + Caused by: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. + at org.jets3t.service.S3Service.sleepOnInternalError(S3Service.java:344) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:374) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1558) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1501) + at org.jets3t.service.S3Service.getObject(S3Service.java:1876) + at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieve(Jets3tNativeFileSystemStore.java:144) + ... 27 more

    JIRA | 3 years ago | Michal Malohlava
    org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request.
  2. 0

    1. step - launch 16 nodes (m2.2xlarge) 2. load via HDFS: s3n://h2o-airlines-unpacked/allyears.csv 3. run GLM on allyears with corresponding columns ignored 4. load s3n://h2o-airlines-unpacked/year2012.csv UI reports: Got Exception DistributedException, with msg from /10.159.2.113:54321; java.lang.RuntimeException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. while mapping key $00000000401700000000$s3n://h2o-airlines-unpacked/year2012.csv Logs reports: 04:38:36.223 FJ-9-18 ERRR WATER: + org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. + at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieve(Jets3tNativeFileSystemStore.java:154) + at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) + at java.lang.reflect.Method.invoke(Method.java:597) + at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) + at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) + at org.apache.hadoop.fs.s3native.$Proxy5.retrieve(Unknown Source) + at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.seek(NativeS3FileSystem.java:111) + at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:76) + at org.apache.hadoop.fs.BufferedFSInputStream.skip(BufferedFSInputStream.java:56) + at java.io.FilterInputStream.skip(FilterInputStream.java:125) + at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:683) + at water.persist.PersistHdfs$2.call(PersistHdfs.java:153) + at water.persist.PersistHdfs.run(PersistHdfs.java:239) + at water.persist.PersistHdfs.load(PersistHdfs.java:143) + at water.Value.loadPersist(Value.java:190) + at water.Value.memOrLoad(Value.java:88) + at water.parser.DParseTask$VAChunkDataIn.getChunkData(DParseTask.java:138) + at water.parser.CsvParser.parallelParse(CsvParser.java:416) + at water.parser.DParseTask.map(DParseTask.java:545) + at water.MRTask.lcompute(MRTask.java:66) + at water.DRemoteTask.compute2(DRemoteTask.java:91) + at water.H2O$H2OCountedCompleter.compute(H2O.java:683) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) + Caused by: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request. + at org.jets3t.service.S3Service.sleepOnInternalError(S3Service.java:344) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:374) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1558) + at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1501) + at org.jets3t.service.S3Service.getObject(S3Service.java:1876) + at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieve(Jets3tNativeFileSystemStore.java:144) + ... 27 more

    JIRA | 3 years ago | Michal Malohlava
    org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Encountered too many S3 Internal Server errors (6), aborting request.
  3. 0

    handle authentication exception when connection to private repo fails

    GitHub | 2 years ago | jumblerg
    org.jets3t.service.S3ServiceException: S3 Error Message. GET '/snapshot%2Ftailrecursion%2Fclojure-adapter-servlet%2F0.1.0-SNAPSHOT%2Fmaven-metadata.xml' on Host '<my>.repository.s3.amazonaws.com' @ 'Fri, 03 Oct 2014 15:30:57 GMT' -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E7EDF9C66A8B855D</RequestId><HostId>U40MhHaosnmvEeU0X8l6wOpNbNY75pcRHaiN6ELQaREnknbQi5bKkRTSdWRPugIK</HostId></Error>
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    DegreeGridNEDTileSource attempting to load non-existent NED tiles

    GitHub | 5 years ago | demory
    java.lang.RuntimeException: org.jets3t.service.S3ServiceException: S3 Error Message. GET '/n39w074.tiff' on Host 'ned13.s3.amazonaws.com' @ 'Fri, 27 Apr 2012 13:41:51 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>n39w074.tiff</Key><RequestId>CFA9AA884873B12D</RequestId><HostId>7tp2V7Fw65BLD035BBomwZutkYSmP2G7mHScL1AVSNEcXI9ASQPyn/ncN98Pjnb1</HostId></Error>
  6. 0

    Benign error on first upload to S3 · spring-projects/spring-build-gradle Wiki · GitHub

    github.com | 1 year ago
    org.jets3t.service.S3ServiceException: S3 Error Message. GET '/milestone%2Forg%2Fspringframework%2Fmobile%2Fspring-mobile-device%2Fmaven-metadata.xml' on Host 'maven.springframework.org.s3.amazonaws.com' @ 'Thu, 18 Nov 2010 21:49:58 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>milestone/org/springframework/mobile/spring-mobile-device/maven-metadata.xml</Key><RequestId>6D57E03E89170217</RequestId><HostId>AUqRbON70RPGuk/BPsnwEUzWNOZGvEEMzglqD1pYFJVcdZR9huq97PN5v34XCt9n</HostId></Error>

    2 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.jets3t.service.S3ServiceException

      Encountered too many S3 Internal Server errors (6), aborting request.

      at org.jets3t.service.S3Service.sleepOnInternalError()
    2. JetS3t
      S3Service.getObject
      1. org.jets3t.service.S3Service.sleepOnInternalError(S3Service.java:344)
      2. org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:374)
      3. org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
      4. org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1558)
      5. org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1501)
      6. org.jets3t.service.S3Service.getObject(S3Service.java:1876)
      6 frames
    3. Hadoop
      Jets3tNativeFileSystemStore.retrieve
      1. org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieve(Jets3tNativeFileSystemStore.java:144)
      1 frame
    4. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      3. java.lang.reflect.Method.invoke(Method.java:597)
      3 frames
    5. Hadoop
      BufferedFSInputStream.skip
      1. org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      2. org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      3. org.apache.hadoop.fs.s3native.$Proxy5.retrieve(Unknown Source)
      4. org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.seek(NativeS3FileSystem.java:111)
      5. org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:76)
      6. org.apache.hadoop.fs.BufferedFSInputStream.skip(BufferedFSInputStream.java:56)
      6 frames
    6. Java RT
      FilterInputStream.skip
      1. java.io.FilterInputStream.skip(FilterInputStream.java:125)
      1 frame
    7. Guava
      ByteStreams.skipFully
      1. com.google.common.io.ByteStreams.skipFully(ByteStreams.java:683)
      1 frame
    8. water.persist
      PersistHdfs.load
      1. water.persist.PersistHdfs$2.call(PersistHdfs.java:153)
      2. water.persist.PersistHdfs.run(PersistHdfs.java:239)
      3. water.persist.PersistHdfs.load(PersistHdfs.java:143)
      3 frames
    9. water
      Value.memOrLoad
      1. water.Value.loadPersist(Value.java:190)
      2. water.Value.memOrLoad(Value.java:88)
      2 frames
    10. water.parser
      DParseTask.map
      1. water.parser.DParseTask$VAChunkDataIn.getChunkData(DParseTask.java:138)
      2. water.parser.CsvParser.parallelParse(CsvParser.java:416)
      3. water.parser.DParseTask.map(DParseTask.java:545)
      3 frames
    11. water
      H2O$H2OCountedCompleter.compute
      1. water.MRTask.lcompute(MRTask.java:66)
      2. water.DRemoteTask.compute2(DRemoteTask.java:91)
      3. water.H2O$H2OCountedCompleter.compute(H2O.java:683)
      3 frames
    12. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames