java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId

JIRA | luoli | 12 months ago
  1. 0

    when I run the S3UnderStorageCluster integration test using below command, {noformat} mvn clean -Dtest=UnderStorageSystemInterfaceIntegrationTest -DfailIfNoTests=false test -Ps3Test {noformat} the test will fail: {noformat} ------------------------------------------------------- T E S T S ------------------------------------------------------- Running tachyon.underfs.UnderStorageSystemInterfaceIntegrationTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.594 sec <<< FAILURE! tachyon.underfs.UnderStorageSystemInterfaceIntegrationTest Time elapsed: 0.594 sec <<< ERROR! java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://luolis3first/d9a94bf2-04c8-4e21-8b78-3e03e1f30d90/journal6997639942459671163 java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:106) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:90) at tachyon.util.UnderFileSystemUtils.mkdirIfNotExists(UnderFileSystemUtils.java:54) at tachyon.master.AbstractLocalTachyonCluster.setupTest(AbstractLocalTachyonCluster.java:251) at tachyon.master.AbstractLocalTachyonCluster.start(AbstractLocalTachyonCluster.java:105) at tachyon.LocalTachyonClusterResource.apply(LocalTachyonClusterResource.java:167) at org.junit.rules.RunRules.applyAll(RunRules.java:26) at org.junit.rules.RunRules.<init>(RunRules.java:15) {noformat} Look into the code, I found this is because: The integration TestRule LocalTachyonClusterResource will start a LocalTachyonCluster before every case start, LocalTachyonCluster will setup the test environment, including create some test directory on the S3 bucket, the problem is when create the directory on S3 bucket, the LocalTachyonCluster still don't know the accessKey and accesSecretKey and bucket config in the pom file. the configs in pom are in the System properties and don't know by the tachyonConf instance used by the LocalTachyonCluster. To make those configs known by AbstractLocalTachyonCluster should add some config key (like fs.s3n.awsAccessKeyId) to the TachyonConf returned by AbstractLocalTachyonCluster.newTestConf() 。But this is strange because the under filesystem may not be s3. So what about make the config key in pom just as the key need by S3?

    JIRA | 12 months ago | luoli
    java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId
  2. 0

    when I run the S3UnderStorageCluster integration test using below command, {noformat} mvn clean -Dtest=UnderStorageSystemInterfaceIntegrationTest -DfailIfNoTests=false test -Ps3Test {noformat} the test will fail: {noformat} ------------------------------------------------------- T E S T S ------------------------------------------------------- Running tachyon.underfs.UnderStorageSystemInterfaceIntegrationTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.594 sec <<< FAILURE! tachyon.underfs.UnderStorageSystemInterfaceIntegrationTest Time elapsed: 0.594 sec <<< ERROR! java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://luolis3first/d9a94bf2-04c8-4e21-8b78-3e03e1f30d90/journal6997639942459671163 java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:106) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:90) at tachyon.util.UnderFileSystemUtils.mkdirIfNotExists(UnderFileSystemUtils.java:54) at tachyon.master.AbstractLocalTachyonCluster.setupTest(AbstractLocalTachyonCluster.java:251) at tachyon.master.AbstractLocalTachyonCluster.start(AbstractLocalTachyonCluster.java:105) at tachyon.LocalTachyonClusterResource.apply(LocalTachyonClusterResource.java:167) at org.junit.rules.RunRules.applyAll(RunRules.java:26) at org.junit.rules.RunRules.<init>(RunRules.java:15) {noformat} Look into the code, I found this is because: The integration TestRule LocalTachyonClusterResource will start a LocalTachyonCluster before every case start, LocalTachyonCluster will setup the test environment, including create some test directory on the S3 bucket, the problem is when create the directory on S3 bucket, the LocalTachyonCluster still don't know the accessKey and accesSecretKey and bucket config in the pom file. the configs in pom are in the System properties and don't know by the tachyonConf instance used by the LocalTachyonCluster. To make those configs known by AbstractLocalTachyonCluster should add some config key (like fs.s3n.awsAccessKeyId) to the TachyonConf returned by AbstractLocalTachyonCluster.newTestConf() 。But this is strange because the under filesystem may not be s3. So what about make the config key in pom just as the key need by S3?

    JIRA | 12 months ago | luoli
    java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId
  3. 0

    tachyon0.8.2 deployed with hadoop2.6.0,but the IPC version are not matched

    Stack Overflow | 11 months ago | Inner Ac
    java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I tried to set s3 as the underfs by setting deploy/vagrant/provision/roles/tachyon/tasks/start_mesos_framework.yml the following vars under "environment": TACHYON_UNDERFS_ADDRESS: "s3n://nu-spark/double-entry" S3_KEY: "my_aws_secret_key" S3_ID: "my_aws_access_key" And I got the following errors: a) for TachyonMaster task from mesos: 2016-01-29 19:43:31,920 ERROR MASTER_LOGGER (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file. 2016-01-29 19:43:31,923 ERROR MASTER_LOGGER (TachyonMaster.java:main) - Uncaught exception terminating Master java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://nu-spark/double-entry java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId. at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:100) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:83) at tachyon.master.TachyonMaster.connectToUFS(TachyonMaster.java:412) at tachyon.master.TachyonMaster.startMasters(TachyonMaster.java:280) at tachyon.master.TachyonMaster.start(TachyonMaster.java:261) at tachyon.master.TachyonMaster.main(TachyonMaster.java:64) at tachyon.mesos.TachyonMasterExecutor$1.run(TachyonMasterExecutor.java:71) b) For TachyonWorker task on mesos: 2016-01-29 19:33:32,310 ERROR WORKER_LOGGER (ClientBase.java:connect) - Failed to connect (29) to BlockMaster master @ TachyonMaster/10.187.83.93:19998 : java.net.ConnectException: Connection refused 2016-01-29 19:33:32,311 ERROR WORKER_LOGGER (TachyonWorker.java:main) - Failed to initialize the block worker, exiting. java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts at tachyon.ClientBase.connect(ClientBase.java:134) at tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:101) at tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59) at tachyon.worker.block.BlockWorker.<init>(BlockWorker.java:200) at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42) at tachyon.mesos.TachyonWorkerExecutor$1.run(TachyonWorkerExecutor.java:71) Tachyon Mesos Framework does not send the configured accesskey/secretkey on JAVA_OPTS to the running tasks of Tachyon. As a workaround, I put the accesskeys and secretkeys into my own compiled and uploaded tachyon.tar.gz (on tachyon-env.sh) and then it worked.

    JIRA | 10 months ago | Renan Capaverde
    java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId.
  6. 0

    Configuring Mesos with Tachyon Framework

    Google Groups | 10 months ago | Renan Capaverde
    java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId.

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.RuntimeException

      Invalid configuration key fs.s3n.awsAccessKeyId

      at tachyon.underfs.UnderFileSystemRegistry.create()
    2. tachyon.underfs
      UnderFileSystem.get
      1. tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132)
      2. tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:106)
      3. tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:90)
      3 frames
    3. Tachyon Project Core
      LocalTachyonClusterResource.apply
      1. tachyon.util.UnderFileSystemUtils.mkdirIfNotExists(UnderFileSystemUtils.java:54)
      2. tachyon.master.AbstractLocalTachyonCluster.setupTest(AbstractLocalTachyonCluster.java:251)
      3. tachyon.master.AbstractLocalTachyonCluster.start(AbstractLocalTachyonCluster.java:105)
      4. tachyon.LocalTachyonClusterResource.apply(LocalTachyonClusterResource.java:167)
      4 frames
    4. JUnit
      RunRules.<init>
      1. org.junit.rules.RunRules.applyAll(RunRules.java:26)
      2. org.junit.rules.RunRules.<init>(RunRules.java:15)
      2 frames