java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts

JIRA | Renan Capaverde | 10 months ago
  1. 0

    I tried to set s3 as the underfs by setting deploy/vagrant/provision/roles/tachyon/tasks/start_mesos_framework.yml the following vars under "environment": TACHYON_UNDERFS_ADDRESS: "s3n://nu-spark/double-entry" S3_KEY: "my_aws_secret_key" S3_ID: "my_aws_access_key" And I got the following errors: a) for TachyonMaster task from mesos: 2016-01-29 19:43:31,920 ERROR MASTER_LOGGER (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file. 2016-01-29 19:43:31,923 ERROR MASTER_LOGGER (TachyonMaster.java:main) - Uncaught exception terminating Master java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://nu-spark/double-entry java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId. at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:100) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:83) at tachyon.master.TachyonMaster.connectToUFS(TachyonMaster.java:412) at tachyon.master.TachyonMaster.startMasters(TachyonMaster.java:280) at tachyon.master.TachyonMaster.start(TachyonMaster.java:261) at tachyon.master.TachyonMaster.main(TachyonMaster.java:64) at tachyon.mesos.TachyonMasterExecutor$1.run(TachyonMasterExecutor.java:71) b) For TachyonWorker task on mesos: 2016-01-29 19:33:32,310 ERROR WORKER_LOGGER (ClientBase.java:connect) - Failed to connect (29) to BlockMaster master @ TachyonMaster/10.187.83.93:19998 : java.net.ConnectException: Connection refused 2016-01-29 19:33:32,311 ERROR WORKER_LOGGER (TachyonWorker.java:main) - Failed to initialize the block worker, exiting. java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts at tachyon.ClientBase.connect(ClientBase.java:134) at tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:101) at tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59) at tachyon.worker.block.BlockWorker.<init>(BlockWorker.java:200) at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42) at tachyon.mesos.TachyonWorkerExecutor$1.run(TachyonWorkerExecutor.java:71) Tachyon Mesos Framework does not send the configured accesskey/secretkey on JAVA_OPTS to the running tasks of Tachyon. As a workaround, I put the accesskeys and secretkeys into my own compiled and uploaded tachyon.tar.gz (on tachyon-env.sh) and then it worked.

    JIRA | 10 months ago | Renan Capaverde
    java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts
  2. 0

    I tried to set s3 as the underfs by setting deploy/vagrant/provision/roles/tachyon/tasks/start_mesos_framework.yml the following vars under "environment": TACHYON_UNDERFS_ADDRESS: "s3n://nu-spark/double-entry" S3_KEY: "my_aws_secret_key" S3_ID: "my_aws_access_key" And I got the following errors: a) for TachyonMaster task from mesos: 2016-01-29 19:43:31,920 ERROR MASTER_LOGGER (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file. 2016-01-29 19:43:31,923 ERROR MASTER_LOGGER (TachyonMaster.java:main) - Uncaught exception terminating Master java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://nu-spark/double-entry java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId. at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:100) at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:83) at tachyon.master.TachyonMaster.connectToUFS(TachyonMaster.java:412) at tachyon.master.TachyonMaster.startMasters(TachyonMaster.java:280) at tachyon.master.TachyonMaster.start(TachyonMaster.java:261) at tachyon.master.TachyonMaster.main(TachyonMaster.java:64) at tachyon.mesos.TachyonMasterExecutor$1.run(TachyonMasterExecutor.java:71) b) For TachyonWorker task on mesos: 2016-01-29 19:33:32,310 ERROR WORKER_LOGGER (ClientBase.java:connect) - Failed to connect (29) to BlockMaster master @ TachyonMaster/10.187.83.93:19998 : java.net.ConnectException: Connection refused 2016-01-29 19:33:32,311 ERROR WORKER_LOGGER (TachyonWorker.java:main) - Failed to initialize the block worker, exiting. java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts at tachyon.ClientBase.connect(ClientBase.java:134) at tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:101) at tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59) at tachyon.worker.block.BlockWorker.<init>(BlockWorker.java:200) at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42) at tachyon.mesos.TachyonWorkerExecutor$1.run(TachyonWorkerExecutor.java:71) Tachyon Mesos Framework does not send the configured accesskey/secretkey on JAVA_OPTS to the running tasks of Tachyon. As a workaround, I put the accesskeys and secretkeys into my own compiled and uploaded tachyon.tar.gz (on tachyon-env.sh) and then it worked.

    JIRA | 10 months ago | Renan Capaverde
    java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts
  3. 0

    Configuring Mesos with Tachyon Framework

    Google Groups | 10 months ago | Renan Capaverde
    java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Configuring Mesos with Tachyon Framework

    Google Groups | 10 months ago | Renan Capaverde
    java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts
  6. 0

    hadoop 2.7.1 with Tachyon: failed to connect to BlockMaster

    Google Groups | 1 year ago | Xing Lin
    java.lang.RuntimeException: java.io.IOException: Failed to connect to BlockMaster @ scspr0064610004.gdl.englab.netapp.com/10.235.12.237:19998 after 29 attempts

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts

      at tachyon.ClientBase.connect()
    2. Tachyon Project Core
      WorkerIdRegistry.registerWithBlockMaster
      1. tachyon.ClientBase.connect(ClientBase.java:134)
      2. tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:101)
      3. tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59)
      3 frames
    3. tachyon.worker.block
      BlockWorker.<init>
      1. tachyon.worker.block.BlockWorker.<init>(BlockWorker.java:200)
      1 frame
    4. Tachyon Project Core
      TachyonWorker.main
      1. tachyon.worker.TachyonWorker.main(TachyonWorker.java:42)
      1 frame
    5. tachyon.mesos
      TachyonWorkerExecutor$1.run
      1. tachyon.mesos.TachyonWorkerExecutor$1.run(TachyonWorkerExecutor.java:71)
      1 frame