java.io.IOException: Failed to cache: alluxio.exception. BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed

Google Groups | Amran Chen | 5 months ago
  1. 0

    Hadoop S3 Error

    Google Groups | 5 months ago | Amran Chen
    java.io.IOException: Failed to cache: alluxio.exception. BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed
  2. 0

    Debugging Alluxio fails to request bytes for block (during Spark Terasort)?

    Google Groups | 6 months ago | Tim B
    java.io.IOException: Failed to cache: Unable to request space from worker
  3. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed)
  6. 0

    Using Tachyon with Spark causes failure without Tachyon works fine

    Google Groups | 12 months ago | Sam Stoelinga
    java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 16,777,216 is not available, because it already exists)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Failed to cache: alluxio.exception. BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed

      at alluxio.client.file.FileOutStream.handleCacheWriteException()
    2. alluxio.client.file
      FileOutStream.write
      1. alluxio.client.file.FileOutStream.handleCacheWriteException(FileOutStream.java:342)
      2. alluxio.client.file.FileOutStream.write(FileOutStream.java:256)
      2 frames
    3. Java RT
      DataOutputStream.writeInt
      1. java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
      1 frame
    4. alluxio.examples
      BasicNonByteBufferOperations.call
      1. alluxio.examples.BasicNonByteBufferOperations.write(BasicNonByteBufferOperations.java:85)
      2. alluxio.examples.BasicNonByteBufferOperations.call(BasicNonByteBufferOperations.java:78)
      3. alluxio.examples.BasicNonByteBufferOperations.call(BasicNonByteBufferOperations.java:46)
      3 frames
    5. alluxio.cli
      TestRunner.main
      1. alluxio.cli.CliUtils.runExample(CliUtils.java:51)
      2. alluxio.cli.TestRunner.runTest(TestRunner.java:152)
      3. alluxio.cli.TestRunner.runTests(TestRunner.java:121)
      4. alluxio.cli.TestRunner.main(TestRunner.java:98)
      4 frames