java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)

JIRA | cheng chang | 1 year ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)
  2. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)
  3. 0

    Issues with reading data using spark from tachyon worker remotely

    Google Groups | 1 year ago | rajans...@gmail.com
    java.io.IOException: java.io.IOException: File does not exist. response: RPCBlockResponse(17179869303, 335601486, 0, FILE_DNE)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Issues with reading data from tachyon using spark

    Google Groups | 1 year ago | rajans...@gmail.com
    java.io.IOException: java.util.concurrent.TimeoutException: Timeout waiting for task.

    Root Cause Analysis

    1. java.io.IOException

      Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)

      at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock()
    2. tachyon.client.netty
      NettyRemoteBlockReader.readRemoteBlock
      1. tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87)
      1 frame
    3. tachyon.client.block
      BufferedBlockInStream.read
      1. tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94)
      2. tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57)
      3. tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228)
      4. tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130)
      4 frames
    4. tachyon.client.file
      FileInStream.read
      1. tachyon.client.file.FileInStream.read(FileInStream.java:149)
      2. tachyon.client.file.FileInStream.read(FileInStream.java:127)
      2 frames
    5. Tachyon Project Core
      BasicOperations.main
      1. tachyon.examples.BasicOperations.readFile(BasicOperations.java:96)
      2. tachyon.examples.BasicOperations.call(BasicOperations.java:58)
      3. tachyon.examples.BasicOperations.call(BasicOperations.java:39)
      4. tachyon.examples.Utils.runExample(Utils.java:102)
      5. tachyon.examples.BasicOperations.main(BasicOperations.java:115)
      5 frames