java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)

JIRA | cheng chang | 1 year ago
  1. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)
  2. 0

    I set up a cluster with 3 masters and 5 workers. When leader(TachyonMaster2) was selected, I run "bin/tachyon runTests" on the leader, tests passed. Then I killed the leader, after the new leader(TachyonMaster3) was selected, I could see all files under "/default_tests_files" from the recovered web UI. But when I run "bin/tachyon runTests" on the new leader, 12 tests failed with similar output as: {code:java} /tachyon/bin/tachyon runTest Basic MUST_CACHE Path /default_tests_files/BasicFile_MUST_CACHE does not exist 2015-10-09 13:00:28,702 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,723 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,766 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,767 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,779 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,780 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,828 INFO (BasicOperations.java:createFile) - createFile with fileId 1291845631 took 50 ms. 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,853 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:28,866 INFO (WorkerClient.java:connect) - Connecting local worker @ TachyonMaster3/172.31.18.42:29998 2015-10-09 13:00:28,918 INFO (FileUtils.java:createStorageDirPath) - Folder /mnt/ramdisk/tachyonworker/8365321817942223106 was created! 2015-10-09 13:00:28,920 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /mnt/ramdisk/tachyonworker/8365321817942223106/1275068416 2015-10-09 13:00:28,959 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_MUST_CACHE took 131 ms. 2015-10-09 13:00:29,251 INFO (ClientBase.java:connect) - Tachyon client (version 0.8.0-SNAPSHOT) is trying to connect with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,252 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ TachyonMaster3/172.31.18.42:19998 2015-10-09 13:00:29,277 WARN (FileInStream.java:checkAndAdvanceBlockInStream) - Failed to get TachyonStore stream, the block 1275068416 will not be in TachyonStorage java.io.IOException: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:364) at tachyon.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:63) at tachyon.client.block.TachyonBlockStore.getOutStream(TachyonBlockStore.java:148) at tachyon.client.file.FileInStream.checkAndAdvanceBlockInStream(FileInStream.java:225) at tachyon.client.file.FileInStream.read(FileInStream.java:145) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 1,275,068,416 is not available, because it is already committed) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8913) at tachyon.thrift.WorkerService$requestBlockLocation_result$requestBlockLocation_resultStandardScheme.read(WorkerService.java:8891) at tachyon.thrift.WorkerService$requestBlockLocation_result.read(WorkerService.java:8825) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at tachyon.thrift.WorkerService$Client.recv_requestBlockLocation(WorkerService.java:376) at tachyon.thrift.WorkerService$Client.requestBlockLocation(WorkerService.java:361) at tachyon.worker.WorkerClient.requestBlockLocation(WorkerClient.java:359) ... 10 more 2015-10-09 13:00:29,400 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine TachyonWorker2/172.31.28.247:29999 2015-10-09 13:00:29,491 INFO (NettyRemoteBlockReader.java:readRemoteBlock) - Data 1275068416 from remote machine TachyonWorker2/172.31.28.247:29999 received 2015-10-09 13:00:29,492 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@581c16b1 java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130) at tachyon.client.file.FileInStream.read(FileInStream.java:149) at tachyon.client.file.FileInStream.read(FileInStream.java:127) at tachyon.examples.BasicOperations.readFile(BasicOperations.java:96) at tachyon.examples.BasicOperations.call(BasicOperations.java:58) at tachyon.examples.BasicOperations.call(BasicOperations.java:39) at tachyon.examples.Utils.runExample(Utils.java:102) at tachyon.examples.BasicOperations.main(BasicOperations.java:115) Caused by: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87) ... 11 more Failed the test! {code} After "bin/tachyon runTests" exited, I could see all the test files written from web UI, but if I tried to browse a specific file, errors like the following would be shown on web UI: {code:java} Error: File /default_tests_files/BasicFile_ASYNC_THROUGH is not available java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1610612736, 0, 0, BLOCK_LOCK_ERROR) {code} Then I run "bin/tachyon tfs rmr /default_tests_files", it succeeded, "Browse File System" on web UI showed that the files were deleted, but "Workers" tab showed that the "Space Used" didn't change, then I run "ls -R /mnt/ramdisk", it showed that the blocks were not actually deleted.

    JIRA | 1 year ago | cheng chang
    java.io.IOException: java.io.IOException: Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)
  3. 0

    Issues with reading data using spark from tachyon worker remotely

    Google Groups | 10 months ago | rajans...@gmail.com
    java.io.IOException: java.io.IOException: File does not exist. response: RPCBlockResponse(17179869303, 335601486, 0, FILE_DNE)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Issues with reading data from tachyon using spark

    Google Groups | 11 months ago | rajans...@gmail.com
    java.io.IOException: java.util.concurrent.TimeoutException: Timeout waiting for task.
  6. 0

    Oracle Blob to XLS worksheet

    Stack Overflow | 2 years ago
    java.io.IOException: block[ 0 ] already removed

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Failed to lock block. response: RPCBlockReadResponse(1275068416, 0, 0, BLOCK_LOCK_ERROR)

      at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock()
    2. tachyon.client.netty
      NettyRemoteBlockReader.readRemoteBlock
      1. tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:87)
      1 frame
    3. tachyon.client.block
      BufferedBlockInStream.read
      1. tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:94)
      2. tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57)
      3. tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:228)
      4. tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:130)
      4 frames
    4. tachyon.client.file
      FileInStream.read
      1. tachyon.client.file.FileInStream.read(FileInStream.java:149)
      2. tachyon.client.file.FileInStream.read(FileInStream.java:127)
      2 frames
    5. Tachyon Project Core
      BasicOperations.main
      1. tachyon.examples.BasicOperations.readFile(BasicOperations.java:96)
      2. tachyon.examples.BasicOperations.call(BasicOperations.java:58)
      3. tachyon.examples.BasicOperations.call(BasicOperations.java:39)
      4. tachyon.examples.Utils.runExample(Utils.java:102)
      5. tachyon.examples.BasicOperations.main(BasicOperations.java:115)
      5 frames