java.io.IOException: java.util.concurrent.TimeoutException

JIRA | Tiago Cipriano Pires | 1 year ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    If I have two FileInStream reading from the same file *and* block, when the readType is CACHE_PROMOTE I get a deadlock. I can reproduce this easily with the following program, which will deadlock. If I use different threads or processes to read the files I will have contention - only 1 client can read that block at one time - but it will not deadlock. In my use case I want to read different portions of the same file using two concurrent streams (to be able to join the two blocks of data in a streaming way) and if they happen to share the same block my program deadlocks. This cannot be reproduced if I change the readType from CACHE_PROMOTE to only CACHE. {{public class Test2StreamsNew { final static String FILE_NAME = "/set1/data.bin"; public static void main(String[] args) throws IOException, TachyonException { Stopwatch stopwatch = Stopwatch.createStarted(); TachyonConf conf = ClientContext.getConf(); InStreamOptions options = new InStreamOptions.Builder( conf).setReadType(ReadType.CACHE_PROMOTE).build(); final TachyonURI tachyonURI = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file = tfs.open(tachyonURI); final FileInStream input = tfs.getInStream(file, options); final DataInputStream raf = new DataInputStream(input); long length = raf.readLong(); System.out.println("Read " + length + " value in " + stopwatch.toString()); final TachyonURI tachyonURI2 = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs2 = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file2 = tfs2.open(tachyonURI2); final FileInStream input2 = tfs2.getInStream(file2, options); input2.skip(4); DataInputStream raf2 = new DataInputStream(input2); length = raf2.readLong(); raf.close(); raf2.close(); System.out.println("Read " + length + " value in " + stopwatch.toString()); } } }} Console output: {{ 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Connecting local worker @ /10.14.0.9:29998 Read 254 value in 883.6 ms 16/02/03 11:38:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 WARN : Promotion of block 16777216 failed. 16/02/03 11:39:01 WARN : Worker Client last execution took 29041 ms. Longer than the interval 1000 16/02/03 11:39:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:31 WARN : Failed to open local stream for block 16777216. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out 16/02/03 11:39:31 WARN : Worker Client last execution took 29077 ms. Longer than the interval 1000 16/02/03 11:39:31 WARN : Failed to get TachyonStore stream, the block 16777216 will not be in TachyonStorage. Exception:TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 16,777,216 is not available, because it is already committed) 16/02/03 11:39:31 INFO : Connected to remote machine /10.14.0.9:29999 Exception in thread "main" java.io.IOException: java.util.concurrent.TimeoutException at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143) at tachyon.client.file.FileInStream.read(FileInStream.java:156) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readLong(DataInputStream.java:416) at Test2StreamsNew.main(Test2StreamsNew.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.util.concurrent.TimeoutException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84) at tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72) ... 13 more Process finished with exit code 1 }} According to Calvin Jia this seems like a bug in cache promote. In particular, the promote call immediate tries to take a write lock on the block, which will wait until all readers are done with the block - statement taken from group discussion [https://groups.google.com/forum/#!topic/tachyon-users/_AB5vQRMEBM]

    JIRA | 1 year ago | Tiago Cipriano Pires
    java.io.IOException: java.util.concurrent.TimeoutException
  2. 0

    If I have two FileInStream reading from the same file *and* block, when the readType is CACHE_PROMOTE I get a deadlock. I can reproduce this easily with the following program, which will deadlock. If I use different threads or processes to read the files I will have contention - only 1 client can read that block at one time - but it will not deadlock. In my use case I want to read different portions of the same file using two concurrent streams (to be able to join the two blocks of data in a streaming way) and if they happen to share the same block my program deadlocks. This cannot be reproduced if I change the readType from CACHE_PROMOTE to only CACHE. {{public class Test2StreamsNew { final static String FILE_NAME = "/set1/data.bin"; public static void main(String[] args) throws IOException, TachyonException { Stopwatch stopwatch = Stopwatch.createStarted(); TachyonConf conf = ClientContext.getConf(); InStreamOptions options = new InStreamOptions.Builder( conf).setReadType(ReadType.CACHE_PROMOTE).build(); final TachyonURI tachyonURI = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file = tfs.open(tachyonURI); final FileInStream input = tfs.getInStream(file, options); final DataInputStream raf = new DataInputStream(input); long length = raf.readLong(); System.out.println("Read " + length + " value in " + stopwatch.toString()); final TachyonURI tachyonURI2 = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs2 = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file2 = tfs2.open(tachyonURI2); final FileInStream input2 = tfs2.getInStream(file2, options); input2.skip(4); DataInputStream raf2 = new DataInputStream(input2); length = raf2.readLong(); raf.close(); raf2.close(); System.out.println("Read " + length + " value in " + stopwatch.toString()); } } }} Console output: {{ 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Connecting local worker @ /10.14.0.9:29998 Read 254 value in 883.6 ms 16/02/03 11:38:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 WARN : Promotion of block 16777216 failed. 16/02/03 11:39:01 WARN : Worker Client last execution took 29041 ms. Longer than the interval 1000 16/02/03 11:39:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:31 WARN : Failed to open local stream for block 16777216. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out 16/02/03 11:39:31 WARN : Worker Client last execution took 29077 ms. Longer than the interval 1000 16/02/03 11:39:31 WARN : Failed to get TachyonStore stream, the block 16777216 will not be in TachyonStorage. Exception:TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 16,777,216 is not available, because it is already committed) 16/02/03 11:39:31 INFO : Connected to remote machine /10.14.0.9:29999 Exception in thread "main" java.io.IOException: java.util.concurrent.TimeoutException at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143) at tachyon.client.file.FileInStream.read(FileInStream.java:156) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readLong(DataInputStream.java:416) at Test2StreamsNew.main(Test2StreamsNew.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.util.concurrent.TimeoutException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84) at tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72) ... 13 more Process finished with exit code 1 }} According to Calvin Jia this seems like a bug in cache promote. In particular, the promote call immediate tries to take a write lock on the block, which will wait until all readers are done with the block - statement taken from group discussion [https://groups.google.com/forum/#!topic/tachyon-users/_AB5vQRMEBM]

    JIRA | 1 year ago | Tiago Cipriano Pires
    java.io.IOException: java.util.concurrent.TimeoutException
  3. 0

    GitHub comment 275#264621893

    GitHub | 6 months ago | brcolow
    java.util.concurrent.TimeoutException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.TimeoutException

      No message provided

      at com.google.common.util.concurrent.AbstractFuture.get()
    2. Guava
      AbstractFuture$TrustedFuture.get
      1. com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411)
      2. com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84)
      2 frames
    3. tachyon.client.netty
      NettyRemoteBlockReader.readRemoteBlock
      1. tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61)
      2. tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72)
      2 frames
    4. tachyon.client.block
      BufferedBlockInStream.read
      1. tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95)
      2. tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57)
      3. tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242)
      4. tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143)
      4 frames
    5. tachyon.client.file
      FileInStream.read
      1. tachyon.client.file.FileInStream.read(FileInStream.java:156)
      1 frame
    6. Java RT
      DataInputStream.readLong
      1. java.io.DataInputStream.readFully(DataInputStream.java:195)
      2. java.io.DataInputStream.readLong(DataInputStream.java:416)
      2 frames
    7. Unknown
      Test2StreamsNew.main
      1. Test2StreamsNew.main(Test2StreamsNew.java:52)
      1 frame
    8. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:498)
      4 frames
    9. IDEA
      AppMain.main
      1. com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
      1 frame