java.io.IOException: java.util.concurrent.TimeoutException

JIRA | Tiago Cipriano Pires | 10 months ago
  1. 0

    If I have two FileInStream reading from the same file *and* block, when the readType is CACHE_PROMOTE I get a deadlock. I can reproduce this easily with the following program, which will deadlock. If I use different threads or processes to read the files I will have contention - only 1 client can read that block at one time - but it will not deadlock. In my use case I want to read different portions of the same file using two concurrent streams (to be able to join the two blocks of data in a streaming way) and if they happen to share the same block my program deadlocks. This cannot be reproduced if I change the readType from CACHE_PROMOTE to only CACHE. {{public class Test2StreamsNew { final static String FILE_NAME = "/set1/data.bin"; public static void main(String[] args) throws IOException, TachyonException { Stopwatch stopwatch = Stopwatch.createStarted(); TachyonConf conf = ClientContext.getConf(); InStreamOptions options = new InStreamOptions.Builder( conf).setReadType(ReadType.CACHE_PROMOTE).build(); final TachyonURI tachyonURI = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file = tfs.open(tachyonURI); final FileInStream input = tfs.getInStream(file, options); final DataInputStream raf = new DataInputStream(input); long length = raf.readLong(); System.out.println("Read " + length + " value in " + stopwatch.toString()); final TachyonURI tachyonURI2 = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs2 = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file2 = tfs2.open(tachyonURI2); final FileInStream input2 = tfs2.getInStream(file2, options); input2.skip(4); DataInputStream raf2 = new DataInputStream(input2); length = raf2.readLong(); raf.close(); raf2.close(); System.out.println("Read " + length + " value in " + stopwatch.toString()); } } }} Console output: {{ 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Connecting local worker @ /10.14.0.9:29998 Read 254 value in 883.6 ms 16/02/03 11:38:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 WARN : Promotion of block 16777216 failed. 16/02/03 11:39:01 WARN : Worker Client last execution took 29041 ms. Longer than the interval 1000 16/02/03 11:39:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:31 WARN : Failed to open local stream for block 16777216. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out 16/02/03 11:39:31 WARN : Worker Client last execution took 29077 ms. Longer than the interval 1000 16/02/03 11:39:31 WARN : Failed to get TachyonStore stream, the block 16777216 will not be in TachyonStorage. Exception:TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 16,777,216 is not available, because it is already committed) 16/02/03 11:39:31 INFO : Connected to remote machine /10.14.0.9:29999 Exception in thread "main" java.io.IOException: java.util.concurrent.TimeoutException at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143) at tachyon.client.file.FileInStream.read(FileInStream.java:156) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readLong(DataInputStream.java:416) at Test2StreamsNew.main(Test2StreamsNew.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.util.concurrent.TimeoutException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84) at tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72) ... 13 more Process finished with exit code 1 }} According to Calvin Jia this seems like a bug in cache promote. In particular, the promote call immediate tries to take a write lock on the block, which will wait until all readers are done with the block - statement taken from group discussion [https://groups.google.com/forum/#!topic/tachyon-users/_AB5vQRMEBM]

    JIRA | 10 months ago | Tiago Cipriano Pires
    java.io.IOException: java.util.concurrent.TimeoutException
  2. 0

    If I have two FileInStream reading from the same file *and* block, when the readType is CACHE_PROMOTE I get a deadlock. I can reproduce this easily with the following program, which will deadlock. If I use different threads or processes to read the files I will have contention - only 1 client can read that block at one time - but it will not deadlock. In my use case I want to read different portions of the same file using two concurrent streams (to be able to join the two blocks of data in a streaming way) and if they happen to share the same block my program deadlocks. This cannot be reproduced if I change the readType from CACHE_PROMOTE to only CACHE. {{public class Test2StreamsNew { final static String FILE_NAME = "/set1/data.bin"; public static void main(String[] args) throws IOException, TachyonException { Stopwatch stopwatch = Stopwatch.createStarted(); TachyonConf conf = ClientContext.getConf(); InStreamOptions options = new InStreamOptions.Builder( conf).setReadType(ReadType.CACHE_PROMOTE).build(); final TachyonURI tachyonURI = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file = tfs.open(tachyonURI); final FileInStream input = tfs.getInStream(file, options); final DataInputStream raf = new DataInputStream(input); long length = raf.readLong(); System.out.println("Read " + length + " value in " + stopwatch.toString()); final TachyonURI tachyonURI2 = new TachyonURI(FILE_NAME); final TachyonFileSystem tfs2 = TachyonFileSystem.TachyonFileSystemFactory.get(); final TachyonFile file2 = tfs2.open(tachyonURI2); final FileInStream input2 = tfs2.getInStream(file2, options); input2.skip(4); DataInputStream raf2 = new DataInputStream(input2); length = raf2.readLong(); raf.close(); raf2.close(); System.out.println("Read " + length + " value in " + stopwatch.toString()); } } }} Console output: {{ 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with FileSystemMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Tachyon client (version 0.8.2) is trying to connect with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Client registered with BlockMaster master @ /10.14.0.9:19998 16/02/03 11:38:30 INFO : Connecting local worker @ /10.14.0.9:29998 Read 254 value in 883.6 ms 16/02/03 11:38:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:01 WARN : Promotion of block 16777216 failed. 16/02/03 11:39:01 WARN : Worker Client last execution took 29041 ms. Longer than the interval 1000 16/02/03 11:39:31 INFO : Connecting local worker @ /10.14.0.9:29998 16/02/03 11:39:31 WARN : Failed to open local stream for block 16777216. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out 16/02/03 11:39:31 WARN : Worker Client last execution took 29077 ms. Longer than the interval 1000 16/02/03 11:39:31 WARN : Failed to get TachyonStore stream, the block 16777216 will not be in TachyonStorage. Exception:TachyonTException(type:BLOCK_ALREADY_EXISTS, message:Temp blockId 16,777,216 is not available, because it is already committed) 16/02/03 11:39:31 INFO : Connected to remote machine /10.14.0.9:29999 Exception in thread "main" java.io.IOException: java.util.concurrent.TimeoutException at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:96) at tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95) at tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57) at tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242) at tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143) at tachyon.client.file.FileInStream.read(FileInStream.java:156) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readLong(DataInputStream.java:416) at Test2StreamsNew.main(Test2StreamsNew.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.util.concurrent.TimeoutException at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84) at tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61) at tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72) ... 13 more Process finished with exit code 1 }} According to Calvin Jia this seems like a bug in cache promote. In particular, the promote call immediate tries to take a write lock on the block, which will wait until all readers are done with the block - statement taken from group discussion [https://groups.google.com/forum/#!topic/tachyon-users/_AB5vQRMEBM]

    JIRA | 10 months ago | Tiago Cipriano Pires
    java.io.IOException: java.util.concurrent.TimeoutException
  3. 0

    JODConverter converts files when the service is stopped?

    Stack Overflow | 3 years ago | crownjewel82
    java.util.concurrent.TimeoutException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Not able to connect

    GitHub | 3 years ago | dhwanil18r
    java.util.concurrent.TimeoutException
  6. 0

    Crash of phantomjs

    GitHub | 3 years ago | mklynx
    driver.version: PhantomJSDriver

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.util.concurrent.TimeoutException

      No message provided

      at com.google.common.util.concurrent.AbstractFuture.get()
    2. Guava
      AbstractFuture$TrustedFuture.get
      1. com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:411)
      2. com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:84)
      2 frames
    3. tachyon.client.netty
      NettyRemoteBlockReader.readRemoteBlock
      1. tachyon.client.netty.SingleResponseListener.get(SingleResponseListener.java:61)
      2. tachyon.client.netty.NettyRemoteBlockReader.readRemoteBlock(NettyRemoteBlockReader.java:72)
      2 frames
    4. tachyon.client.block
      BufferedBlockInStream.read
      1. tachyon.client.block.RemoteBlockInStream.readFromRemote(RemoteBlockInStream.java:95)
      2. tachyon.client.block.RemoteBlockInStream.bufferedRead(RemoteBlockInStream.java:57)
      3. tachyon.client.block.BufferedBlockInStream.updateBuffer(BufferedBlockInStream.java:242)
      4. tachyon.client.block.BufferedBlockInStream.read(BufferedBlockInStream.java:143)
      4 frames
    5. tachyon.client.file
      FileInStream.read
      1. tachyon.client.file.FileInStream.read(FileInStream.java:156)
      1 frame
    6. Java RT
      DataInputStream.readLong
      1. java.io.DataInputStream.readFully(DataInputStream.java:195)
      2. java.io.DataInputStream.readLong(DataInputStream.java:416)
      2 frames
    7. Unknown
      Test2StreamsNew.main
      1. Test2StreamsNew.main(Test2StreamsNew.java:52)
      1 frame
    8. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:498)
      4 frames
    9. IDEA
      AppMain.main
      1. com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
      1 frame