org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

JIRA | David Capwell | 2 years ago
  1. 0

    Periodically, the tests won't be able to create a socket to talk to one another. Since we don't have retry logic, this causes threads to fail and tests to fail. Below is from the logs of running the unit tests. Heartbeat thread dies {code} 2014-10-15 20:52:45,477 INFO (TachyonWorker.java:start) - The worker server ends @ NetAddress(mHost:build4-centos65-gpcloud-w-docker, mPort:37631, mSecondaryPort:59844) 2014-10-15 20:52:45,665 ERROR (HeartbeatThread.java:run) - Uncaught exception in heartbeat executor, Heartbeat Thread shutting down java.lang.RuntimeException: java.io.IOException: org.apache.thrift.transport.TTransportException at com.google.common.base.Throwables.propagate(Throwables.java:160) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:27) at tachyon.HeartbeatThread.run(HeartbeatThread.java:37) Caused by: java.io.IOException: org.apache.thrift.transport.TTransportException at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:404) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:25) ... 1 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.WorkerService$Client.recv_userHeartbeat(WorkerService.java:393) at tachyon.thrift.WorkerService$Client.userHeartbeat(WorkerService.java:380) at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:402 {code} {code} 2014-10-15 20:54:08,701 INFO (TachyonMaster.java:start) - Tachyon Master version 0.6.0-SNAPSHOT ended @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,701 ERROR (MasterClient.java:worker_heartbeat) - java.net.SocketException: Connection reset org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.MasterService$Client.recv_worker_heartbeat(MasterService.java:356) at tachyon.thrift.MasterService$Client.worker_heartbeat(MasterService.java:341) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:736) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 14 more 2014-10-15 20:54:08,703 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,704 ERROR (MasterClient.java:connect) - Failed to connect (1) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:09,705 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:09,706 ERROR (MasterClient.java:connect) - Failed to connect (2) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:10,706 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:10,707 ERROR (MasterClient.java:connect) - Failed to connect (3) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:11,707 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:11,708 ERROR (MasterClient.java:connect) - Failed to connect (4) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:12,708 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:12,709 ERROR (MasterClient.java:connect) - Failed to connect (5) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:13,710 ERROR (TachyonWorker.java:run) - Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts at tachyon.master.MasterClient.connect(MasterClient.java:188) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:733) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) at tachyon.master.MasterClient.connect(MasterClient.java:156) ... 4 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 6 more {code}

    JIRA | 2 years ago | David Capwell
    org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
  2. 0

    Periodically, the tests won't be able to create a socket to talk to one another. Since we don't have retry logic, this causes threads to fail and tests to fail. Below is from the logs of running the unit tests. Heartbeat thread dies {code} 2014-10-15 20:52:45,477 INFO (TachyonWorker.java:start) - The worker server ends @ NetAddress(mHost:build4-centos65-gpcloud-w-docker, mPort:37631, mSecondaryPort:59844) 2014-10-15 20:52:45,665 ERROR (HeartbeatThread.java:run) - Uncaught exception in heartbeat executor, Heartbeat Thread shutting down java.lang.RuntimeException: java.io.IOException: org.apache.thrift.transport.TTransportException at com.google.common.base.Throwables.propagate(Throwables.java:160) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:27) at tachyon.HeartbeatThread.run(HeartbeatThread.java:37) Caused by: java.io.IOException: org.apache.thrift.transport.TTransportException at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:404) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:25) ... 1 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.WorkerService$Client.recv_userHeartbeat(WorkerService.java:393) at tachyon.thrift.WorkerService$Client.userHeartbeat(WorkerService.java:380) at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:402 {code} {code} 2014-10-15 20:54:08,701 INFO (TachyonMaster.java:start) - Tachyon Master version 0.6.0-SNAPSHOT ended @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,701 ERROR (MasterClient.java:worker_heartbeat) - java.net.SocketException: Connection reset org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.MasterService$Client.recv_worker_heartbeat(MasterService.java:356) at tachyon.thrift.MasterService$Client.worker_heartbeat(MasterService.java:341) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:736) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 14 more 2014-10-15 20:54:08,703 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,704 ERROR (MasterClient.java:connect) - Failed to connect (1) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:09,705 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:09,706 ERROR (MasterClient.java:connect) - Failed to connect (2) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:10,706 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:10,707 ERROR (MasterClient.java:connect) - Failed to connect (3) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:11,707 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:11,708 ERROR (MasterClient.java:connect) - Failed to connect (4) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:12,708 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:12,709 ERROR (MasterClient.java:connect) - Failed to connect (5) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:13,710 ERROR (TachyonWorker.java:run) - Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts at tachyon.master.MasterClient.connect(MasterClient.java:188) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:733) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) at tachyon.master.MasterClient.connect(MasterClient.java:156) ... 4 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 6 more {code}

    JIRA | 2 years ago | David Capwell
    org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
  3. 0

    Hive metastore not working in HDInsight 3.0

    microsoft.com | 8 months ago
    org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hive metastore not working in HDInsight 3.0

    microsoft.com | 11 months ago
    org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
  6. 0

    TTransportException when using Spark 1.6.0 on top of Tachyon 0.8.2

    Google Groups | 10 months ago | Jia Zou
    org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

  1. Andreas Häber 4 times, last 2 months ago
  2. sriharshakiran 1 times, last 7 months ago
14 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.net.SocketException

    Connection reset

    at java.net.SocketInputStream.read()
  2. Java RT
    BufferedInputStream.read
    1. java.net.SocketInputStream.read(SocketInputStream.java:196)
    2. java.net.SocketInputStream.read(SocketInputStream.java:122)
    3. java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    4. java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    5. java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    5 frames
  3. Apache Thrift
    TServiceClient.receiveBase
    1. org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    2. org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    3. org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    4. org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    5. org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    6. org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    7. org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    8. org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    9. org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    9 frames
  4. Tachyon Project Core
    TachyonWorker.run
    1. tachyon.thrift.MasterService$Client.recv_worker_heartbeat(MasterService.java:356)
    2. tachyon.thrift.MasterService$Client.worker_heartbeat(MasterService.java:341)
    3. tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:736)
    4. tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599)
    5. tachyon.worker.TachyonWorker.run(TachyonWorker.java:245)
    5 frames
  5. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:745)
    1 frame