java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts

JIRA | David Capwell | 2 years ago
  1. 0

    Periodically, the tests won't be able to create a socket to talk to one another. Since we don't have retry logic, this causes threads to fail and tests to fail. Below is from the logs of running the unit tests. Heartbeat thread dies {code} 2014-10-15 20:52:45,477 INFO (TachyonWorker.java:start) - The worker server ends @ NetAddress(mHost:build4-centos65-gpcloud-w-docker, mPort:37631, mSecondaryPort:59844) 2014-10-15 20:52:45,665 ERROR (HeartbeatThread.java:run) - Uncaught exception in heartbeat executor, Heartbeat Thread shutting down java.lang.RuntimeException: java.io.IOException: org.apache.thrift.transport.TTransportException at com.google.common.base.Throwables.propagate(Throwables.java:160) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:27) at tachyon.HeartbeatThread.run(HeartbeatThread.java:37) Caused by: java.io.IOException: org.apache.thrift.transport.TTransportException at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:404) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:25) ... 1 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.WorkerService$Client.recv_userHeartbeat(WorkerService.java:393) at tachyon.thrift.WorkerService$Client.userHeartbeat(WorkerService.java:380) at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:402 {code} {code} 2014-10-15 20:54:08,701 INFO (TachyonMaster.java:start) - Tachyon Master version 0.6.0-SNAPSHOT ended @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,701 ERROR (MasterClient.java:worker_heartbeat) - java.net.SocketException: Connection reset org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.MasterService$Client.recv_worker_heartbeat(MasterService.java:356) at tachyon.thrift.MasterService$Client.worker_heartbeat(MasterService.java:341) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:736) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 14 more 2014-10-15 20:54:08,703 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,704 ERROR (MasterClient.java:connect) - Failed to connect (1) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:09,705 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:09,706 ERROR (MasterClient.java:connect) - Failed to connect (2) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:10,706 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:10,707 ERROR (MasterClient.java:connect) - Failed to connect (3) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:11,707 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:11,708 ERROR (MasterClient.java:connect) - Failed to connect (4) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:12,708 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:12,709 ERROR (MasterClient.java:connect) - Failed to connect (5) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:13,710 ERROR (TachyonWorker.java:run) - Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts at tachyon.master.MasterClient.connect(MasterClient.java:188) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:733) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) at tachyon.master.MasterClient.connect(MasterClient.java:156) ... 4 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 6 more {code}

    JIRA | 2 years ago | David Capwell
    java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts
  2. 0

    Periodically, the tests won't be able to create a socket to talk to one another. Since we don't have retry logic, this causes threads to fail and tests to fail. Below is from the logs of running the unit tests. Heartbeat thread dies {code} 2014-10-15 20:52:45,477 INFO (TachyonWorker.java:start) - The worker server ends @ NetAddress(mHost:build4-centos65-gpcloud-w-docker, mPort:37631, mSecondaryPort:59844) 2014-10-15 20:52:45,665 ERROR (HeartbeatThread.java:run) - Uncaught exception in heartbeat executor, Heartbeat Thread shutting down java.lang.RuntimeException: java.io.IOException: org.apache.thrift.transport.TTransportException at com.google.common.base.Throwables.propagate(Throwables.java:160) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:27) at tachyon.HeartbeatThread.run(HeartbeatThread.java:37) Caused by: java.io.IOException: org.apache.thrift.transport.TTransportException at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:404) at tachyon.worker.WorkerClientHeartbeatExecutor.heartbeat(WorkerClientHeartbeatExecutor.java:25) ... 1 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.WorkerService$Client.recv_userHeartbeat(WorkerService.java:393) at tachyon.thrift.WorkerService$Client.userHeartbeat(WorkerService.java:380) at tachyon.worker.WorkerClient.userHeartbeat(WorkerClient.java:402 {code} {code} 2014-10-15 20:54:08,701 INFO (TachyonMaster.java:start) - Tachyon Master version 0.6.0-SNAPSHOT ended @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,701 ERROR (MasterClient.java:worker_heartbeat) - java.net.SocketException: Connection reset org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at tachyon.thrift.MasterService$Client.recv_worker_heartbeat(MasterService.java:356) at tachyon.thrift.MasterService$Client.worker_heartbeat(MasterService.java:341) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:736) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 14 more 2014-10-15 20:54:08,703 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:08,704 ERROR (MasterClient.java:connect) - Failed to connect (1) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:09,705 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:09,706 ERROR (MasterClient.java:connect) - Failed to connect (2) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:10,706 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:10,707 ERROR (MasterClient.java:connect) - Failed to connect (3) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:11,707 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:11,708 ERROR (MasterClient.java:connect) - Failed to connect (4) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:12,708 INFO (MasterClient.java:connect) - Tachyon client (version 0.6.0-SNAPSHOT) is trying to connect master @ build4-centos65-gpcloud-w-docker/172.17.0.5:52447 2014-10-15 20:54:12,709 ERROR (MasterClient.java:connect) - Failed to connect (5) to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 : java.net.ConnectException: Connection refused 2014-10-15 20:54:13,710 ERROR (TachyonWorker.java:run) - Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts at tachyon.master.MasterClient.connect(MasterClient.java:188) at tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:733) at tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599) at tachyon.worker.TachyonWorker.run(TachyonWorker.java:245) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) at tachyon.master.MasterClient.connect(MasterClient.java:156) ... 4 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 6 more {code}

    JIRA | 2 years ago | David Capwell
    java.io.IOException: Failed to connect to master build4-centos65-gpcloud-w-docker/172.17.0.5:52447 after 5 attempts
  3. 0

    Issue regarding YCSB client and cassandra server

    GitHub | 3 years ago | sneha1012
    com.yahoo.ycsb.DBException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Cassandra Refusing Connection

    Stack Overflow | 2 years ago
    org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
  6. 0

    Titan - Cassandra - Elasticsearch :: Connection refused

    Stack Overflow | 2 years ago | Antoine Wils
    es.xml: <base-uri>http://my-domain.com</base-uri> <index.search.hostname>192.168.1.20</index.search.hostname> Stacktrace java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager

  1. treefolk 1 times, last 1 week ago
  2. kjhdofjosvs 20 times, last 1 week ago
  3. Andreas Häber 9 times, last 2 weeks ago
  4. Aria Pahlavan 2 times, last 2 weeks ago
  5. stigkj 10 times, last 2 weeks ago
44 more registered users
51 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.net.ConnectException

    Connection refused

    at java.net.PlainSocketImpl.socketConnect()
  2. Java RT
    Socket.connect
    1. java.net.PlainSocketImpl.socketConnect(Native Method)
    2. java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    3. java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    4. java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    5. java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    6. java.net.Socket.connect(Socket.java:579)
    6 frames
  3. Apache Thrift
    TFramedTransport.open
    1. org.apache.thrift.transport.TSocket.open(TSocket.java:180)
    2. org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
    2 frames
  4. Tachyon Project Core
    TachyonWorker.run
    1. tachyon.master.MasterClient.connect(MasterClient.java:156)
    2. tachyon.master.MasterClient.worker_heartbeat(MasterClient.java:733)
    3. tachyon.worker.WorkerStorage.heartbeat(WorkerStorage.java:599)
    4. tachyon.worker.TachyonWorker.run(TachyonWorker.java:245)
    4 frames
  5. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:745)
    1 frame