java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "sdevin-clster53/10.215.16.72"; destination host is: "sdevin-clster54":63415;

Apache's JIRA Issue Tracker | Zhijie Shen | 2 years ago
  1. 0

    [HADOOP-10718] "IOException: An existing connection was forcibly closed by the remote host" frequently happens on Windows - ASF JIRA

    apache.org | 11 months ago
    org.apache.hadoop.mapred.Task: Failure sending commit pending: java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: ; destination host is: :63415;
  2. 0

    After HADOOP-317, we still observed that on windows platform, there're a number of IOException: An existing connection was forcibly closed by the remote host when running a MR job. For example, {code} 2014-06-09 09:11:40,675 INFO [Socket Reader #3 for port 59622] org.apache.hadoop.ipc.Server: Socket Reader #3 for port 59622: readAndProcess from client 10.215.30.53 threw exception [java.io.IOException: An existing connection was forcibly closed by the remote host] java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) at org.apache.hadoop.ipc.Server.channelRead(Server.java:2558) at org.apache.hadoop.ipc.Server.access$2800(Server.java:130) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1459) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:750) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:624) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595) {code} {code} 2014-06-09 09:15:38,539 WARN [main] org.apache.hadoop.mapred.Task: Failure sending commit pending: java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "sdevin-clster53/10.215.16.72"; destination host is: "sdevin-clster54":63415; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231) at com.sun.proxy.$Proxy9.commitPending(Unknown Source) at org.apache.hadoop.mapred.Task.done(Task.java:1006) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:397) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:510) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1054) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) {code} And the latter one results in the issue of MAPREDUCE-5924.

    Apache's JIRA Issue Tracker | 2 years ago | Zhijie Shen
    org.apache.hadoop.mapred.Task: Failure sending commit pending: java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "sdevin-clster53/10.215.16.72"; destination host is: "sdevin-clster54":63415;
  3. 0

    After HADOOP-317, we still observed that on windows platform, there're a number of IOException: An existing connection was forcibly closed by the remote host when running a MR job. For example, {code} 2014-06-09 09:11:40,675 INFO [Socket Reader #3 for port 59622] org.apache.hadoop.ipc.Server: Socket Reader #3 for port 59622: readAndProcess from client 10.215.30.53 threw exception [java.io.IOException: An existing connection was forcibly closed by the remote host] java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) at org.apache.hadoop.ipc.Server.channelRead(Server.java:2558) at org.apache.hadoop.ipc.Server.access$2800(Server.java:130) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1459) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:750) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:624) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595) {code} {code} 2014-06-09 09:15:38,539 WARN [main] org.apache.hadoop.mapred.Task: Failure sending commit pending: java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "sdevin-clster53/10.215.16.72"; destination host is: "sdevin-clster54":63415; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231) at com.sun.proxy.$Proxy9.commitPending(Unknown Source) at org.apache.hadoop.mapred.Task.done(Task.java:1006) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:397) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:510) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1054) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) {code} And the latter one results in the issue of MAPREDUCE-5924.

    Apache's JIRA Issue Tracker | 2 years ago | Zhijie Shen
    java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "sdevin-clster53/10.215.16.72"; destination host is: "sdevin-clster54":63415;
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hadoop: Task Tracker won't start java.io.IOException

    Stack Overflow | 5 years ago | CatShoes
    java.io.IOException: Connection reset by peer

  1. Raggyman 2 times, last 2 months ago
  2. bpbhat77 4 times, last 3 months ago
  3. kid 28 times, last 3 months ago
  4. Piz 4 times, last 4 months ago
  5. poroszd 1 times, last 5 months ago
7 more registered users
88 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.IOException

    An existing connection was forcibly closed by the remote host

    at sun.nio.ch.SocketDispatcher.read0()
  2. Java RT
    SocketChannelImpl.read
    1. sun.nio.ch.SocketDispatcher.read0(Native Method)
    2. sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
    3. sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
    4. sun.nio.ch.IOUtil.read(IOUtil.java:198)
    5. sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
    5 frames
  3. Hadoop
    SocketInputStream.read
    1. org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    2. org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    3. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    4. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    4 frames
  4. Java RT
    FilterInputStream.read
    1. java.io.FilterInputStream.read(FilterInputStream.java:133)
    2. java.io.FilterInputStream.read(FilterInputStream.java:133)
    2 frames
  5. Hadoop
    Client$Connection$PingInputStream.read
    1. org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:510)
    1 frame
  6. Java RT
    DataInputStream.readInt
    1. java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    2. java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    3. java.io.DataInputStream.readInt(DataInputStream.java:387)
    3 frames
  7. Hadoop
    Client$Connection.run
    1. org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1054)
    2. org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
    2 frames