org.apache.tez.dag.api.TezUncheckedException: java.lang.RuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Apache's JIRA Issue Tracker | Kurt Muehlner | 8 months ago
  1. 0

    We are experiencing occasional application hangs, when testing an existing Pig MapReduce script, executing on Tez. When this occurs, we find this in the syslog for the executing dag: 016-03-21 16:39:01,643 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e11_1437886552023_169758_01_000822, containerExpiryTime=1458603541415, idleTimeout=5000, taskRequestsCount=0, heldContainers=112, delayedContainers=27, isNew=false 2016-03-21 16:39:01,825 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e11_1437886552023_169758_01_000824, containerExpiryTime=1458603541692, idleTimeout=5000, taskRequestsCount=0, heldContainers=111, delayedContainers=26, isNew=false 2016-03-21 16:39:01,990 [INFO] [Socket Reader #1 for port 53324] |ipc.Server|: Socket Reader #1 for port 53324: readAndProcess from client 10.102.173.86 threw exception [java.io.IOException: Connection reset by peer] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.hadoop.ipc.Server.channelRead(Server.java:2593) at org.apache.hadoop.ipc.Server.access$2800(Server.java:135) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1471) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:762) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:636) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607) 2016-03-21 16:39:02,032 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e11_1437886552023_169758_01_000811, containerExpiryTime=1458603541828, idleTimeout=5000, taskRequestsCount=0, heldContainers=110, delayedContainers=25, isNew=false In all cases I've been able to analyze so far, this also correlates with a warning in the node identified in the IOException: 2016-03-21 16:36:13,641 [WARN] [I/O Setup 2 Initialize: {scope-178}] |retry.RetryInvocationHandler|: A failover has occurred since the start of this method invocation attempt. However, it does not appear that any namenode failover has actually occurred (the most recent failover we see in logs is from 2015). Attached: syslog_dag_1437886552023_169758_3.gz: syslog for the dag which hangs 10.102.173.86.logs.gz: aggregated logs from the host identified in the IOException

    Apache's JIRA Issue Tracker | 8 months ago | Kurt Muehlner
    org.apache.tez.dag.api.TezUncheckedException: java.lang.RuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
  2. 0

    Run Spark App Error - Cloudera Community

    cloudera.com | 11 months ago
    java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs
  3. 0

    Could not find configured network bridge

    Oracle Community | 5 years ago | 897928
    oracle.as.assemblybuilder.wloc.deployer.DeploymentException: Could not start service for the resource pool serverpool in the resource manager DefaultConnection[[
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Could not find configured network bridge

    Oracle Community | 5 years ago | 897928
    oracle.as.assemblybuilder.wloc.deployer.DeploymentException: Could not start service for the resource pool serverpool1 in the resource manager DefaultConnection[[

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.RuntimeException

      Could not find any configured addresses for URI hdfs://stage-stage-hdfs-nameservice

      at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>()
    2. Apache Hadoop HDFS
      ConfiguredFailoverProxyProvider.<init>
      1. org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
      1 frame
    3. Java RT
      Constructor.newInstance
      1. sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      2. sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      3. sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      4. java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      4 frames
    4. Apache Hadoop HDFS
      DistributedFileSystem.initialize
      1. org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:513)
      2. org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:172)
      3. org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:665)
      4. org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:601)
      5. org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
      5 frames
    5. Hadoop
      Path.getFileSystem
      1. org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
      2. org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
      3. org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
      4. org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
      5. org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      6. org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
      7. org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
      8. org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
      8 frames
    6. Hadoop
      FileOutputFormat.setOutputPath
      1. org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:160)
      1 frame
    7. org.apache.pig
      PigOutputFormatTez.getOutputCommitter
      1. org.apache.pig.builtin.PigStorage.setStoreLocation(PigStorage.java:455)
      2. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:175)
      3. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:116)
      4. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:90)
      5. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:71)
      6. org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez$PigOutputCommitterTez.<init>(PigOutputFormatTez.java:80)
      7. org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getOutputCommitter(PigOutputFormatTez.java:55)
      7 frames
    8. org.apache.tez
      VertexImpl$2.run
      1. org.apache.tez.mapreduce.committer.MROutputCommitter.getOutputCommitter(MROutputCommitter.java:137)
      2. org.apache.tez.mapreduce.committer.MROutputCommitter.initialize(MROutputCommitter.java:81)
      3. org.apache.tez.dag.app.dag.impl.VertexImpl$2.run(VertexImpl.java:2227)
      4. org.apache.tez.dag.app.dag.impl.VertexImpl$2.run(VertexImpl.java:2206)
      4 frames
    9. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:415)
      2 frames
    10. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      1 frame
    11. org.apache.tez
      VertexImpl$InitTransition.transition
      1. org.apache.tez.dag.app.dag.impl.VertexImpl.initializeCommitters(VertexImpl.java:2206)
      2. org.apache.tez.dag.app.dag.impl.VertexImpl.initializeVertex(VertexImpl.java:2263)
      3. org.apache.tez.dag.app.dag.impl.VertexImpl.access$3500(VertexImpl.java:204)
      4. org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2782)
      5. org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2675)
      6. org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2657)
      6 frames
    12. hadoop-yarn-common
      StateMachineFactory$InternalStateMachine.doTransition
      1. org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
      2. org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      3. org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      4. org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      4 frames
    13. org.apache.tez
      AsyncDispatcher$1.run
      1. org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
      2. org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1799)
      3. org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
      4. org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2180)
      5. org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2166)
      6. org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
      7. org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
      7 frames
    14. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:744)
      1 frame