hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81

Jenkins JIRA | michal_grzejszczak | 7 years ago
  1. 0

    We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject" I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause. Full stack trace: {code} Started on May 24, 2010 12:00:54 AM FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81 hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81 at hudson.FilePath.act(FilePath.java:743) at hudson.FilePath.act(FilePath.java:729) at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813) at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310) at hudson.scm.SCM.poll(SCM.java:370) at hudson.model.AbstractProject.poll(AbstractProject.java:1153) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Remote call on slave-81 failed at hudson.remoting.Channel.call(Channel.java:560) at hudson.FilePath.act(FilePath.java:736) ... 14 more Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject" at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2291) at java.lang.Class.getDeclaredField(Class.java:1880) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.UserRequest.deserialize(UserRequest.java:178) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) ... 6 more Done. Took 63 ms No changes {code} If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.

    Jenkins JIRA | 7 years ago | michal_grzejszczak
    hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
  2. 0

    We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject" I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause. Full stack trace: {code} Started on May 24, 2010 12:00:54 AM FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81 hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81 at hudson.FilePath.act(FilePath.java:743) at hudson.FilePath.act(FilePath.java:729) at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813) at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310) at hudson.scm.SCM.poll(SCM.java:370) at hudson.model.AbstractProject.poll(AbstractProject.java:1153) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Remote call on slave-81 failed at hudson.remoting.Channel.call(Channel.java:560) at hudson.FilePath.act(FilePath.java:736) ... 14 more Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject" at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2291) at java.lang.Class.getDeclaredField(Class.java:1880) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.UserRequest.deserialize(UserRequest.java:178) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) ... 6 more Done. Took 63 ms No changes {code} If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.

    Jenkins JIRA | 7 years ago | michal_grzejszczak
    hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
  3. 0

    hudson.util.IOException2: remote file operation failed:

    Eclipse Bugzilla | 5 years ago | mtaal
    hudson.util.IOException2: remote file operation failed: /opt/users/hudsonbuild/workspace/emft-texo-nightly at hudson.remoting.Channel@2e22f376:Fastlane
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    "remote" access problems and classcast exceptions on slaves

    Eclipse Bugzilla | 5 years ago | david_williams
    hudson.util.IOException2: remote file operation failed: /opt/users/hudsonbuild/workspace/juno.runAggregator at hudson.remoting.Channel@5f9b5648:hudson-slave1
  6. 0

    stack trace: {code} Last Built Revision: Revision 11c9bcbacb275a3a7b589a9c104d09f1ec6d93c9 (origin/master) hudson.util.IOException2: remote file operation failed: /home/jenkins/jenkins-slave/workspace/helix at hudson.remoting.Channel@5181fa4:ubuntu5 at hudson.FilePath.act(FilePath.java:838) at hudson.FilePath.act(FilePath.java:824) at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:986) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1142) at hudson.model.AbstractProject.checkout(AbstractProject.java:1256) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494) at hudson.model.Run.execute(Run.java:1502) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:477) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) Caused by: java.io.IOException: Remote call on ubuntu5 failed at hudson.remoting.Channel.call(Channel.java:673) at hudson.FilePath.act(FilePath.java:831) ... 11 more Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/Run" at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:634) at java.lang.ClassLoader.defineClass(ClassLoader.java:480) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:152) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2308) at java.lang.Class.getDeclaredField(Class.java:1897) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1624) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:69) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:442) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:430) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:327) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:564) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1600) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1963) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1887) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1963) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1887) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1770) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Retrying after 10 seconds {code}

    Jenkins JIRA | 4 years ago | olamy
    hudson.util.IOException2: remote file operation failed: /home/jenkins/jenkins-slave/workspace/helix at hudson.remoting.Channel@5181fa4:ubuntu5

    19 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.LinkageError

      loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

      at java.lang.ClassLoader.defineClass1()
    2. Java RT
      ClassLoader.defineClass
      1. java.lang.ClassLoader.defineClass1(Native Method)
      2. java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      3. java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      4. java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      4 frames
    3. Hudson :: Remoting Layer
      RemoteClassLoader.findClass
      1. hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      2. hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      2 frames
    4. Java RT
      ClassLoader.defineClass
      1. java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      2. java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      3. java.lang.ClassLoader.defineClass1(Native Method)
      4. java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      5. java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      6. java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      6 frames
    5. Hudson :: Remoting Layer
      RemoteClassLoader.findClass
      1. hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      2. hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      2 frames
    6. Java RT
      ObjectInputStream.readObject
      1. java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      2. java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      3. java.lang.Class.getDeclaredFields0(Native Method)
      4. java.lang.Class.privateGetDeclaredFields(Class.java:2291)
      5. java.lang.Class.getDeclaredField(Class.java:1880)
      6. java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
      7. java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
      8. java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
      9. java.security.AccessController.doPrivileged(Native Method)
      10. java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
      11. java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
      12. java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
      13. java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
      14. java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
      15. java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
      16. java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      17. java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      18. java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      19. java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      20. java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      21. java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      22. java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      23. java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      24. java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      25. java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      25 frames
    7. Hudson :: Remoting Layer
      Request$2.run
      1. hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
      2. hudson.remoting.UserRequest.perform(UserRequest.java:98)
      3. hudson.remoting.UserRequest.perform(UserRequest.java:48)
      4. hudson.remoting.Request$2.run(Request.java:270)
      4 frames
    8. Java RT
      Thread.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      2. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      3. java.util.concurrent.FutureTask.run(FutureTask.java:138)
      4. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      5. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      6. java.lang.Thread.run(Thread.java:619)
      6 frames