java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/***-**********-config-test at hudson.remoting.Channel@236ff568:WSCPSLRDREAPP15

Jenkins JIRA | David Hoffman | 2 years ago
  1. 0

    I have a situation where I am running out of permgen space on a regular basis. I can literally what the permgen use grow until the server falls over. I have seen numerous other ticket that point toward this actually being a plugin issue, and that the only way to troubleshoot this is with a Memory Dump. I am of course able to increase the amount of PermGen, but all this does is delay the inevitable. No matter what levels I set permgen to, it eventually falls over. I have attached a memory dump. WARNING: Disk usage plugin fails during build calculation disk space of job ***-**********-config-test java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/***-**********-config-test at hudson.remoting.Channel@236ff568:WSCPSLRDREAPP15 at hudson.FilePath.act(FilePath.java:916) at hudson.FilePath.act(FilePath.java:893) at hudson.FilePath.exists(FilePath.java:1325) at hudson.plugins.disk_usage.DiskUsageUtil.calculateWorkspaceDiskUsageForPath(DiskUsageUtil.java:292) at hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:60) at hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:23) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:199) at hudson.model.Run.execute(Run.java:1783) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) Caused by: java.io.IOException: Remote call on WSCPSLRDREAPP15 failed at hudson.remoting.Channel.call(Channel.java:748) at hudson.FilePath.act(FilePath.java:909) ... 10 more Caused by: java.lang.OutOfMemoryError: PermGen space

    Jenkins JIRA | 2 years ago | David Hoffman
    java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/***-**********-config-test at hudson.remoting.Channel@236ff568:WSCPSLRDREAPP15
  2. 0

    I have a situation where I am running out of permgen space on a regular basis. I can literally what the permgen use grow until the server falls over. I have seen numerous other ticket that point toward this actually being a plugin issue, and that the only way to troubleshoot this is with a Memory Dump. I am of course able to increase the amount of PermGen, but all this does is delay the inevitable. No matter what levels I set permgen to, it eventually falls over. I have attached a memory dump. WARNING: Disk usage plugin fails during build calculation disk space of job ***-**********-config-test java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/***-**********-config-test at hudson.remoting.Channel@236ff568:WSCPSLRDREAPP15 at hudson.FilePath.act(FilePath.java:916) at hudson.FilePath.act(FilePath.java:893) at hudson.FilePath.exists(FilePath.java:1325) at hudson.plugins.disk_usage.DiskUsageUtil.calculateWorkspaceDiskUsageForPath(DiskUsageUtil.java:292) at hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:60) at hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:23) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:199) at hudson.model.Run.execute(Run.java:1783) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:234) Caused by: java.io.IOException: Remote call on WSCPSLRDREAPP15 failed at hudson.remoting.Channel.call(Channel.java:748) at hudson.FilePath.act(FilePath.java:909) ... 10 more Caused by: java.lang.OutOfMemoryError: PermGen space

    Jenkins JIRA | 2 years ago | David Hoffman
    java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/***-**********-config-test at hudson.remoting.Channel@236ff568:WSCPSLRDREAPP15
  3. 0

    We recently converted our Windows slaves from using JNLP service to being logged in via ssh. At that time, several of our CTest-running jobs started failing with the stack trace attached. Note the reference to "/var/lib/jenkins/userContent/xunit/CTest" in the output - that path clearly doesn't exist on our Windows slaves. I don't know that that is the cause of the problem, of course, but it seems potentially relevant. Perhaps the xUnit plugin is somehow assuming that the remote side is Unix since it is logged into via ssh? [xUnit] [INFO] - Starting to record. [xUnit] [INFO] - Processing CTest-Version N/A (default) [xUnit] [INFO] - [CTest-Version N/A (default)] - 1 test report file(s) were found with the pattern 'build/memcached/Testing/**/Test.xml' relative to 'C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows' for the testing framework 'CTest-Version N/A (default)'. [xUnit] [INFO] - Converting 'C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows\build\memcached\Testing\20160112-0155\Test.xml' . [xUnit] [ERROR] - Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception java.io.IOException: remote file operation failed: C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows at hudson.remoting.Channel@34a40002:watson-windows-01 at hudson.FilePath.act(FilePath.java:910) at hudson.FilePath.act(FilePath.java:887) at org.jenkinsci.plugins.xunit.XUnitProcessor.performTests(XUnitProcessor.java:138) at org.jenkinsci.plugins.xunit.XUnitProcessor.performXUnit(XUnitProcessor.java:81) at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:112) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:725) at hudson.model.Run.execute(Run.java:1701) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: hudson.util.IOException2: There are some problems during the conversion into JUnit reports: at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:181) at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:42) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2461) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.jenkinsci.plugins.xunit.exception.XUnitException: Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:91) at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:144) ... 10 more Caused by: java.io.IOException: remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel at hudson.FilePath.act(FilePath.java:910) at hudson.FilePath.act(FilePath.java:887) at hudson.FilePath.exists(FilePath.java:1319) at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convertInputMetricXSL(XUnitConversionService.java:106) at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:80) ... 11 more Caused by: java.io.IOException: Remote call on channel failed at hudson.remoting.Channel.call(Channel.java:731) at hudson.FilePath.act(FilePath.java:903) ... 15 more Caused by: java.lang.NoClassDefFoundError: hudson/remoting/RemoteClassLoader$RemoteIClassLoader at hudson.remoting.RemoteClassLoader.export(RemoteClassLoader.java:656) at hudson.remoting.UserRequest.<init>(UserRequest.java:69) at hudson.remoting.Channel.call(Channel.java:721) ... 16 more

    Jenkins JIRA | 11 months ago | Chris Hillery
    java.io.IOException: remote file operation failed: C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows at hudson.remoting.Channel@34a40002:watson-windows-01
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    We recently converted our Windows slaves from using JNLP service to being logged in via ssh. At that time, several of our CTest-running jobs started failing with the stack trace attached. Note the reference to "/var/lib/jenkins/userContent/xunit/CTest" in the output - that path clearly doesn't exist on our Windows slaves. I don't know that that is the cause of the problem, of course, but it seems potentially relevant. Perhaps the xUnit plugin is somehow assuming that the remote side is Unix since it is logged into via ssh? [xUnit] [INFO] - Starting to record. [xUnit] [INFO] - Processing CTest-Version N/A (default) [xUnit] [INFO] - [CTest-Version N/A (default)] - 1 test report file(s) were found with the pattern 'build/memcached/Testing/**/Test.xml' relative to 'C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows' for the testing framework 'CTest-Version N/A (default)'. [xUnit] [INFO] - Converting 'C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows\build\memcached\Testing\20160112-0155\Test.xml' . [xUnit] [ERROR] - Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception java.io.IOException: remote file operation failed: C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows at hudson.remoting.Channel@34a40002:watson-windows-01 at hudson.FilePath.act(FilePath.java:910) at hudson.FilePath.act(FilePath.java:887) at org.jenkinsci.plugins.xunit.XUnitProcessor.performTests(XUnitProcessor.java:138) at org.jenkinsci.plugins.xunit.XUnitProcessor.performXUnit(XUnitProcessor.java:81) at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:112) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:725) at hudson.model.Run.execute(Run.java:1701) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: hudson.util.IOException2: There are some problems during the conversion into JUnit reports: at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:181) at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:42) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2461) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.jenkinsci.plugins.xunit.exception.XUnitException: Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:91) at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:144) ... 10 more Caused by: java.io.IOException: remote file operation failed: /var/lib/jenkins/userContent/xunit/CTest at hudson.remoting.Channel@b07fd3:channel at hudson.FilePath.act(FilePath.java:910) at hudson.FilePath.act(FilePath.java:887) at hudson.FilePath.exists(FilePath.java:1319) at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convertInputMetricXSL(XUnitConversionService.java:106) at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:80) ... 11 more Caused by: java.io.IOException: Remote call on channel failed at hudson.remoting.Channel.call(Channel.java:731) at hudson.FilePath.act(FilePath.java:903) ... 15 more Caused by: java.lang.NoClassDefFoundError: hudson/remoting/RemoteClassLoader$RemoteIClassLoader at hudson.remoting.RemoteClassLoader.export(RemoteClassLoader.java:656) at hudson.remoting.UserRequest.<init>(UserRequest.java:69) at hudson.remoting.Channel.call(Channel.java:721) ... 16 more

    Jenkins JIRA | 11 months ago | Chris Hillery
    java.io.IOException: remote file operation failed: C:\Jenkins\workspace\memcached-gerrit-master\label_exp\watson-windows at hudson.remoting.Channel@34a40002:watson-windows-01

    7 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      Remote call on WSCPSLRDREAPP15 failed

      at hudson.remoting.Channel.call()
    2. Hudson :: Remoting Layer
      Channel.call
      1. hudson.remoting.Channel.call(Channel.java:748)
      1 frame
    3. Hudson
      FilePath.exists
      1. hudson.FilePath.act(FilePath.java:909)
      2. hudson.FilePath.act(FilePath.java:893)
      3. hudson.FilePath.exists(FilePath.java:1325)
      3 frames
    4. hudson.plugins.disk_usage
      DiskUsageBuildListener.onCompleted
      1. hudson.plugins.disk_usage.DiskUsageUtil.calculateWorkspaceDiskUsageForPath(DiskUsageUtil.java:292)
      2. hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:60)
      3. hudson.plugins.disk_usage.DiskUsageBuildListener.onCompleted(DiskUsageBuildListener.java:23)
      3 frames
    5. Hudson
      Run.execute
      1. hudson.model.listeners.RunListener.fireCompleted(RunListener.java:199)
      2. hudson.model.Run.execute(Run.java:1783)
      2 frames
    6. Hudson Maven Embedder
      MavenModuleSetBuild.run
      1. hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
      1 frame
    7. Hudson
      Executor.run
      1. hudson.model.ResourceController.execute(ResourceController.java:88)
      2. hudson.model.Executor.run(Executor.java:234)
      2 frames