java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)

Jenkins JIRA | Mike Dillon | 7 years ago
  1. 0

    [JENKINS-4176] Reference counting error for RemotableSVNAuthenticationProviderImpl singleton - Jenkins JIRA

    jenkins-ci.org | 11 months ago
    java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)
  2. 0

    As reported in this thread on the dev list, we have been seeing some odd intermittent behavior since the Channel.unexport fix from issue 4045 was released in Hudson 1.317: http://www.nabble.com/Error-in-SubversionSCM-"Unable-to-call-getCredential"-td24799653.html java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206) at hudson.remoting.UserRequest.perform(UserRequest.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:46) at hudson.remoting.Request$2.run(Request.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) The issue is related to the fact that RemotableSVNAuthenticationProviderImpl is a singleton that is exported multiple times during overlapping builds on the same node. Here is roughly what happens in a two-executor scenario with two SVN-based builds set up to trigger at the same time: 1. Executor #1 calls Entry.addRef as part of the first call to FilePath.act in SubversionSCM (call A; reference count is now 1; proxy A is exported to slave) 2. Executor #0 calls Entry.addRef as part of the first call to FilePath.act in SubversionSCM (call B; reference count is now 2; proxy B is exported to slave) 3. Executor #0 calls Entry.release as part of the first call to FilePath.act in SubversionSCM (call B; reference count is now 1; proxy B is unexported, but still uncollected on slave) 4. Executor #0 calls Entry.addRef as part of the second call to FilePath.act in SubversionSCM (call C; reference count is now 2; proxy C is exported to slave) 5. Executor #1 calls Entry.release as part of the first call to FilePath.act in SubversionSCM (call A; reference count is now 1; proxy A is unexported, but still uncollected on slave) 6. Executor #1 calls Entry.addRef as part of the second call to FilePath.act in SubversionSCM (call D; reference count is now 2; proxy D is exported to slave) 7. GC is triggered on the slave, causing proxy A and proxy B to be collected and triggering UnexportCommand to be sent for each from RemoteInvocationHandler.finalize 8. Channel reader calls Entry.release in response to the first UnexportCommand (reference count is now 1) 9. Channel reader calls Entry.release in response to the second UnexportCommand (reference count is now 0; Entry is removed from the ExportTable, but still in the ExportLists for calls C and D) 10. Slave Executor #0 invokes getCredential, causing an RPCRequest to be sent back to the master with the oid that was just removed, resulting in exception 11. Slave Executor #1 invokes getCredential, causing an RPCRequest to be sent back to the master with the oid that was just removed, resulting in exception 12. Executor #0 calls Entry.release as part of the second call to FilePath.act in SubversionSCM (call C; reference count is now -1) 13. Executor #1 calls Entry.release as part of the second call to FilePath.act in SubversionSCM (call D; reference count is now -2) The reason this is intermittent is that the GC that cleans up the proxies it not predictable. If it doesn't happen until after call C and call D complete, the UnexportCommand's are simply ignored. I just got lucky being able to reproduce it because I noticed that it would happen for the first build right after I disconnected and reconnected the slave. The reason I believe this ended up letting me reproduce it reliably was that the slave VM starts off with a low default heap size and the chances of it needing to GC immediately when the two builds hit it at the same time are much higher. Also, the reason this can't happen in a single executor setup is that the two FilePath.act calls are serialized and the Proxy for the auth provider has a different oid each time. It is only in the case of these closely overlapping calls with an intervening GC on the slave that you see the spurious ref-count decrement happen. I've successfully fixed this in our internal copy of Hudson by changing the RemotableSVNAuthenticationProviderImpl from a true singleton to a per-thread singleton tied to each executor thread. I have a patch for this and will provide it shortly. Also, this issue is not going to be limited to RemotableSVNAuthenticationProviderImpl. Any place where a singleton is multiply exported from overlapping calls could see the same problem. The reason we didn't see this before I fixed issue 4045 was that the Channel.unexport(int) command used to be broken and didn't have an effect on the reference count. This meant that the UnexportCommands sent in step #7 above were effectively ignored by the master.

    Jenkins JIRA | 7 years ago | Mike Dillon
    java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)
  3. 0

    As reported in this thread on the dev list, we have been seeing some odd intermittent behavior since the Channel.unexport fix from issue 4045 was released in Hudson 1.317: http://www.nabble.com/Error-in-SubversionSCM-"Unable-to-call-getCredential"-td24799653.html java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206) at hudson.remoting.UserRequest.perform(UserRequest.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:46) at hudson.remoting.Request$2.run(Request.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) The issue is related to the fact that RemotableSVNAuthenticationProviderImpl is a singleton that is exported multiple times during overlapping builds on the same node. Here is roughly what happens in a two-executor scenario with two SVN-based builds set up to trigger at the same time: 1. Executor #1 calls Entry.addRef as part of the first call to FilePath.act in SubversionSCM (call A; reference count is now 1; proxy A is exported to slave) 2. Executor #0 calls Entry.addRef as part of the first call to FilePath.act in SubversionSCM (call B; reference count is now 2; proxy B is exported to slave) 3. Executor #0 calls Entry.release as part of the first call to FilePath.act in SubversionSCM (call B; reference count is now 1; proxy B is unexported, but still uncollected on slave) 4. Executor #0 calls Entry.addRef as part of the second call to FilePath.act in SubversionSCM (call C; reference count is now 2; proxy C is exported to slave) 5. Executor #1 calls Entry.release as part of the first call to FilePath.act in SubversionSCM (call A; reference count is now 1; proxy A is unexported, but still uncollected on slave) 6. Executor #1 calls Entry.addRef as part of the second call to FilePath.act in SubversionSCM (call D; reference count is now 2; proxy D is exported to slave) 7. GC is triggered on the slave, causing proxy A and proxy B to be collected and triggering UnexportCommand to be sent for each from RemoteInvocationHandler.finalize 8. Channel reader calls Entry.release in response to the first UnexportCommand (reference count is now 1) 9. Channel reader calls Entry.release in response to the second UnexportCommand (reference count is now 0; Entry is removed from the ExportTable, but still in the ExportLists for calls C and D) 10. Slave Executor #0 invokes getCredential, causing an RPCRequest to be sent back to the master with the oid that was just removed, resulting in exception 11. Slave Executor #1 invokes getCredential, causing an RPCRequest to be sent back to the master with the oid that was just removed, resulting in exception 12. Executor #0 calls Entry.release as part of the second call to FilePath.act in SubversionSCM (call C; reference count is now -1) 13. Executor #1 calls Entry.release as part of the second call to FilePath.act in SubversionSCM (call D; reference count is now -2) The reason this is intermittent is that the GC that cleans up the proxies it not predictable. If it doesn't happen until after call C and call D complete, the UnexportCommand's are simply ignored. I just got lucky being able to reproduce it because I noticed that it would happen for the first build right after I disconnected and reconnected the slave. The reason I believe this ended up letting me reproduce it reliably was that the slave VM starts off with a low default heap size and the chances of it needing to GC immediately when the two builds hit it at the same time are much higher. Also, the reason this can't happen in a single executor setup is that the two FilePath.act calls are serialized and the Proxy for the auth provider has a different oid each time. It is only in the case of these closely overlapping calls with an intervening GC on the slave that you see the spurious ref-count decrement happen. I've successfully fixed this in our internal copy of Hudson by changing the RemotableSVNAuthenticationProviderImpl from a true singleton to a per-thread singleton tied to each executor thread. I have a patch for this and will provide it shortly. Also, this issue is not going to be limited to RemotableSVNAuthenticationProviderImpl. Any place where a singleton is multiply exported from overlapping calls could see the same problem. The reason we didn't see this before I fixed issue 4045 was that the Channel.unexport(int) command used to be broken and didn't have an effect on the reference count. This meant that the UnexportCommands sent in step #7 above were effectively ignored by the master.

    Jenkins JIRA | 7 years ago | Mike Dillon
    java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Jenkins issues - [Issue 4176] New - Reference counting error for RemotableSVNAuthenticationProviderImpl singleton

    nabble.com | 11 months ago
    java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)
  6. 0

    When checking out my build, everything works correctly for the first URL and all the source seems to have downloaded just fine. Then the second SVN URL tries to download and I get the stack trace below. Is this a problem with hudson or a problem with my SCM? This only seemed to be introduced once I upgraded hudson past v1.315. FATAL: Unable to call getCredential. Invalid object ID 517 java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 517 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:268) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215) at hudson.remoting.UserRequest.perform(UserRequest.java:103) at hudson.remoting.UserRequest.perform(UserRequest.java:47) at hudson.remoting.Request$2.run(Request.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

    Jenkins JIRA | 7 years ago | clark_malmgren
    java.lang.IllegalStateException: Unable to call getCredential. Invalid object ID 517 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:268) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalStateException

      Unable to call getCredential. Invalid object ID 476 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:259) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:246) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:206)

      at hudson.remoting.UserRequest.perform()
    2. Hudson :: Remoting Layer
      Request$2.run
      1. hudson.remoting.UserRequest.perform(UserRequest.java:92)
      2. hudson.remoting.UserRequest.perform(UserRequest.java:46)
      3. hudson.remoting.Request$2.run(Request.java:236)
      3 frames
    3. Java RT
      FutureTask.run
      1. java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      2. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      3. java.util.concurrent.FutureTask.run(FutureTask.java:138)
      3 frames