java.lang.UnsatisfiedLinkError

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • On some Ubuntu configurations, an attempt to start Puppet Server 1.0.8 or the Puppet Server in PE 3.8 could fail with messages like the following appearing in the {{puppetserver-daemon.log}}: {noformat} Failed to load feature test for posix: can't find user for 0 Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32 Puppet::Error: Cannot determine basic system flavour {noformat} One reason for this error occurring could be lack of sufficient permissions for Puppet Server to read, write, and execute from the system tmpdir. See SERVER-160. In this case, however, the problem was due to one of the JRuby dependencies of Puppet Server being unable to locate and load the "libcrypt.so" shared library. Puppet Server 1.0.8 depends upon JRuby version 1.7.19. JRuby versions 1.7.17 and later depend upon jnr-posix versions 3.0.8 or later. The later jnr-posix version has support for "crypto" functionality, for which "libcrypt.so" must be loaded by the Java process. In an Ubuntu 14.04 distribution, "libcrypt.so" commonly resides at {{/lib/x86_64-linux-gnu}}. We've found in some cases, however, that the default {{LD_LIBRARY_PATH}} does not include the {{/lib/x86_64-linux-gnu}} directory, the {{libcrypt.so}} shared library fails to be loaded, and Puppet Server ultimately fails to start with the non-intuitive error messages above. Note from the following thread that other direct users of JRuby have encountered this same problem - https://www.ruby-forum.com/topic/6563521. On an Ubuntu amd64 Amazon AMI, we found that the {{LD_LIBRARY_PATH}} was set by default to "/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib". We were able to get the Puppet Server to start by just setting that same path but with the missing "/lib/x86_64-linux-gnu" appended to end of the {{JAVA_ARGS}} in Puppet Server's {{/etc/default/puppetserver}} and restarting the service: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/lib/x86_64-linux-gnu" {noformat} Management of the {{JAVA_ARGS}} setting in PE should be done through the PE module. A more appropriate workaround with instructions for PE users is documented here - http://docs.puppetlabs.com/pe/latest/release_notes_known_issues.html#ubuntu-conflict-when-youre-running-on-fusion-vm-and-amazon-ec2. --- The root cause for this issue was that the {{pe-java}} JDK distribution in Puppet Enterprise, unlike its 1.7.0.75 OpenJDK siblings, did not include the {{/lib/x86_64-linux-gnu}} path in its default library path. Work to rectify this will be tracked in RE-4557. This ticket should only cover a couple of minor updates to Puppet Server to help more easily troubleshoot issues like these in the future. 1) Add a couple of debug log statements to Puppet Server code to output the value of the {{java.library.path}} system property and {{LD_LIBRARY_PATH}} environment variable. Having this available in Puppet Server without needing to modify source code would have been helpful for debugging this issue more easily. [~chris] discussed doing this in the {{init}} function of the {{jruby-puppet-pooled-service}}. 2) Improve documentation on how to troubleshoot a problem like this. The error messages above are not helpful for diagnosing the source of this problem. JRuby has a property, "jruby.native.verbose", that you can set to get more verbose output for native library invocations. With an Ubuntu AMI in the failing state noted in this ticket, one could have set the JAVA_ARGS to: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djruby.native.verbose=true" {noformat} If that change had been made prior to starting the service, the following messages would have appeared in the "puppetserver-daemon.log" file: {noformat} Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87) ... {noformat} This would tell you what library couldn't be loaded and where it was being loaded from - which is really helpful for this case. At the very least, we should reference the use of this flag in Puppet Server documentation. h3. QA ---- Risk assessment: high (QA to perform functional review and write an acceptance test) Risk probability: medium (seems intermittent, and we missed in in CI) Risk severity: high (puppet-server does not start) Test layer: acceptance
    via by Jeremy Barlow,
  • On some Ubuntu configurations, an attempt to start Puppet Server 1.0.8 or the Puppet Server in PE 3.8 could fail with messages like the following appearing in the {{puppetserver-daemon.log}}: {noformat} Failed to load feature test for posix: can't find user for 0 Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32 Puppet::Error: Cannot determine basic system flavour {noformat} One reason for this error occurring could be lack of sufficient permissions for Puppet Server to read, write, and execute from the system tmpdir. See SERVER-160. In this case, however, the problem was due to one of the JRuby dependencies of Puppet Server being unable to locate and load the "libcrypt.so" shared library. Puppet Server 1.0.8 depends upon JRuby version 1.7.19. JRuby versions 1.7.17 and later depend upon jnr-posix versions 3.0.8 or later. The later jnr-posix version has support for "crypto" functionality, for which "libcrypt.so" must be loaded by the Java process. In an Ubuntu 14.04 distribution, "libcrypt.so" commonly resides at {{/lib/x86_64-linux-gnu}}. We've found in some cases, however, that the default {{LD_LIBRARY_PATH}} does not include the {{/lib/x86_64-linux-gnu}} directory, the {{libcrypt.so}} shared library fails to be loaded, and Puppet Server ultimately fails to start with the non-intuitive error messages above. Note from the following thread that other direct users of JRuby have encountered this same problem - https://www.ruby-forum.com/topic/6563521. On an Ubuntu amd64 Amazon AMI, we found that the {{LD_LIBRARY_PATH}} was set by default to "/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib". We were able to get the Puppet Server to start by just setting that same path but with the missing "/lib/x86_64-linux-gnu" appended to end of the {{JAVA_ARGS}} in Puppet Server's {{/etc/default/puppetserver}} and restarting the service: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/lib/x86_64-linux-gnu" {noformat} Management of the {{JAVA_ARGS}} setting in PE should be done through the PE module. A more appropriate workaround with instructions for PE users is documented here - http://docs.puppetlabs.com/pe/latest/release_notes_known_issues.html#ubuntu-conflict-when-youre-running-on-fusion-vm-and-amazon-ec2. --- The root cause for this issue was that the {{pe-java}} JDK distribution in Puppet Enterprise, unlike its 1.7.0.75 OpenJDK siblings, did not include the {{/lib/x86_64-linux-gnu}} path in its default library path. Work to rectify this will be tracked in RE-4557. This ticket should only cover a couple of minor updates to Puppet Server to help more easily troubleshoot issues like these in the future. 1) Add a couple of debug log statements to Puppet Server code to output the value of the {{java.library.path}} system property and {{LD_LIBRARY_PATH}} environment variable. Having this available in Puppet Server without needing to modify source code would have been helpful for debugging this issue more easily. [~chris] discussed doing this in the {{init}} function of the {{jruby-puppet-pooled-service}}. 2) Improve documentation on how to troubleshoot a problem like this. The error messages above are not helpful for diagnosing the source of this problem. JRuby has a property, "jruby.native.verbose", that you can set to get more verbose output for native library invocations. With an Ubuntu AMI in the failing state noted in this ticket, one could have set the JAVA_ARGS to: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djruby.native.verbose=true" {noformat} If that change had been made prior to starting the service, the following messages would have appeared in the "puppetserver-daemon.log" file: {noformat} Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87) ... {noformat} This would tell you what library couldn't be loaded and where it was being loaded from - which is really helpful for this case. At the very least, we should reference the use of this flag in Puppet Server documentation. h3. QA ---- Risk assessment: high (QA to perform functional review and write an acceptance test) Risk probability: medium (seems intermittent, and we missed in in CI) Risk severity: high (puppet-server does not start) Test layer: acceptance
    via by Jeremy Barlow,
  • GitHub comment 3266#132630421
    via GitHub by pinya
    ,
  • GitHub comment 1#144228446
    via GitHub by btiernay
    ,
  • GitHub comment 3380#148988666
    via GitHub by mkristian
    ,
  • GitHub comment 3409#150225537
    via GitHub by Slicertje
    ,
  • GitHub comment 3409#150499165
    via GitHub by akkie
    ,
  • GitHub comment 3409#154031386
    via GitHub by richtmat
    ,
  • when starting an ssh agent on the slave the following stack trace occurs. Downgrading to 1.3 resolves the issue. Started by user Mark Building remotely on EC2-Jenkins-Slave (i-5a34613e) in workspace /var/jenkins/workspace/Mirror_Global_Commerce [ssh-agent] Using credentials jenkinsadmin [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] FATAL: Could not find a suitable ssh-agent provider [ssh-agent] Diagnostic report [ssh-agent] * Java/JNR ssh-agent [ssh-agent] java.io.IOException: Remote call on EC2-Jenkins-Slave (i-5a34613e) failed [ssh-agent] at hudson.remoting.Channel.call(Channel.java:723) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentFactory.start(JNRRemoteAgentFactory.java:61) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.<init>(SSHAgentBuildWrapper.java:211) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.createSSHAgentEnvironment(SSHAgentBuildWrapper.java:123) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:93) [ssh-agent] at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:78) [ssh-agent] at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:556) [ssh-agent] at hudson.model.Run.execute(Run.java:1665) [ssh-agent] at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) [ssh-agent] at hudson.model.ResourceController.execute(ResourceController.java:88) [ssh-agent] at hudson.model.Executor.run(Executor.java:230) [ssh-agent] Caused by: java.lang.UnsatisfiedLinkError: /lib/libc.so.6: wrong ELF class: ELFCLASS32 [ssh-agent] at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87) [ssh-agent] at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:70) [ssh-agent] at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:49) [ssh-agent] at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:59) [ssh-agent] at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:125) [ssh-agent] at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:63) [ssh-agent] at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:43) [ssh-agent] at jnr.ffi.LibraryLoader.load(LibraryLoader.java:228) [ssh-agent] at jnr.ffi.Library.loadLibrary(Library.java:123) [ssh-agent] at jnr.ffi.Library.loadLibrary(Library.java:80) [ssh-agent] at jnr.unixsocket.Native$LibC.<clinit>(Native.java:40) [ssh-agent] at jnr.unixsocket.Native.libsocket(Native.java:60) [ssh-agent] at jnr.unixsocket.Native.socket(Native.java:68) [ssh-agent] at jnr.unixsocket.UnixServerSocketChannel.<init>(UnixServerSocketChannel.java:38) [ssh-agent] at jnr.unixsocket.UnixServerSocket.<init>(UnixServerSocket.java:29) [ssh-agent] at jnr.unixsocket.UnixServerSocketChannel.open(UnixServerSocketChannel.java:48) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer.start(AgentServer.java:67) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.<init>(JNRRemoteAgent.java:64) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentStarter.call(JNRRemoteAgentStarter.java:54) [ssh-agent] at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgentStarter.call(JNRRemoteAgentStarter.java:35) [ssh-agent] at hudson.remoting.UserRequest.perform(UserRequest.java:118) [ssh-agent] at hudson.remoting.UserRequest.perform(UserRequest.java:48) [ssh-agent] at hudson.remoting.Request$2.run(Request.java:326) [ssh-agent] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) [ssh-agent] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [ssh-agent] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [ssh-agent] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [ssh-agent] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [ssh-agent] at java.lang.Thread.run(Thread.java:679) FATAL: [ssh-agent] Unable to start agent hudson.util.IOException2: [ssh-agent] Unable to start agent at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.createSSHAgentEnvironment(SSHAgentBuildWrapper.java:130) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:93) at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:78) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:556) at hudson.model.Run.execute(Run.java:1665) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Caused by: java.lang.RuntimeException: [ssh-agent] Could not find a suitable ssh-agent provider. at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.<init>(SSHAgentBuildWrapper.java:229) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.createSSHAgentEnvironment(SSHAgentBuildWrapper.java:123) ... 7 more
    via by Mark Klunder,
    • java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87)

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,