java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory

JIRA | Jeremy Barlow | 2 years ago
  1. 0

    On some Ubuntu configurations, an attempt to start Puppet Server 1.0.8 or the Puppet Server in PE 3.8 could fail with messages like the following appearing in the {{puppetserver-daemon.log}}: {noformat} Failed to load feature test for posix: can't find user for 0 Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32 Puppet::Error: Cannot determine basic system flavour {noformat} One reason for this error occurring could be lack of sufficient permissions for Puppet Server to read, write, and execute from the system tmpdir. See SERVER-160. In this case, however, the problem was due to one of the JRuby dependencies of Puppet Server being unable to locate and load the "libcrypt.so" shared library. Puppet Server 1.0.8 depends upon JRuby version 1.7.19. JRuby versions 1.7.17 and later depend upon jnr-posix versions 3.0.8 or later. The later jnr-posix version has support for "crypto" functionality, for which "libcrypt.so" must be loaded by the Java process. In an Ubuntu 14.04 distribution, "libcrypt.so" commonly resides at {{/lib/x86_64-linux-gnu}}. We've found in some cases, however, that the default {{LD_LIBRARY_PATH}} does not include the {{/lib/x86_64-linux-gnu}} directory, the {{libcrypt.so}} shared library fails to be loaded, and Puppet Server ultimately fails to start with the non-intuitive error messages above. Note from the following thread that other direct users of JRuby have encountered this same problem - https://www.ruby-forum.com/topic/6563521. On an Ubuntu amd64 Amazon AMI, we found that the {{LD_LIBRARY_PATH}} was set by default to "/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib". We were able to get the Puppet Server to start by just setting that same path but with the missing "/lib/x86_64-linux-gnu" appended to end of the {{JAVA_ARGS}} in Puppet Server's {{/etc/default/puppetserver}} and restarting the service: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/lib/x86_64-linux-gnu" {noformat} Management of the {{JAVA_ARGS}} setting in PE should be done through the PE module. A more appropriate workaround with instructions for PE users is documented here - http://docs.puppetlabs.com/pe/latest/release_notes_known_issues.html#ubuntu-conflict-when-youre-running-on-fusion-vm-and-amazon-ec2. --- The root cause for this issue was that the {{pe-java}} JDK distribution in Puppet Enterprise, unlike its 1.7.0.75 OpenJDK siblings, did not include the {{/lib/x86_64-linux-gnu}} path in its default library path. Work to rectify this will be tracked in RE-4557. This ticket should only cover a couple of minor updates to Puppet Server to help more easily troubleshoot issues like these in the future. 1) Add a couple of debug log statements to Puppet Server code to output the value of the {{java.library.path}} system property and {{LD_LIBRARY_PATH}} environment variable. Having this available in Puppet Server without needing to modify source code would have been helpful for debugging this issue more easily. [~chris] discussed doing this in the {{init}} function of the {{jruby-puppet-pooled-service}}. 2) Improve documentation on how to troubleshoot a problem like this. The error messages above are not helpful for diagnosing the source of this problem. JRuby has a property, "jruby.native.verbose", that you can set to get more verbose output for native library invocations. With an Ubuntu AMI in the failing state noted in this ticket, one could have set the JAVA_ARGS to: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djruby.native.verbose=true" {noformat} If that change had been made prior to starting the service, the following messages would have appeared in the "puppetserver-daemon.log" file: {noformat} Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87) ... {noformat} This would tell you what library couldn't be loaded and where it was being loaded from - which is really helpful for this case. At the very least, we should reference the use of this flag in Puppet Server documentation. h3. QA ---- Risk assessment: high (QA to perform functional review and write an acceptance test) Risk probability: medium (seems intermittent, and we missed in in CI) Risk severity: high (puppet-server does not start) Test layer: acceptance

    JIRA | 2 years ago | Jeremy Barlow
    java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory
  2. 0

    On some Ubuntu configurations, an attempt to start Puppet Server 1.0.8 or the Puppet Server in PE 3.8 could fail with messages like the following appearing in the {{puppetserver-daemon.log}}: {noformat} Failed to load feature test for posix: can't find user for 0 Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32 Puppet::Error: Cannot determine basic system flavour {noformat} One reason for this error occurring could be lack of sufficient permissions for Puppet Server to read, write, and execute from the system tmpdir. See SERVER-160. In this case, however, the problem was due to one of the JRuby dependencies of Puppet Server being unable to locate and load the "libcrypt.so" shared library. Puppet Server 1.0.8 depends upon JRuby version 1.7.19. JRuby versions 1.7.17 and later depend upon jnr-posix versions 3.0.8 or later. The later jnr-posix version has support for "crypto" functionality, for which "libcrypt.so" must be loaded by the Java process. In an Ubuntu 14.04 distribution, "libcrypt.so" commonly resides at {{/lib/x86_64-linux-gnu}}. We've found in some cases, however, that the default {{LD_LIBRARY_PATH}} does not include the {{/lib/x86_64-linux-gnu}} directory, the {{libcrypt.so}} shared library fails to be loaded, and Puppet Server ultimately fails to start with the non-intuitive error messages above. Note from the following thread that other direct users of JRuby have encountered this same problem - https://www.ruby-forum.com/topic/6563521. On an Ubuntu amd64 Amazon AMI, we found that the {{LD_LIBRARY_PATH}} was set by default to "/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib". We were able to get the Puppet Server to start by just setting that same path but with the missing "/lib/x86_64-linux-gnu" appended to end of the {{JAVA_ARGS}} in Puppet Server's {{/etc/default/puppetserver}} and restarting the service: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib:/lib/x86_64-linux-gnu" {noformat} Management of the {{JAVA_ARGS}} setting in PE should be done through the PE module. A more appropriate workaround with instructions for PE users is documented here - http://docs.puppetlabs.com/pe/latest/release_notes_known_issues.html#ubuntu-conflict-when-youre-running-on-fusion-vm-and-amazon-ec2. --- The root cause for this issue was that the {{pe-java}} JDK distribution in Puppet Enterprise, unlike its 1.7.0.75 OpenJDK siblings, did not include the {{/lib/x86_64-linux-gnu}} path in its default library path. Work to rectify this will be tracked in RE-4557. This ticket should only cover a couple of minor updates to Puppet Server to help more easily troubleshoot issues like these in the future. 1) Add a couple of debug log statements to Puppet Server code to output the value of the {{java.library.path}} system property and {{LD_LIBRARY_PATH}} environment variable. Having this available in Puppet Server without needing to modify source code would have been helpful for debugging this issue more easily. [~chris] discussed doing this in the {{init}} function of the {{jruby-puppet-pooled-service}}. 2) Improve documentation on how to troubleshoot a problem like this. The error messages above are not helpful for diagnosing the source of this problem. JRuby has a property, "jruby.native.verbose", that you can set to get more verbose output for native library invocations. With an Ubuntu AMI in the failing state noted in this ticket, one could have set the JAVA_ARGS to: {noformat} JAVA_ARGS="-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djruby.native.verbose=true" {noformat} If that change had been made prior to starting the service, the following messages would have appeared in the "puppetserver-daemon.log" file: {noformat} Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87) ... {noformat} This would tell you what library couldn't be loaded and where it was being loaded from - which is really helpful for this case. At the very least, we should reference the use of this flag in Puppet Server documentation. h3. QA ---- Risk assessment: high (QA to perform functional review and write an acceptance test) Risk probability: medium (seems intermittent, and we missed in in CI) Risk severity: high (puppet-server does not start) Test layer: acceptance

    JIRA | 2 years ago | Jeremy Barlow
    java.lang.UnsatisfiedLinkError: libcrypt.so: cannot open shared object file: No such file or directory
  3. 0

    Add support for musl libc

    GitHub | 8 months ago | jirutka
    java.lang.UnsatisfiedLinkError: Error loading shared library libcrypt.so: No such file or directory
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Add support for musl libc

    GitHub | 8 months ago | jirutka
    java.lang.UnsatisfiedLinkError: Error loading shared library libcrypt.so: No such file or directory
  6. 0

    Add support for musl libc

    GitHub | 8 months ago | jirutka
    java.lang.UnsatisfiedLinkError: Error loading shared library libcrypt.so: No such file or directory

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.UnsatisfiedLinkError

      libcrypt.so: cannot open shared object file: No such file or directory

      at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries()
    2. JRuby Main Maven Artifact
      NativeLibrary.loadNativeLibraries
      1. jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87)
      1 frame