org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

Apache's JIRA Issue Tracker | Mostafa Elhemali | 4 years ago
  1. 0

    In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows: a) LD_LIBRARY_PATH doesn't mean anything in Windows. b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%. The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error: {code} 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235) at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) {code}

    Apache's JIRA Issue Tracker | 4 years ago | Mostafa Elhemali
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  2. 0

    In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows: a) LD_LIBRARY_PATH doesn't mean anything in Windows. b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%. The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error: {code} 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235) at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) {code}

    Apache's JIRA Issue Tracker | 4 years ago | Mostafa Elhemali
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  3. 0

    [jira] [Created] (YARN-729) MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows - Mostafa Elhemali (JIRA) - org.apache.hadoop.yarn-dev - MarkMail

    markmail.org | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Druid-0.9.0-rc2 batch ingestion failing

    Google Groups | 9 months ago | Nook
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
  6. 0

    Edit YARN's classpath in Oozie

    Stack Overflow | 6 months ago | Nik
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

      at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0()
    2. Hadoop
      LocalDirAllocator.getLocalPathToRead
      1. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
      2. org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
      3. org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
      4. org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
      5. org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
      6. org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
      7. org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
      8. org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
      9. org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
      9 frames
    3. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
      2. org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
      3. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
      3 frames