org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

Apache's JIRA Issue Tracker | Mostafa Elhemali | 4 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows: a) LD_LIBRARY_PATH doesn't mean anything in Windows. b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%. The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error: {code} 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235) at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) {code}

    Apache's JIRA Issue Tracker | 4 years ago | Mostafa Elhemali
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  2. 0

    In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows: a) LD_LIBRARY_PATH doesn't mean anything in Windows. b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%. The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error: {code} 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177) at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235) at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) {code}

    Apache's JIRA Issue Tracker | 4 years ago | Mostafa Elhemali
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

      at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0()
    2. Hadoop
      LocalDirAllocator.getLocalPathToRead
      1. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
      2. org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
      3. org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
      4. org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
      5. org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
      6. org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
      7. org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
      8. org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
      9. org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
      9 frames
    3. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
      2. org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
      3. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
      3 frames