org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed

Spring JIRA | Radim Kolar | 4 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Newer hadoops have implemented FileSystem object cache. it means that calling Filesystem.get() didnt always return new object, it prefers to return cached instance. Problem is that after calling fs.close() no other work with hdfs is possible. I am using spring context in mapper task. in xml i have standard: <hadoop:configuration/> <hadoop:file-system/> after finishing my task, i am closing spring context and this closes filesystem too. This is problem because hadoop mapper can not finish its cleanup tasks and marks entire task attempt as FAILED. I need you to add another boolean property to https://github.com/SpringSource/spring-hadoop/blob/master/src/main/java/org/springframework/data/hadoop/fs/FileSystemFactoryBean.java called close (default true). If set to false then filesystem is not closed when spring context is shutting down. 2012-08-29 17:55:05,670 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing org.springframework.context.support.ClassPathXmlApplicationContext@67148603: startup date [Wed Aug 29 17:53:44 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,674 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,724 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,725 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4adf2940: defining beans [classloader-fix,default,fetch,generate,parse,index]; root of factory hierarchy 2012-08-29 17:55:05,725 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'generate': startup date [Wed Aug 29 17:54:15 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,746 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,746 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b: defining beans [org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopConfiguration,hadoopFs,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,urlfilters,urlnormalizers,parsers,parsefilters,mimedetectors,encodingdetectors,indexingfilters,indexwriters,protocols,scoringfilters,signatures,cacheManager,org.springframework.aop.config.internalAutoProxyCreator,org.springframework.cache.annotation.AnnotationCacheOperationSource#0,org.springframework.cache.interceptor.CacheInterceptor#0,org.springframework.cache.config.internalCacheAdvisor,pluginmanager-skeleton,classloader-fix,jsr303validator,pluginmanager,nutchunsort,selector-mapper,selector-reducer,selector-partitioner,partition-mode,schedule,generate,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy 2012-08-29 17:55:05,754 INFO [main] org.apache.nutch.plugin.PluginManager: Closing ApplicationContext 'pluginmanager': startup date [Wed Aug 29 17:54:34 CEST 2012]; parent: ApplicationContext 'generate' 2012-08-29 17:55:05,754 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c825cf3: defining beans [lib-domutils,creativecommons,creativecommons/index,creativecommons/parse,encoding-icu,encoding-icu/detector,feed,feed/parse,feed/index,htmlparser-neko,htmlparser-neko/dombuilder,htmlparser-tagsoup,htmlparser-tagsoup/dombuilder,index-anchor,index-anchor/index,index-basic,index-basic/index,index-more/index,index-more,index-static,index-static/index,language-identifier,language-identifier/parse,language-identifier/index,lib-crawlercommons,robotstxt,robotstxt/manager,lib-http,lib-regex-filter,lib-robots,lib-robots/parser,lib-xml,microformats-reltag,microformats-reltag/index,microformats-reltag/parse,mime-tikamagic,mime-tikamagic/detect,parse-ext,parse-ext/parser,parse-html,parse-html/parse,parse-js,parse-js/parse,parse-swf,parse-swf/parse,parse-tika,parse-tika/parse,parse-zip,parse-zip/parse,protocol-file,protocol-file/file,protocol-ftp,protocol-ftp/ftp,protocol-http,protocol-http/http,protocol-httpclient,protocol-httpclient/http,protocol-httpclient/https,scoring-link,scoring-link/scoring,scoring-opic,scoring-opic/score,subcollection,subcollection/index,subcollection/filter,tld,tld/index,tld/score,urlfilter-automaton,urlfilter-automaton/filter,urlfilter-domain,urlfilter-domain/filter,urlfilter-prefix,urlfilter-prefix/filter,urlfilter-regex,urlfilter-regex/filter,urlfilter-suffix,urlfilter-suffix/filter,urlfilter-validator,urlfilter-validator/filter,urlmeta,urlmeta/score,urlmeta/index,urlnormalizer-basic,urlnormalizer-basic/normalize,urlnormalizer-pass,urlnormalizer-pass/normalize,urlnormalizer-regex,urlnormalizer-regex/normalize,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jsr303validator,cz-domainfilter,cz-domains-res,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b 2012-08-29 17:55:06,241 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:crawler (auth:SIMPLE) cause:java.io.IOException: Filesystem closed 2012-08-29 17:55:06,242 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:371) at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:453) at java.io.FilterInputStream.close(FilterInputStream.java:172) at org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1916) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.close(SequenceFileRecordReader.java:105) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:463) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)

    Spring JIRA | 4 years ago | Radim Kolar
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed
  2. 0

    Newer hadoops have implemented FileSystem object cache. it means that calling Filesystem.get() didnt always return new object, it prefers to return cached instance. Problem is that after calling fs.close() no other work with hdfs is possible. I am using spring context in mapper task. in xml i have standard: <hadoop:configuration/> <hadoop:file-system/> after finishing my task, i am closing spring context and this closes filesystem too. This is problem because hadoop mapper can not finish its cleanup tasks and marks entire task attempt as FAILED. I need you to add another boolean property to https://github.com/SpringSource/spring-hadoop/blob/master/src/main/java/org/springframework/data/hadoop/fs/FileSystemFactoryBean.java called close (default true). If set to false then filesystem is not closed when spring context is shutting down. 2012-08-29 17:55:05,670 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing org.springframework.context.support.ClassPathXmlApplicationContext@67148603: startup date [Wed Aug 29 17:53:44 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,674 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,724 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,725 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4adf2940: defining beans [classloader-fix,default,fetch,generate,parse,index]; root of factory hierarchy 2012-08-29 17:55:05,725 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'generate': startup date [Wed Aug 29 17:54:15 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,746 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,746 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b: defining beans [org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopConfiguration,hadoopFs,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,urlfilters,urlnormalizers,parsers,parsefilters,mimedetectors,encodingdetectors,indexingfilters,indexwriters,protocols,scoringfilters,signatures,cacheManager,org.springframework.aop.config.internalAutoProxyCreator,org.springframework.cache.annotation.AnnotationCacheOperationSource#0,org.springframework.cache.interceptor.CacheInterceptor#0,org.springframework.cache.config.internalCacheAdvisor,pluginmanager-skeleton,classloader-fix,jsr303validator,pluginmanager,nutchunsort,selector-mapper,selector-reducer,selector-partitioner,partition-mode,schedule,generate,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy 2012-08-29 17:55:05,754 INFO [main] org.apache.nutch.plugin.PluginManager: Closing ApplicationContext 'pluginmanager': startup date [Wed Aug 29 17:54:34 CEST 2012]; parent: ApplicationContext 'generate' 2012-08-29 17:55:05,754 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c825cf3: defining beans [lib-domutils,creativecommons,creativecommons/index,creativecommons/parse,encoding-icu,encoding-icu/detector,feed,feed/parse,feed/index,htmlparser-neko,htmlparser-neko/dombuilder,htmlparser-tagsoup,htmlparser-tagsoup/dombuilder,index-anchor,index-anchor/index,index-basic,index-basic/index,index-more/index,index-more,index-static,index-static/index,language-identifier,language-identifier/parse,language-identifier/index,lib-crawlercommons,robotstxt,robotstxt/manager,lib-http,lib-regex-filter,lib-robots,lib-robots/parser,lib-xml,microformats-reltag,microformats-reltag/index,microformats-reltag/parse,mime-tikamagic,mime-tikamagic/detect,parse-ext,parse-ext/parser,parse-html,parse-html/parse,parse-js,parse-js/parse,parse-swf,parse-swf/parse,parse-tika,parse-tika/parse,parse-zip,parse-zip/parse,protocol-file,protocol-file/file,protocol-ftp,protocol-ftp/ftp,protocol-http,protocol-http/http,protocol-httpclient,protocol-httpclient/http,protocol-httpclient/https,scoring-link,scoring-link/scoring,scoring-opic,scoring-opic/score,subcollection,subcollection/index,subcollection/filter,tld,tld/index,tld/score,urlfilter-automaton,urlfilter-automaton/filter,urlfilter-domain,urlfilter-domain/filter,urlfilter-prefix,urlfilter-prefix/filter,urlfilter-regex,urlfilter-regex/filter,urlfilter-suffix,urlfilter-suffix/filter,urlfilter-validator,urlfilter-validator/filter,urlmeta,urlmeta/score,urlmeta/index,urlnormalizer-basic,urlnormalizer-basic/normalize,urlnormalizer-pass,urlnormalizer-pass/normalize,urlnormalizer-regex,urlnormalizer-regex/normalize,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jsr303validator,cz-domainfilter,cz-domains-res,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b 2012-08-29 17:55:06,241 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:crawler (auth:SIMPLE) cause:java.io.IOException: Filesystem closed 2012-08-29 17:55:06,242 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:371) at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:453) at java.io.FilterInputStream.close(FilterInputStream.java:172) at org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1916) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.close(SequenceFileRecordReader.java:105) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:463) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)

    Spring JIRA | 4 years ago | Radim Kolar
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed
  3. 0

    Job submitted in oozie is getting killed

    Stack Overflow | 3 years ago | user3410414
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Exception running child : java.io.IOException: Filesystem closed

      at org.apache.hadoop.hdfs.DFSClient.checkOpen()
    2. Apache Hadoop HDFS
      DFSInputStream.close
      1. org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:371)
      2. org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:453)
      2 frames
    3. Java RT
      FilterInputStream.close
      1. java.io.FilterInputStream.close(FilterInputStream.java:172)
      1 frame
    4. Hadoop
      SequenceFile$Reader.close
      1. org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1916)
      1 frame
    5. Hadoop
      YarnChild$2.run
      1. org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.close(SequenceFileRecordReader.java:105)
      2. org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:463)
      3. org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729)
      4. org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      5. org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154)
      5 frames
    6. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:416)
      2 frames
    7. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
      1 frame
    8. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)
      1 frame