org.apache.hadoop.mapred.YarnChild

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • GitHub comment 1186#241139147
    via GitHub by jinhyukchang
    ,
  • Job submitted in oozie is getting killed
    via Stack Overflow by user3410414
    ,
  • Newer hadoops have implemented FileSystem object cache. it means that calling Filesystem.get() didnt always return new object, it prefers to return cached instance. Problem is that after calling fs.close() no other work with hdfs is possible. I am using spring context in mapper task. in xml i have standard: <hadoop:configuration/> <hadoop:file-system/> after finishing my task, i am closing spring context and this closes filesystem too. This is problem because hadoop mapper can not finish its cleanup tasks and marks entire task attempt as FAILED. I need you to add another boolean property to https://github.com/SpringSource/spring-hadoop/blob/master/src/main/java/org/springframework/data/hadoop/fs/FileSystemFactoryBean.java called close (default true). If set to false then filesystem is not closed when spring context is shutting down. 2012-08-29 17:55:05,670 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing org.springframework.context.support.ClassPathXmlApplicationContext@67148603: startup date [Wed Aug 29 17:53:44 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,674 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,724 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,725 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4adf2940: defining beans [classloader-fix,default,fetch,generate,parse,index]; root of factory hierarchy 2012-08-29 17:55:05,725 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'generate': startup date [Wed Aug 29 17:54:15 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,746 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,746 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b: defining beans [org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopConfiguration,hadoopFs,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,urlfilters,urlnormalizers,parsers,parsefilters,mimedetectors,encodingdetectors,indexingfilters,indexwriters,protocols,scoringfilters,signatures,cacheManager,org.springframework.aop.config.internalAutoProxyCreator,org.springframework.cache.annotation.AnnotationCacheOperationSource#0,org.springframework.cache.interceptor.CacheInterceptor#0,org.springframework.cache.config.internalCacheAdvisor,pluginmanager-skeleton,classloader-fix,jsr303validator,pluginmanager,nutchunsort,selector-mapper,selector-reducer,selector-partitioner,partition-mode,schedule,generate,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy 2012-08-29 17:55:05,754 INFO [main] org.apache.nutch.plugin.PluginManager: Closing ApplicationContext 'pluginmanager': startup date [Wed Aug 29 17:54:34 CEST 2012]; parent: ApplicationContext 'generate' 2012-08-29 17:55:05,754 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c825cf3: defining beans [lib-domutils,creativecommons,creativecommons/index,creativecommons/parse,encoding-icu,encoding-icu/detector,feed,feed/parse,feed/index,htmlparser-neko,htmlparser-neko/dombuilder,htmlparser-tagsoup,htmlparser-tagsoup/dombuilder,index-anchor,index-anchor/index,index-basic,index-basic/index,index-more/index,index-more,index-static,index-static/index,language-identifier,language-identifier/parse,language-identifier/index,lib-crawlercommons,robotstxt,robotstxt/manager,lib-http,lib-regex-filter,lib-robots,lib-robots/parser,lib-xml,microformats-reltag,microformats-reltag/index,microformats-reltag/parse,mime-tikamagic,mime-tikamagic/detect,parse-ext,parse-ext/parser,parse-html,parse-html/parse,parse-js,parse-js/parse,parse-swf,parse-swf/parse,parse-tika,parse-tika/parse,parse-zip,parse-zip/parse,protocol-file,protocol-file/file,protocol-ftp,protocol-ftp/ftp,protocol-http,protocol-http/http,protocol-httpclient,protocol-httpclient/http,protocol-httpclient/https,scoring-link,scoring-link/scoring,scoring-opic,scoring-opic/score,subcollection,subcollection/index,subcollection/filter,tld,tld/index,tld/score,urlfilter-automaton,urlfilter-automaton/filter,urlfilter-domain,urlfilter-domain/filter,urlfilter-prefix,urlfilter-prefix/filter,urlfilter-regex,urlfilter-regex/filter,urlfilter-suffix,urlfilter-suffix/filter,urlfilter-validator,urlfilter-validator/filter,urlmeta,urlmeta/score,urlmeta/index,urlnormalizer-basic,urlnormalizer-basic/normalize,urlnormalizer-pass,urlnormalizer-pass/normalize,urlnormalizer-regex,urlnormalizer-regex/normalize,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jsr303validator,cz-domainfilter,cz-domains-res,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b 2012-08-29 17:55:06,241 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:crawler (auth:SIMPLE) cause:java.io.IOException: Filesystem closed 2012-08-29 17:55:06,242 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:371) at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:453) at java.io.FilterInputStream.close(FilterInputStream.java:172) at org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1916) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.close(SequenceFileRecordReader.java:105) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:463) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)
    via by Radim Kolar,
  • Newer hadoops have implemented FileSystem object cache. it means that calling Filesystem.get() didnt always return new object, it prefers to return cached instance. Problem is that after calling fs.close() no other work with hdfs is possible. I am using spring context in mapper task. in xml i have standard: <hadoop:configuration/> <hadoop:file-system/> after finishing my task, i am closing spring context and this closes filesystem too. This is problem because hadoop mapper can not finish its cleanup tasks and marks entire task attempt as FAILED. I need you to add another boolean property to https://github.com/SpringSource/spring-hadoop/blob/master/src/main/java/org/springframework/data/hadoop/fs/FileSystemFactoryBean.java called close (default true). If set to false then filesystem is not closed when spring context is shutting down. 2012-08-29 17:55:05,670 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing org.springframework.context.support.ClassPathXmlApplicationContext@67148603: startup date [Wed Aug 29 17:53:44 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,674 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,724 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,725 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4adf2940: defining beans [classloader-fix,default,fetch,generate,parse,index]; root of factory hierarchy 2012-08-29 17:55:05,725 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'generate': startup date [Wed Aug 29 17:54:15 CEST 2012]; root of context hierarchy 2012-08-29 17:55:05,746 INFO [main] org.springframework.context.support.DefaultLifecycleProcessor: Stopping beans in phase 0 2012-08-29 17:55:05,746 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b: defining beans [org.springframework.data.hadoop.mapreduce.MapReducePropertyEditorRegistrar.ns.registration,hadoopConfiguration,hadoopFs,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,urlfilters,urlnormalizers,parsers,parsefilters,mimedetectors,encodingdetectors,indexingfilters,indexwriters,protocols,scoringfilters,signatures,cacheManager,org.springframework.aop.config.internalAutoProxyCreator,org.springframework.cache.annotation.AnnotationCacheOperationSource#0,org.springframework.cache.interceptor.CacheInterceptor#0,org.springframework.cache.config.internalCacheAdvisor,pluginmanager-skeleton,classloader-fix,jsr303validator,pluginmanager,nutchunsort,selector-mapper,selector-reducer,selector-partitioner,partition-mode,schedule,generate,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy 2012-08-29 17:55:05,754 INFO [main] org.apache.nutch.plugin.PluginManager: Closing ApplicationContext 'pluginmanager': startup date [Wed Aug 29 17:54:34 CEST 2012]; parent: ApplicationContext 'generate' 2012-08-29 17:55:05,754 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4c825cf3: defining beans [lib-domutils,creativecommons,creativecommons/index,creativecommons/parse,encoding-icu,encoding-icu/detector,feed,feed/parse,feed/index,htmlparser-neko,htmlparser-neko/dombuilder,htmlparser-tagsoup,htmlparser-tagsoup/dombuilder,index-anchor,index-anchor/index,index-basic,index-basic/index,index-more/index,index-more,index-static,index-static/index,language-identifier,language-identifier/parse,language-identifier/index,lib-crawlercommons,robotstxt,robotstxt/manager,lib-http,lib-regex-filter,lib-robots,lib-robots/parser,lib-xml,microformats-reltag,microformats-reltag/index,microformats-reltag/parse,mime-tikamagic,mime-tikamagic/detect,parse-ext,parse-ext/parser,parse-html,parse-html/parse,parse-js,parse-js/parse,parse-swf,parse-swf/parse,parse-tika,parse-tika/parse,parse-zip,parse-zip/parse,protocol-file,protocol-file/file,protocol-ftp,protocol-ftp/ftp,protocol-http,protocol-http/http,protocol-httpclient,protocol-httpclient/http,protocol-httpclient/https,scoring-link,scoring-link/scoring,scoring-opic,scoring-opic/score,subcollection,subcollection/index,subcollection/filter,tld,tld/index,tld/score,urlfilter-automaton,urlfilter-automaton/filter,urlfilter-domain,urlfilter-domain/filter,urlfilter-prefix,urlfilter-prefix/filter,urlfilter-regex,urlfilter-regex/filter,urlfilter-suffix,urlfilter-suffix/filter,urlfilter-validator,urlfilter-validator/filter,urlmeta,urlmeta/score,urlmeta/index,urlnormalizer-basic,urlnormalizer-basic/normalize,urlnormalizer-pass,urlnormalizer-pass/normalize,urlnormalizer-regex,urlnormalizer-regex/normalize,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jsr303validator,cz-domainfilter,cz-domains-res,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@16ad339b 2012-08-29 17:55:06,241 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:crawler (auth:SIMPLE) cause:java.io.IOException: Filesystem closed 2012-08-29 17:55:06,242 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:371) at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:453) at java.io.FilterInputStream.close(FilterInputStream.java:172) at org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1916) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.close(SequenceFileRecordReader.java:105) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:463) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)
    via by Radim Kolar,
    • org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399) at gobblin.metastore.FsStateStore.put(FsStateStore.java:135) at gobblin.runtime.AbstractJobLauncher.runWorkUnits(AbstractJobLauncher.java:644) at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.run(MRJobLauncher.java:597) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
    No Bugmate found.