java.io.IOException: Could not create FileClient

SpringSource Issue Tracker | Yifeng Xiao | 4 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    1.Create a default Mapr Cluster. {code} cluster name: default_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------ master [mapr_cldb, mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_tasktracker, mapr_metrics] 3 2 7680 LOCAL 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- default_mapr-master-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.89.74 Service Ready default_mapr-master-1 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.240 Service Ready default_mapr-master-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.89.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- zookeeper [mapr_zookeeper] 3 1 1024 SHARED 20 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- default_mapr-zookeeper-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready default_mapr-zookeeper-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.74 Service Ready default_mapr-zookeeper-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.239 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 3 2 4096 LOCAL 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- default_mapr-worker-1 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.211 Service Ready default_mapr-worker-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.63 Service Ready default_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.227 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 4096 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ default_mapr-client-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.72 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ----------------------------------------------------------------------------------------------- default_mapr-mysqlserver-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.23 Service Ready {code} 2.Log on client vm. 3.Run following hdfs command on client vm. {code} [mapr@10 ~]$ hadoop fs -ls / 13/01/29 02:32:26 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 02:32:26,4263 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1091656000 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) ls: Could not create FileClient [mapr@10 ~]$ hadoop fs -put /etc/yum.conf /myvolume 13/01/29 01:22:30 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 01:22:30,8009 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1081342272 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) put: Could not create FileClient hadoop fs -mkdir /myvolume/count/in 13/01/29 01:21:26 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 01:21:26,3330 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1096509760 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) mkdir: Could not create FileClient hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/01/29 01:23:58 INFO fs.TestDFSIO: nrFiles = 5 13/01/29 01:23:58 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/01/29 01:23:58 INFO fs.TestDFSIO: bufferSize = 1000000 13/01/29 01:23:58 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 13/01/29 01:23:58 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/01/29 01:23:58 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 2013-01-29 01:23:58,7547 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1076787520 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) java.io.IOException: Could not create FileClient at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:237) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:249) at com.mapr.fs.MapRFileSystem.delete(MapRFileSystem.java:504) at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:175) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:677) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Could not create FileClient
  2. 0

    1.Create a default Mapr Cluster. {code} cluster name: default_mapr, distro: mapr, status: RUNNING GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------ master [mapr_cldb, mapr_jobtracker, mapr_nfs, mapr_webserver, mapr_fileserver, mapr_tasktracker, mapr_metrics] 3 2 7680 LOCAL 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- default_mapr-master-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.89.74 Service Ready default_mapr-master-1 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.240 Service Ready default_mapr-master-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.89.75 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) ---------------------------------------------------------------------- zookeeper [mapr_zookeeper] 3 1 1024 SHARED 20 NODE NAME HOST IP STATUS ---------------------------------------------------------------------------------------------- default_mapr-zookeeper-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.146 Service Ready default_mapr-zookeeper-1 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.74 Service Ready default_mapr-zookeeper-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.239 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------------------------------- worker [mapr_nfs, mapr_fileserver, mapr_tasktracker] 3 2 4096 LOCAL 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------- default_mapr-worker-1 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.211 Service Ready default_mapr-worker-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.63 Service Ready default_mapr-worker-2 sin2-pekaurora-bdcqe010.eng.vmware.com 10.111.88.227 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) --------------------------------------------------------------------------- client [mapr_pig, mapr_hive] 1 1 4096 SHARED 50 NODE NAME HOST IP STATUS ------------------------------------------------------------------------------------------ default_mapr-client-0 sin2-pekaurora-bdcqe007.eng.vmware.com 10.111.88.72 Service Ready GROUP NAME ROLES INSTANCE CPU MEM(MB) TYPE SIZE(GB) -------------------------------------------------------------------------- mysqlserver [mapr_mysql_server] 1 1 1024 SHARED 20 NODE NAME HOST IP STATUS ----------------------------------------------------------------------------------------------- default_mapr-mysqlserver-0 sin2-pekaurora-bdcqe011.eng.vmware.com 10.111.88.23 Service Ready {code} 2.Log on client vm. 3.Run following hdfs command on client vm. {code} [mapr@10 ~]$ hadoop fs -ls / 13/01/29 02:32:26 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 02:32:26,4263 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1091656000 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) ls: Could not create FileClient [mapr@10 ~]$ hadoop fs -put /etc/yum.conf /myvolume 13/01/29 01:22:30 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 01:22:30,8009 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1081342272 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) put: Could not create FileClient hadoop fs -mkdir /myvolume/count/in 13/01/29 01:21:26 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 2013-01-29 01:21:26,3330 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1096509760 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) mkdir: Could not create FileClient hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-test.jar TestDFSIO -write -nrFiles 5 -fileSize 100 TestDFSIO.0.0.4 13/01/29 01:23:58 INFO fs.TestDFSIO: nrFiles = 5 13/01/29 01:23:58 INFO fs.TestDFSIO: fileSize (MB) = 100.0 13/01/29 01:23:58 INFO fs.TestDFSIO: bufferSize = 1000000 13/01/29 01:23:58 WARN fs.MapRFileSystem: Could not find any cluster, defaulting to localhost 13/01/29 01:23:58 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 13/01/29 01:23:58 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 5 files 2013-01-29 01:23:58,7547 ERROR Client fs/client/fileclient/cc/client.cc:304 Thread: 1076787520 Failed to initialize client for cluster 127.0.0.1:7222, error Connection reset by peer(104) java.io.IOException: Could not create FileClient at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:237) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:249) at com.mapr.fs.MapRFileSystem.delete(MapRFileSystem.java:504) at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:175) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:677) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {code}

    SpringSource Issue Tracker | 4 years ago | Yifeng Xiao
    java.io.IOException: Could not create FileClient
  3. 0

    Trouble running TestDFSIO over Alluxio-1.3/MapR-FS 5.0.0

    Google Groups | 6 months ago | ZT
    java.io.IOException: Could not create FileClient
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hello, I am facing error in Build Cube - Step 2 - 4 Reducers are ending with excpetion java.io.IOException: Filesystem closed .... I have searched a lot and tried various versions (1.5.1, 1.5.0 - binary / compiled from sources ...) and always with same result ... all on MapR 4.1 Hadoop The problem seems related also to data ... because exactly same cube defined in different Hive schema (DB) - with exactly same metadata - worked fine. BUT trying to the the fact table and lookup table join (one lookup table only) on the Hive tables as defined in the failing Cube Model works fine !!!! So if it is data issue -> then it is strange that select count(*) from fact left outer lookup works without issues ... Attaching syslog of one failed reducers ... in case anyone gets an idea what could be the cause of this weird error ... Log Type: stderr Log Length: 507 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/filecache/10/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Log Type: stdout Log Length: 304 spill values to disk... create file /kylin/kylin_metadata/kylin-7c7f995d-4925-4fcc-b0bc-c4a97a547c2c/cube_xxxxx_model_150_2ndattempt/fact_distinct_columns/BILLDATE append file /kylin/kylin_metadata/kylin-7c7f995d-4925-4fcc-b0bc-c4a97a547c2c/cube_xxxxx_model_150_2ndattempt/fact_distinct_columns/BILLDATE Log Type: syslog Log Length: 10167 Showing 4096 bytes of 10167 total. Click here for the full log. es from disk 2016-05-03 18:38:11,275 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2016-05-03 18:38:11,275 INFO [main] org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2016-05-03 18:38:11,279 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 8605536 bytes 2016-05-03 18:38:11,322 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2016-05-03 18:38:11,342 INFO [main] org.apache.kylin.engine.mr.common.AbstractHadoopJob: The absolute path for meta dir is /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,347 INFO [main] org.apache.kylin.common.KylinConfig: Use KYLIN_CONF=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,352 INFO [main] org.apache.kylin.common.KylinConfigBase: Kylin Config was updated with kylin.metadata.url : /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,371 INFO [main] org.apache.kylin.cube.CubeManager: Initializing CubeManager with config /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,376 INFO [main] org.apache.kylin.common.persistence.ResourceStore: Using metadata url /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta for resource store 2016-05-03 18:38:11,380 WARN [main] org.apache.kylin.common.persistence.ResourceStore: Failed to load ResourceStore impl class: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/Filter 2016-05-03 18:38:11,772 INFO [main] org.apache.kylin.cube.CubeDescManager: Initializing CubeDescManager with config /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,772 INFO [main] org.apache.kylin.cube.CubeDescManager: Reloading Cube Metadata from folder /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta/cube_desc 2016-05-03 18:38:14,530 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at com.mapr.fs.MapRFileSystem.checkOpen(MapRFileSystem.java:1469) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:555) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:649) at com.mapr.fs.MapRFileSystem.delete(MapRFileSystem.java:1048) at org.apache.hadoop.mapred.Merger$Segment.close(Merger.java:348) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:521) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:533) at org.apache.hadoop.mapred.ReduceTask$4.next(ReduceTask.java:601) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:155) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) 2016-05-03 18:38:14,536 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task ALSO ADDING FULL SYSLOG: ========================= Log Type: syslog Log Length: 10167 2016-05-03 18:38:05,984 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2016-05-03 18:38:05,986 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2016-05-03 18:38:06,213 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2016-05-03 18:38:06,314 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2016-05-03 18:38:06,314 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system started 2016-05-03 18:38:06,435 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2016-05-03 18:38:06,484 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1461856462442_0134, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6bb31fae) 2016-05-03 18:38:06,536 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2016-05-03 18:38:06,842 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134 2016-05-03 18:38:06,999 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2016-05-03 18:38:07,000 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2016-05-03 18:38:07,481 INFO [main] org.apache.hadoop.mapred.Task: mapOutputFile class: org.apache.hadoop.mapred.MapRFsOutputFile 2016-05-03 18:38:07,481 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2016-05-03 18:38:07,521 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2016-05-03 18:38:07,603 INFO [main] org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.DirectShuffle@728cacdc 2016-05-03 18:38:07,632 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=256, memToMemMergeOutputsThreshold=256 2016-05-03 18:38:07,638 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleEventFetcher: attempt_1461856462442_0134_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-05-03 18:38:07,656 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleEventFetcher: attempt_1461856462442_0134_r_000000_0: Got 1 new map-outputs 2016-05-03 18:38:07,685 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,692 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,693 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,695 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,696 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,697 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,698 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,699 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,700 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,702 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,703 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,705 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,767 INFO [MapOutputCopier task_1461856462442_0134_r_000000.0] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy] 2016-05-03 18:38:07,770 INFO [MapOutputCopier task_1461856462442_0134_r_000000.0] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleFetcher: fetcher#0 about to shuffle output of map attempt_1461856462442_0134_m_000000_0 to MEMORY 2016-05-03 18:38:08,046 INFO [MapOutputCopier task_1461856462442_0134_r_000000.0] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: closeInMemoryFile -> map-output of size: 52779014, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->52779014 2016-05-03 18:38:08,047 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleEventFetcher: EventFetcher is interrupted.. Returning 2016-05-03 18:38:08,141 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-05-03 18:38:08,151 INFO [main] org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2016-05-03 18:38:08,152 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 52778982 bytes 2016-05-03 18:38:08,160 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.snappy] 2016-05-03 18:38:11,273 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: Merged 1 segments, 52779014 bytes to disk to satisfy reduce memory limit 2016-05-03 18:38:11,274 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: Merging 1 files, 8644203 bytes from disk 2016-05-03 18:38:11,275 INFO [main] org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2016-05-03 18:38:11,275 INFO [main] org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2016-05-03 18:38:11,279 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 8605536 bytes 2016-05-03 18:38:11,322 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2016-05-03 18:38:11,342 INFO [main] org.apache.kylin.engine.mr.common.AbstractHadoopJob: The absolute path for meta dir is /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,347 INFO [main] org.apache.kylin.common.KylinConfig: Use KYLIN_CONF=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,352 INFO [main] org.apache.kylin.common.KylinConfigBase: Kylin Config was updated with kylin.metadata.url : /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,371 INFO [main] org.apache.kylin.cube.CubeManager: Initializing CubeManager with config /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,376 INFO [main] org.apache.kylin.common.persistence.ResourceStore: Using metadata url /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta for resource store 2016-05-03 18:38:11,380 WARN [main] org.apache.kylin.common.persistence.ResourceStore: Failed to load ResourceStore impl class: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/Filter 2016-05-03 18:38:11,772 INFO [main] org.apache.kylin.cube.CubeDescManager: Initializing CubeDescManager with config /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta 2016-05-03 18:38:11,772 INFO [main] org.apache.kylin.cube.CubeDescManager: Reloading Cube Metadata from folder /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1461856462442_0134/container_1461856462442_0134_01_000003/meta/cube_desc 2016-05-03 18:38:14,530 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Filesystem closed at com.mapr.fs.MapRFileSystem.checkOpen(MapRFileSystem.java:1469) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:555) at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:649) at com.mapr.fs.MapRFileSystem.delete(MapRFileSystem.java:1048) at org.apache.hadoop.mapred.Merger$Segment.close(Merger.java:348) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:521) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:533) at org.apache.hadoop.mapred.ReduceTask$4.next(ReduceTask.java:601) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:155) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) 2016-05-03 18:38:14,536 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task

    Apache's JIRA Issue Tracker | 10 months ago | Richard Calaba
    java.io.IOException: Filesystem closed

    Root Cause Analysis

    1. java.io.IOException

      Could not create FileClient

      at com.mapr.fs.MapRFileSystem.lookupClient()
    2. com.mapr.fs
      MapRFileSystem.delete
      1. com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:237)
      2. com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:249)
      3. com.mapr.fs.MapRFileSystem.delete(MapRFileSystem.java:504)
      3 frames
    3. Hadoop
      TestDFSIO.main
      1. org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:175)
      2. org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:677)
      3. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      4. org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      5. org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:537)
      5 frames
    4. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    5. Hadoop
      ProgramDriver.driver
      1. org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      2. org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      2 frames
    6. org.apache.hadoop
      AllTestDriver.main
      1. org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:83)
      1 frame
    7. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    8. Hadoop
      RunJar.main
      1. org.apache.hadoop.util.RunJar.main(RunJar.java:197)
      1 frame