water.DException$DistributedException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Parsing MNIST test in GZIP on 3 nodes in runit_deeplearning_stacked_autoencoder_large.R On driver node: 07-22 11:17:47.537 172.17.2.164:45000 12693 FJ-0-59 INFO: Parse result for test.hex_2 (10000 rows): onExCompletion for water.fvec.RollupStats$1@32807f5 water.DException$DistributedException: from /172.17.2.164:45000; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:45002; by class water.fvec.RollupStats$Roll; class water.DException$DistributedException: from /172.17.2.164:45004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: Missing chunk $05ffba01000014000000$nfs://home2/0xdiag/bigdata/laptop/mnist/test.csv.gz at water.fvec.Vec.chunkIdx(Vec.java:757) at water.fvec.Vec.chunkForChunkIdx(Vec.java:820) at water.MRTask.compute2(MRTask.java:624) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.H2O$H2OCountedCompleter.compute(H2O.java:947) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) On remote: 07-22 11:17:47.972 172.17.2.164:45004 12695 FJ-2-5 ERRR: Error: Missing chunk 20 for $04ffba010000ffffffff$nfs://home2/0xdiag/bigdata/laptop/mnist/test.csv.gz java.lang.Error at water.fvec.Vec.checkMissing(Vec.java:763) at water.fvec.Vec.chunkIdx(Vec.java:757) at water.fvec.Vec.chunkForChunkIdx(Vec.java:820) at water.MRTask.compute2(MRTask.java:624) at water.H2O$H2OCountedCompleter.compute(H2O.java:947) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    via by Arno Candel,
  • Parsing MNIST test in GZIP on 3 nodes in runit_deeplearning_stacked_autoencoder_large.R On driver node: 07-22 11:17:47.537 172.17.2.164:45000 12693 FJ-0-59 INFO: Parse result for test.hex_2 (10000 rows): onExCompletion for water.fvec.RollupStats$1@32807f5 water.DException$DistributedException: from /172.17.2.164:45000; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:45002; by class water.fvec.RollupStats$Roll; class water.DException$DistributedException: from /172.17.2.164:45004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: Missing chunk $05ffba01000014000000$nfs://home2/0xdiag/bigdata/laptop/mnist/test.csv.gz at water.fvec.Vec.chunkIdx(Vec.java:757) at water.fvec.Vec.chunkForChunkIdx(Vec.java:820) at water.MRTask.compute2(MRTask.java:624) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.H2O$H2OCountedCompleter.compute(H2O.java:947) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) On remote: 07-22 11:17:47.972 172.17.2.164:45004 12695 FJ-2-5 ERRR: Error: Missing chunk 20 for $04ffba010000ffffffff$nfs://home2/0xdiag/bigdata/laptop/mnist/test.csv.gz java.lang.Error at water.fvec.Vec.checkMissing(Vec.java:763) at water.fvec.Vec.chunkIdx(Vec.java:757) at water.fvec.Vec.chunkForChunkIdx(Vec.java:820) at water.MRTask.compute2(MRTask.java:624) at water.H2O$H2OCountedCompleter.compute(H2O.java:947) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    via by Arno Candel,
  • Seen on Jul 10, 2015 3:05 PM Jul 10, 2015 6:32 PM in h2o_master_DEV_runit_medium_large - runit_deeplearning_stacked_autoencoder_large.py From JAVA log: 07-10 19:00:54.621 172.17.2.164:44000 13543 #464-1329 INFO: Parse chunk size 4194304 07-10 19:00:55.443 172.17.2.164:44000 13543 FJ-0-145 INFO: Parse result for test.hex_2 (10000 rows): onExCompletion for water.fvec.RollupStats$1@2df1fb9c water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null at water.fvec.Vec.chunkIdx(Vec.java:731) at water.fvec.Vec.chunkForChunkIdx(Vec.java:793) at water.MRTask.compute2(MRTask.java:624) at water.H2O$H2OCountedCompleter.compute(H2O.java:867) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 07-10 19:00:55.823 172.17.2.164:44000 13543 FJ-0-145 INFO: ColV2 type min max NAs constant numLevels 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C1: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C2: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C3: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C4: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C5: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C6: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C7: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C8: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: C9: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: C10: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: Additional column information only sent to log file... Remote rollups failed with an exception, wrapping and rethrowing: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null 07-10 19:00:56.695 172.17.2.164:44000 13543 #464-1607 INFO: Method: GET , URI: /3/Frames/test.hex_2, route: /3/Frames/(?<frameid>.*), parms: {frame_id=test.hex_2} Remote rollups failed with an exception, wrapping and rethrowing: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null 07-10 19:00:56.699 172.17.2.164:44000 13543 #464-1607 WARN: Caught exception: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null; Stacktrace: [water.fvec.RollupStats.get(RollupStats.java:283), water.fvec.RollupStats.get(RollupStats.java:292), water.fvec.Vec.rollupStats(Vec.java:625), water.fvec.Vec.checksum_impl(Vec.java:644), water.Keyed.checksum(Keyed.java:56), water.fvec.Frame.checksum_impl(Frame.java:433), water.Keyed.checksum(Keyed.java:56), water.api.FrameV3.fillFromImpl(FrameV3.java:231), water.api.FrameV3.<init>(FrameV3.java:216), water.api.FrameV3.<init>(FrameV3.java:212), water.api.FramesHandler.doFetch(FramesHandler.java:241), water.api.FramesHandler.fetch(FramesHandler.java:225), sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:606), water.api.Handler.handle(Handler.java:56), water.api.RequestServer.handle(RequestServer.java:660), water.api.RequestServer.serve(RequestServer.java:597), water.JettyHTTPD$H2oDefaultServlet.doGeneric(JettyHTTPD.java:454), water.JettyHTTPD$H2oDefaultServlet.doGet(JettyHTTPD.java:399), javax.servlet.http.HttpServlet.service(HttpServlet.java:687), javax.servlet.http.HttpServlet.service(HttpServlet.java:790), org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808), org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587), org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143), org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577), org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223), org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127), org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515), org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185), org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061), org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141), org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110), org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97), org.eclipse.jetty.server.Server.handle(Server.java:499), org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310), org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257), org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540), org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635), org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555), java.lang.Thread.run(Thread.java:745)] 07-10 19:00:58.634 172.17.2.164:44000 13543 #464-1253 INFO: Method: GET , URI: /, route: , parms: {} 07-10 19:00:58.639 172.17.2.164:44000 13543 #464-1381 INFO: Method: GET , URI: /, route: , parms: {} 07-10 19:00:58.674 172.17.2.164:44000 13543 #464-1367 INFO: Method: GET , URI: /3/InitID, route: /3/InitID, parms: {} 07-10 19:01:01.074 172.17.2.164:44000 13543 #464-1368 INFO: ------------------------------------------------------------
    via by Brandon Hill,
    • water.DException$DistributedException: from /172.17.2.164:45000; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:45002; by class water.fvec.RollupStats$Roll; class water.DException$DistributedException: from /172.17.2.164:45004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: Missing chunk $05ffba01000014000000$nfs://home2/0xdiag/bigdata/laptop/mnist/test.csv.gz at water.fvec.Vec.chunkIdx(Vec.java:757) at water.fvec.Vec.chunkForChunkIdx(Vec.java:820) at water.MRTask.compute2(MRTask.java:624) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.MRTask.compute2(MRTask.java:599) at water.H2O$H2OCountedCompleter.compute(H2O.java:947) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    No Bugmate found.