water.DException$DistributedException: from /172.16.2.164:44000; by class hex.tree.ScoreBuildHistogram; class java.lang.AssertionError: null

JIRA | Neeraja Madabhushi | 2 years ago
  1. 0

    Seen on Jul 10, 2015 3:05 PM Jul 10, 2015 6:32 PM in h2o_master_DEV_runit_medium_large - runit_deeplearning_stacked_autoencoder_large.py From JAVA log: 07-10 19:00:54.621 172.17.2.164:44000 13543 #464-1329 INFO: Parse chunk size 4194304 07-10 19:00:55.443 172.17.2.164:44000 13543 FJ-0-145 INFO: Parse result for test.hex_2 (10000 rows): onExCompletion for water.fvec.RollupStats$1@2df1fb9c water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null at water.fvec.Vec.chunkIdx(Vec.java:731) at water.fvec.Vec.chunkForChunkIdx(Vec.java:793) at water.MRTask.compute2(MRTask.java:624) at water.H2O$H2OCountedCompleter.compute(H2O.java:867) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 07-10 19:00:55.823 172.17.2.164:44000 13543 FJ-0-145 INFO: ColV2 type min max NAs constant numLevels 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C1: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C2: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C3: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C4: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C5: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C6: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C7: numeric 0.00000 0.00000 constant 07-10 19:00:55.824 172.17.2.164:44000 13543 FJ-0-145 INFO: C8: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: C9: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: C10: numeric 0.00000 0.00000 constant 07-10 19:00:55.825 172.17.2.164:44000 13543 FJ-0-145 INFO: Additional column information only sent to log file... Remote rollups failed with an exception, wrapping and rethrowing: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null 07-10 19:00:56.695 172.17.2.164:44000 13543 #464-1607 INFO: Method: GET , URI: /3/Frames/test.hex_2, route: /3/Frames/(?<frameid>.*), parms: {frame_id=test.hex_2} Remote rollups failed with an exception, wrapping and rethrowing: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null 07-10 19:00:56.699 172.17.2.164:44000 13543 #464-1607 WARN: Caught exception: water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null; Stacktrace: [water.fvec.RollupStats.get(RollupStats.java:283), water.fvec.RollupStats.get(RollupStats.java:292), water.fvec.Vec.rollupStats(Vec.java:625), water.fvec.Vec.checksum_impl(Vec.java:644), water.Keyed.checksum(Keyed.java:56), water.fvec.Frame.checksum_impl(Frame.java:433), water.Keyed.checksum(Keyed.java:56), water.api.FrameV3.fillFromImpl(FrameV3.java:231), water.api.FrameV3.<init>(FrameV3.java:216), water.api.FrameV3.<init>(FrameV3.java:212), water.api.FramesHandler.doFetch(FramesHandler.java:241), water.api.FramesHandler.fetch(FramesHandler.java:225), sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:606), water.api.Handler.handle(Handler.java:56), water.api.RequestServer.handle(RequestServer.java:660), water.api.RequestServer.serve(RequestServer.java:597), water.JettyHTTPD$H2oDefaultServlet.doGeneric(JettyHTTPD.java:454), water.JettyHTTPD$H2oDefaultServlet.doGet(JettyHTTPD.java:399), javax.servlet.http.HttpServlet.service(HttpServlet.java:687), javax.servlet.http.HttpServlet.service(HttpServlet.java:790), org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808), org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587), org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143), org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577), org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223), org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127), org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515), org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185), org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061), org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141), org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110), org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97), org.eclipse.jetty.server.Server.handle(Server.java:499), org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310), org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257), org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540), org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635), org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555), java.lang.Thread.run(Thread.java:745)] 07-10 19:00:58.634 172.17.2.164:44000 13543 #464-1253 INFO: Method: GET , URI: /, route: , parms: {} 07-10 19:00:58.639 172.17.2.164:44000 13543 #464-1381 INFO: Method: GET , URI: /, route: , parms: {} 07-10 19:00:58.674 172.17.2.164:44000 13543 #464-1367 INFO: Method: GET , URI: /3/InitID, route: /3/InitID, parms: {} 07-10 19:01:01.074 172.17.2.164:44000 13543 #464-1368 INFO: ------------------------------------------------------------

    JIRA | 1 year ago | Brandon Hill
    water.DException$DistributedException: from /172.17.2.164:44002; by class water.fvec.RollupStats$ComputeRollupsTask; class water.DException$DistributedException: from /172.17.2.164:44004; by class water.fvec.RollupStats$Roll; class java.lang.AssertionError: null
  2. Speed up your debug routine!

    Automated exception search integrated into your IDE

  3. 0

    From [~kbn]: nishant got this GBM "Trying to unlock null" assertion during pyunit_citi_bike_large.py. Seems like it's a delete of some key related to GBM. test seemed to keep going though. from http://mr-0xa1:8080/view/nishant/job/nishant_code_coverage/41/artifact/h2o-py/tests/results/java_0_0.out.txt He later got other assertions that have appeared elsewhere with the pyunit_citi_bike_large.py here's the one I hadn't seen before: 06-30 18:01:35.534 172.17.2.154:56789 3951 # Session INFO: Method: GET , URI: /3/Models/GBMModel__8c033c5ded17b06a9f57036a08014faa, route: /3/Models/(?<modelid>.*), parms: {model_id=GBMModel__8c033c5ded17b06a9f57036a08014faa} 06-30 18:01:35.541 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'pyfdf9ce18-09a0-4dff-98ea-353bf6c7e119')} 06-30 18:01:54.250 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'py3b2531b3-1876-4756-8e0b-454f46b87fb3')} 06-30 18:01:54.286 172.17.2.154:56789 3951 # Session INFO: Method: DELETE, URI: /3/DKV/GBMModel__ae8e7b4651349614921bec0629064c9b, route: /3/DKV/(?<key>.*), parms: {key=GBMModel__ae8e7b4651349614921bec0629064c9b} barrier onExCompletion for hex.tree.gbm.GBM$GBMDriver@6c5d2aea water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null! at water.Lockable$Unlock.atomic(Lockable.java:180) at water.Lockable$Unlock.atomic(Lockable.java:176) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.H2O$H2OCountedCompleter.compute(H2O.java:698) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 06-30 18:01:54.570 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(, (gput py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (cbind %FALSE 'py99fef45f-7d80-41b3-88df-e65c578e2677' 'py1844e0f1-6095-4b69-9e03-8c3b3a9bd336' 'py82e77fc3-c2d4-4093-b77f-c983ced3e0c4' 'py554158f7-9142-469d-bc9b-2d1479b2b118' 'pyce700595-02fb-455b-a13d-6d4ea0f978f3' 'pyc37f39b1-ba43-4eed-bb0d-9f9e6bc19032' 'pyfa208086-445e-4507-bff5-b42a50f6b1ed' 'pybf318908-1f6b-4625-9a3c-2cd1a24affd9' 'pyf97296f1-6bb5-485f-b77b-ad224f498648' 'py74884c42-f4f2-443e-b015-9c736069f699')) (colnames= %py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (: #0 #9) (slist "Days" "start station name" "Month" "DayOfWeek" "Humidity Fraction" "Rain (mm)" "Temperature (C)" "WC1" "Dew Point (C)" "bikes")}

    JIRA | 1 year ago | Raymond Peck
    water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null!
  4. 0

    From [~kbn]: nishant got this GBM "Trying to unlock null" assertion during pyunit_citi_bike_large.py. Seems like it's a delete of some key related to GBM. test seemed to keep going though. from http://mr-0xa1:8080/view/nishant/job/nishant_code_coverage/41/artifact/h2o-py/tests/results/java_0_0.out.txt He later got other assertions that have appeared elsewhere with the pyunit_citi_bike_large.py here's the one I hadn't seen before: 06-30 18:01:35.534 172.17.2.154:56789 3951 # Session INFO: Method: GET , URI: /3/Models/GBMModel__8c033c5ded17b06a9f57036a08014faa, route: /3/Models/(?<modelid>.*), parms: {model_id=GBMModel__8c033c5ded17b06a9f57036a08014faa} 06-30 18:01:35.541 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'pyfdf9ce18-09a0-4dff-98ea-353bf6c7e119')} 06-30 18:01:54.250 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'py3b2531b3-1876-4756-8e0b-454f46b87fb3')} 06-30 18:01:54.286 172.17.2.154:56789 3951 # Session INFO: Method: DELETE, URI: /3/DKV/GBMModel__ae8e7b4651349614921bec0629064c9b, route: /3/DKV/(?<key>.*), parms: {key=GBMModel__ae8e7b4651349614921bec0629064c9b} barrier onExCompletion for hex.tree.gbm.GBM$GBMDriver@6c5d2aea water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null! at water.Lockable$Unlock.atomic(Lockable.java:180) at water.Lockable$Unlock.atomic(Lockable.java:176) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.H2O$H2OCountedCompleter.compute(H2O.java:698) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 06-30 18:01:54.570 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(, (gput py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (cbind %FALSE 'py99fef45f-7d80-41b3-88df-e65c578e2677' 'py1844e0f1-6095-4b69-9e03-8c3b3a9bd336' 'py82e77fc3-c2d4-4093-b77f-c983ced3e0c4' 'py554158f7-9142-469d-bc9b-2d1479b2b118' 'pyce700595-02fb-455b-a13d-6d4ea0f978f3' 'pyc37f39b1-ba43-4eed-bb0d-9f9e6bc19032' 'pyfa208086-445e-4507-bff5-b42a50f6b1ed' 'pybf318908-1f6b-4625-9a3c-2cd1a24affd9' 'pyf97296f1-6bb5-485f-b77b-ad224f498648' 'py74884c42-f4f2-443e-b015-9c736069f699')) (colnames= %py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (: #0 #9) (slist "Days" "start station name" "Month" "DayOfWeek" "Humidity Fraction" "Rain (mm)" "Temperature (C)" "WC1" "Dew Point (C)" "bikes")}

    JIRA | 1 year ago | Raymond Peck
    water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null!

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. water.DException$DistributedException

      from /172.16.2.164:44000; by class hex.tree.ScoreBuildHistogram; class java.lang.AssertionError: null

      at hex.tree.DHistogram.init()
    2. hex.tree
      ScoreBuildHistogram.setupLocal
      1. hex.tree.DHistogram.init(DHistogram.java:131)
      2. hex.tree.ScoreBuildHistogram.setupLocal(ScoreBuildHistogram.java:78)
      2 frames
    3. water
      H2O$H2OCountedCompleter.compute
      1. water.MRTask.setupLocal0(MRTask.java:335)
      2. water.MRTask.dinvoke(MRTask.java:278)
      3. water.RPC$RPCCall.compute2(RPC.java:324)
      4. water.H2O$H2OCountedCompleter.compute(H2O.java:580)
      4 frames
    4. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames