java.lang.RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122

JIRA | Kevin Normoyle | 2 years ago
  1. 0

    Not sure why this assertion is WARN? if it's an exception, it should be an ERROR? The assertion repros if the seed is used as shown. If not, I seem to get a bad json response, which is printed in red. I used a test that works on h2o that has minimal port to h2o-dev..i.e. it uses base h2o methods like import/parse/frames which I tested with other tests. the unique thing is this uses utf8 characters. I had sorted out all known issues with utf8 (i.e. what works) on h2o. I was suprised to see a bad json response from h2o-dev. I reduced the number of rows to get the domain values shorter for debug. this time, I got an assertion error from h2o-dev I'll leave the test as is (not sure it will complete but it should) it creates a random dataset. The seed can be forced with -s if needed to replicate ( I think not needed) actually I think I'll hardwire the seed in the test so you don't need the -s param shown. I also attached the file created in sandbox/syn_datasets to run cd h2o-dev/py2/testdir_multi_jvm (it does 2 jvms. don't know if that matters) python test_parse_rand_utf8.py -s 1364157389021990032 to run on existing cloud python test_parse_rand_utf8.py -s 1364157389021990032 -uc or python test_parse_rand_utf8.py -s 1364157389021990032 -uc localhost:54321 if you need to change the default 54321 port frames parameters: {'find_compatible_models': 0, 'len': 5, 'offset': 0} JSON call returned non-200 status with http://192.168.0.34:54323/3/Frames.json?find_compatible_models=0&len=5&offset=0 r.status_code: 500 r.headers: {'date': 'Sun, 16 Nov 2014 04:44:31 GMT', 'content-type': 'application/json'} r.text: {"schema_version":-1,"schema_name":"Schema","schema_type":"Iced","status_header":"500 Internal Server Error","errmsg":"RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122","errurl":null,"stacktrace":["water.MRTask.getResult(MRTask.java:261)","water.MRTask.doAll(MRTask.java:291)","water.MRTask.doAllNodes(MRTask.java:283)","water.KeySnapshot.globalSnapshot(KeySnapshot.java:208)","water.KeySnapshot.globalSnapshot(KeySnapshot.java:195)","water.api.FramesHandler$Frames.fetchAll(FramesHandler.java:29)","water.api.FramesHandler.list(FramesHandler.java:116)","sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)","sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:606)","water.api.Handler.handle(Handler.java:55)","water.api.RequestServer.handle(RequestServer.java:476)","water.api.RequestServer.serve(RequestServer.java:454)","water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:432)","java.lang.Thread.run(Thread.java:745)"]} ERROR 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: java.lang.RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.getResult(MRTask.java:261) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.doAll(MRTask.java:291) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.doAllNodes(MRTask.java:283) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.KeySnapshot.globalSnapshot(KeySnapshot.java:208) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.KeySnapshot.globalSnapshot(KeySnapshot.java:195) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.FramesHandler$Frames.fetchAll(FramesHandler.java:29) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.FramesHandler.list(FramesHandler.java:116) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at java.lang.reflect.Method.invoke(Method.java:606) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.Handler.handle(Handler.java:55) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.RequestServer.handle(RequestServer.java:476) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.RequestServer.serve(RequestServer.java:454) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:432) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at java.lang.Thread.run(Thread.java:745)

    JIRA | 2 years ago | Kevin Normoyle
    java.lang.RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122
  2. 0

    Not sure why this assertion is WARN? if it's an exception, it should be an ERROR? The assertion repros if the seed is used as shown. If not, I seem to get a bad json response, which is printed in red. I used a test that works on h2o that has minimal port to h2o-dev..i.e. it uses base h2o methods like import/parse/frames which I tested with other tests. the unique thing is this uses utf8 characters. I had sorted out all known issues with utf8 (i.e. what works) on h2o. I was suprised to see a bad json response from h2o-dev. I reduced the number of rows to get the domain values shorter for debug. this time, I got an assertion error from h2o-dev I'll leave the test as is (not sure it will complete but it should) it creates a random dataset. The seed can be forced with -s if needed to replicate ( I think not needed) actually I think I'll hardwire the seed in the test so you don't need the -s param shown. I also attached the file created in sandbox/syn_datasets to run cd h2o-dev/py2/testdir_multi_jvm (it does 2 jvms. don't know if that matters) python test_parse_rand_utf8.py -s 1364157389021990032 to run on existing cloud python test_parse_rand_utf8.py -s 1364157389021990032 -uc or python test_parse_rand_utf8.py -s 1364157389021990032 -uc localhost:54321 if you need to change the default 54321 port frames parameters: {'find_compatible_models': 0, 'len': 5, 'offset': 0} JSON call returned non-200 status with http://192.168.0.34:54323/3/Frames.json?find_compatible_models=0&len=5&offset=0 r.status_code: 500 r.headers: {'date': 'Sun, 16 Nov 2014 04:44:31 GMT', 'content-type': 'application/json'} r.text: {"schema_version":-1,"schema_name":"Schema","schema_type":"Iced","status_header":"500 Internal Server Error","errmsg":"RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122","errurl":null,"stacktrace":["water.MRTask.getResult(MRTask.java:261)","water.MRTask.doAll(MRTask.java:291)","water.MRTask.doAllNodes(MRTask.java:283)","water.KeySnapshot.globalSnapshot(KeySnapshot.java:208)","water.KeySnapshot.globalSnapshot(KeySnapshot.java:195)","water.api.FramesHandler$Frames.fetchAll(FramesHandler.java:29)","water.api.FramesHandler.list(FramesHandler.java:116)","sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)","sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:606)","water.api.Handler.handle(Handler.java:55)","water.api.RequestServer.handle(RequestServer.java:476)","water.api.RequestServer.serve(RequestServer.java:454)","water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:432)","java.lang.Thread.run(Thread.java:745)"]} ERROR 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: java.lang.RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.getResult(MRTask.java:261) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.doAll(MRTask.java:291) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.MRTask.doAllNodes(MRTask.java:283) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.KeySnapshot.globalSnapshot(KeySnapshot.java:208) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.KeySnapshot.globalSnapshot(KeySnapshot.java:195) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.FramesHandler$Frames.fetchAll(FramesHandler.java:29) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.FramesHandler.list(FramesHandler.java:116) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at java.lang.reflect.Method.invoke(Method.java:606) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.Handler.handle(Handler.java:55) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.RequestServer.handle(RequestServer.java:476) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.api.RequestServer.serve(RequestServer.java:454) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:432) 11-15 20:44:31.361 192.168.0.34:54323 8816 # Session WARN: at java.lang.Thread.run(Thread.java:745)

    JIRA | 2 years ago | Kevin Normoyle
    java.lang.RuntimeException: water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122
  3. 0

    Steps to reproduce: {code} val data = sc.parallelize((1 to 100000).map(_=>java.util.UUID.randomUUID().toString+" "+java.util.UUID.randomUUID().toString)) val schema = StructType(Seq(StructField("c1", StringType, true))) val rowRDD = data.map(Row(_)) val stringColumnTable = sqlContext.applySchema(rowRDD, schema) val h2oFrame = toDataFrame(stringColumnTable) scala> h2oFrame.vecs.map(_.get_type_str) res2: Array[String] = Array(string) scala> org.apache.spark.examples.h2o.DemoUtils.splitFrame(h2oFrame, Array("1","2","3"), Array(0.6,0.2,0.2)) onExCompletion for hex.splitframe.ShuffleSplitFrame$1@5d9445ed onExCompletion for hex.splitframe.ShuffleSplitFrame$1@5d9445ed water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) water.DException$DistributedException: from /192.168.1.4:54323; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.MRTask.compute2(MRTask.java:374) at water.MRTask.compute2(MRTask.java:374) at water.MRTask.compute2(MRTask.java:374) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.RuntimeException: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.MRTask.getResult(MRTask.java:265) at water.MRTask.doAll(MRTask.java:220) at water.MRTask.doAll(MRTask.java:217) at hex.splitframe.ShuffleSplitFrame.shuffleSplitFrame(ShuffleSplitFrame.java:29) at org.apache.spark.examples.h2o.DemoUtils$.splitFrame(DemoUtils.scala:110) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC$$iwC.<init>(<console>:48) at $iwC$$iwC.<init>(<console>:50) at $iwC.<init>(<console>:52) at <init>(<console>:54) at .<init>(<console>:58) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636) at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) {code} Reproduced on this build: http://h2o-release.s3.amazonaws.com/sparkling-water/master/86/index.html

    JIRA | 2 years ago | Peter Rudenko
    java.lang.RuntimeException: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Steps to reproduce: {code} val data = sc.parallelize((1 to 100000).map(_=>java.util.UUID.randomUUID().toString+" "+java.util.UUID.randomUUID().toString)) val schema = StructType(Seq(StructField("c1", StringType, true))) val rowRDD = data.map(Row(_)) val stringColumnTable = sqlContext.applySchema(rowRDD, schema) val h2oFrame = toDataFrame(stringColumnTable) scala> h2oFrame.vecs.map(_.get_type_str) res2: Array[String] = Array(string) scala> org.apache.spark.examples.h2o.DemoUtils.splitFrame(h2oFrame, Array("1","2","3"), Array(0.6,0.2,0.2)) onExCompletion for hex.splitframe.ShuffleSplitFrame$1@5d9445ed onExCompletion for hex.splitframe.ShuffleSplitFrame$1@5d9445ed water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) water.DException$DistributedException: from /192.168.1.4:54323; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.MRTask.compute2(MRTask.java:374) at water.MRTask.compute2(MRTask.java:374) at water.MRTask.compute2(MRTask.java:374) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.RuntimeException: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.MRTask.getResult(MRTask.java:265) at water.MRTask.doAll(MRTask.java:220) at water.MRTask.doAll(MRTask.java:217) at hex.splitframe.ShuffleSplitFrame.shuffleSplitFrame(ShuffleSplitFrame.java:29) at org.apache.spark.examples.h2o.DemoUtils$.splitFrame(DemoUtils.scala:110) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC$$iwC.<init>(<console>:48) at $iwC$$iwC.<init>(<console>:50) at $iwC.<init>(<console>:52) at <init>(<console>:54) at .<init>(<console>:58) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636) at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector. at water.fvec.CStrChunk.atd_impl(CStrChunk.java:39) at water.fvec.Chunk.atd(Chunk.java:252) at hex.splitframe.ShuffleSplitFrame$1.map(ShuffleSplitFrame.java:40) at water.MRTask.compute2(MRTask.java:430) at water.H2O$H2OCountedCompleter.compute(H2O.java:582) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) {code} Reproduced on this build: http://h2o-release.s3.amazonaws.com/sparkling-water/master/86/index.html

    JIRA | 2 years ago | Peter Rudenko
    java.lang.RuntimeException: water.DException$DistributedException: from /192.168.1.4:54327; by class hex.splitframe.ShuffleSplitFrame$1; class water.DException$DistributedException: from /192.168.1.4:54321; by class hex.splitframe.ShuffleSplitFrame$1; class java.lang.IllegalArgumentException: Operation not allowed on string vector.
  6. 0

    on mattd_scratch: x <- h2o.importFile("smalldata/jira/pubdev_2325_x.csv", col.types=list(by.col.name=c("shared1"),types=c("Enum"))) y <- h2o.importFile("smalldata/jira/pubdev_2325_y.csv", col.types=list(by.col.name=c("shared1"),types=c("Enum"))) h2o.merge(x=x, y=y, all.x=T, all.y=T, method="radix") gives: /99/Rapids, route: /99/Rapids, parms: {ast=(tmp= RTMP_10 (merge pubdev_XXXX_x.hex_8 pubdev_XXXX_y.hex_9 TRUE TRUE 1 1 "radix"))} Creating left index ... onExCompletion for water.rapids.RadixCount@49514b3a java.lang.IllegalArgumentException: at8_abs but value is missing at water.fvec.C2Chunk.at8_impl(C2Chunk.java:15) at water.fvec.Chunk.at8(Chunk.java:258) at water.rapids.RadixCount.map(RadixOrder.java:48) at water.MRTask.compute2(MRTask.java:653) 11-03 10:03:07.847 172.16.2.28:54321 42577 #76490-13 ERRR: java.lang.RuntimeException: water.DException$DistributedException: from /172.16.2.28:54321; by class water.rapids.RadixCount; class java.lang.IllegalArgumentException: at8_abs but value is missing at water.H2O$H2OCountedCompleter.compute(H2O.java:1065) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 11-03 10:03:07.848 172.16.2.28:54321 42577 #76490-13 ERRR: Caught exception: water.DException$DistributedException: from /172.16.2.28:54321; by class water.rapids.RadixCount; class java.lang.IllegalArgumentException: at8_abs but value is missing; Stacktrace: [water.MRTask.getResult(MRTask.java:503), water.MRTask.doAll(MRTask.java:377), water.MRTask.doAll(MRTask.java:364), water.MRTask.doAll(MRTask.java:363), water.rapids.RadixOrder.<init>(RadixOrder.java:576), water.rapids.Merge.merge(Merge.java:43), water.rapids.ASTMerge.sortingMerge(ASTMerge.java:174), water.rapids.ASTMerge.apply(ASTMerge.java:105), water.rapids.ASTExec.exec(ASTExec.java:44), water.rapids.ASTTmpAssign.apply(ASTAssign.java:241), water.rapids.ASTTmpAssign.apply(ASTAssign.java:235), water.rapids.ASTExec.exec(ASTExec.java:44), water.rapids.Exec.execute(Exec.java:51), water.rapids.Exec.exec(Exec.java:44), water.rapids.Exec.exec(Exec.java:38), water.api.RapidsHandler.exec(RapidsHandler.java:16), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:606), water.api.Handler.handle(Handler.java:64), water.api.RequestServer.handle(RequestServer.java:643), water.api.RequestServer.serve(RequestServer.java:584), water.JettyHTTPD$H2oDefaultServlet.doGeneric(JettyHTTPD.java:617), water.JettyHTTPD$H2oDefaultServlet.doPost(JettyHTTPD.java:565), javax.servlet.http.HttpServlet.service(HttpServlet.java:755), javax.servlet.http.HttpServlet.service(HttpServlet.java:848), org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)]

    JIRA | 1 year ago | Eric Eckstrand
    java.lang.RuntimeException: water.DException$DistributedException: from /172.16.2.28:54321; by class water.rapids.RadixCount; class java.lang.IllegalArgumentException: at8_abs but value is missing

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.RuntimeException

      water.DException$DistributedException: from /192.168.0.34:54323; by class water.KeySnapshot$GlobalUKeySetTask; class water.DException$DistributedException: from /192.168.0.34:54321; by class water.KeySnapshot$GlobalUKeySetTask; class java.lang.AssertionError: *** Attempting to block on task (class water.TaskGetKey) with equal or lower priority. Can lead to deadlock! 122 <= 122

      at water.MRTask.getResult()
    2. water
      KeySnapshot.globalSnapshot
      1. water.MRTask.getResult(MRTask.java:261)
      2. water.MRTask.doAll(MRTask.java:291)
      3. water.MRTask.doAllNodes(MRTask.java:283)
      4. water.KeySnapshot.globalSnapshot(KeySnapshot.java:208)
      5. water.KeySnapshot.globalSnapshot(KeySnapshot.java:195)
      5 frames
    3. water.api
      FramesHandler.list
      1. water.api.FramesHandler$Frames.fetchAll(FramesHandler.java:29)
      2. water.api.FramesHandler.list(FramesHandler.java:116)
      2 frames
    4. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:606)
      4 frames
    5. water.api
      RequestServer.serve
      1. water.api.Handler.handle(Handler.java:55)
      2. water.api.RequestServer.handle(RequestServer.java:476)
      3. water.api.RequestServer.serve(RequestServer.java:454)
      3 frames
    6. water
      NanoHTTPD$HTTPSession.run
      1. water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:432)
      1 frame
    7. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)
      1 frame