water.DException$DistributedException: from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name.

JIRA | Kevin Normoyle | 3 years ago
  1. 0

    cd testdir_multi_jvm python test_GBM_cancel_model_reuse.py starts 5 gbm jobs, then uses the jobs list to get id's to cancel all of them then repeats. The 2nd (and more) passes reuse the model keys from the first pass after the cancel, the job keys should be reusable, but they get the exception below gets stack trace on the 2nd pass 2014-02-06 20:59:36.453011 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad0&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.463305 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad1&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.471699 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad2&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.491283 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad3&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.510315 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad4&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:38.561355 -- Start http://192.168.0.7:54323/Jobs.json? 2014-02-06 20:59:38.645897 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_b879788d0e0cb142329256355414c77 2014-02-06 20:59:38.783900 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_ab576bde271a67c190b53ecf532d443d 2014-02-06 20:59:38.907229 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_aa38007d2111319a601e97c838b44be3 2014-02-06 20:59:38.991482 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_a46e367dbedd026e6a6d2d6805174a7a 2014-02-06 20:59:39.041373 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_b732df72604ff457a9766af25b354e36 2014-02-06 20:59:39.087883 -- Start http://192.168.0.7:54323/Jobs.json? 2014-02-06 20:59:39.141825 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad0&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:39.200337 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad1&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:39.236226 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad2&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 08:59:39.289 FJ-9-9 ERRR WATER: + water.DException$DistributedException: from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:57) + at water.DTask.dinvoke(DTask.java:78) + at water.RPC$RPCCall.compute2(RPC.java:276) + at water.H2O$H2OCountedCompleter.compute(H2O.java:712) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

    JIRA | 3 years ago | Kevin Normoyle
    water.DException$DistributedException: from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name.
  2. 0

    cd testdir_multi_jvm python test_GBM_cancel_model_reuse.py starts 5 gbm jobs, then uses the jobs list to get id's to cancel all of them then repeats. The 2nd (and more) passes reuse the model keys from the first pass after the cancel, the job keys should be reusable, but they get the exception below gets stack trace on the 2nd pass 2014-02-06 20:59:36.453011 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad0&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.463305 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad1&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.471699 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad2&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.491283 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad3&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:36.510315 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad4&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:38.561355 -- Start http://192.168.0.7:54323/Jobs.json? 2014-02-06 20:59:38.645897 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_b879788d0e0cb142329256355414c77 2014-02-06 20:59:38.783900 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_ab576bde271a67c190b53ecf532d443d 2014-02-06 20:59:38.907229 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_aa38007d2111319a601e97c838b44be3 2014-02-06 20:59:38.991482 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_a46e367dbedd026e6a6d2d6805174a7a 2014-02-06 20:59:39.041373 -- Start http://192.168.0.7:54323/Cancel.json?key=$0301c0a8000734d4ffffffff$_b732df72604ff457a9766af25b354e36 2014-02-06 20:59:39.087883 -- Start http://192.168.0.7:54323/Jobs.json? 2014-02-06 20:59:39.141825 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad0&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:39.200337 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad1&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 2014-02-06 20:59:39.236226 -- Start http://192.168.0.7:54323/2/GBM.json?learn_rate=0.1&destination_key=GBMBad2&classification=0&min_rows=1&ntrees=2&response=C379&ignored_cols_by_name=C4,C5,C6,C7,C8,C9,C10,C11,C12,C15,C17,C18,C19,C20,C21,C425,C426,C427,C541,C542,C379&source=c.hex&grid_parallelism=4&max_depth=8 08:59:39.289 FJ-9-9 ERRR WATER: + water.DException$DistributedException: from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:57) + at water.DTask.dinvoke(DTask.java:78) + at water.RPC$RPCCall.compute2(RPC.java:276) + at water.H2O$H2OCountedCompleter.compute(H2O.java:712) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

    JIRA | 3 years ago | Kevin Normoyle
    water.DException$DistributedException: from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name.
  3. 0
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    (not shown here, but a prior parse got a stack trace, and apparently left a key locked) I'm transitioning to using remove_all()...so I'm just noting this in case we want remove_key to remove even if a key is locked (say due to an error) or whether we want a -force param details the test suite is running multiple tests on a single cloud...so I'm not sure what's going on now with the new behavior around locked keys. I used to remove all keys between tests. I probably have to update to the remove_all that forces key removal? this is an interesting message though. ...we probably don't wnat to cause exception on this? does this just send exception info to the browser, with no bad side effect on h2o (no crash? from /192.168.1.172:54355; java.lang.IllegalArgumentException: Dataset nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:83) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:72) /var/lib/jenkins/jobs/h2o_release_tests/workspace/py/testdir_release/c7/test_c7_rel.py check_sandbox_for_errors: Errors in sandbox stdout or stderr (or R stdout/stderr). Could have occurred at any prior time 10:01:12.429 # Session ERRR WATER: + water.DException$DistributedException: from /192.168.1.177:54355; java.lang.NullPointerException 10:01:12.435 # Session INFO HTTPD: GET /Remove.json key=nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz 10:01:12.437 # Session ERRR WATER: + water.DException$DistributedException: from /192.168.1.172:54355; java.lang.IllegalArgumentException: Dataset nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:83) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:72) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:57) + at water.DTask.dinvoke(DTask.java:78) + at water.RPC$RPCCall.compute2(RPC.java:276) + at water.H2O$H2OCountedCompleter.compute(H2O.java:712) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 10:01:12.444 # Session INFO HTTPD: GET /Remove.json key=nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz

    JIRA | 3 years ago | Kevin Normoyle
    water.DException$DistributedException: from /192.168.1.172:54355; java.lang.IllegalArgumentException: Dataset nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz is already in use. Unable to use it now. Consider using a different destination name.
  6. 0

    From [~kbn]: nishant got this GBM "Trying to unlock null" assertion during pyunit_citi_bike_large.py. Seems like it's a delete of some key related to GBM. test seemed to keep going though. from http://mr-0xa1:8080/view/nishant/job/nishant_code_coverage/41/artifact/h2o-py/tests/results/java_0_0.out.txt He later got other assertions that have appeared elsewhere with the pyunit_citi_bike_large.py here's the one I hadn't seen before: 06-30 18:01:35.534 172.17.2.154:56789 3951 # Session INFO: Method: GET , URI: /3/Models/GBMModel__8c033c5ded17b06a9f57036a08014faa, route: /3/Models/(?<modelid>.*), parms: {model_id=GBMModel__8c033c5ded17b06a9f57036a08014faa} 06-30 18:01:35.541 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'pyfdf9ce18-09a0-4dff-98ea-353bf6c7e119')} 06-30 18:01:54.250 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(removeframe 'py3b2531b3-1876-4756-8e0b-454f46b87fb3')} 06-30 18:01:54.286 172.17.2.154:56789 3951 # Session INFO: Method: DELETE, URI: /3/DKV/GBMModel__ae8e7b4651349614921bec0629064c9b, route: /3/DKV/(?<key>.*), parms: {key=GBMModel__ae8e7b4651349614921bec0629064c9b} barrier onExCompletion for hex.tree.gbm.GBM$GBMDriver@6c5d2aea water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null! at water.Lockable$Unlock.atomic(Lockable.java:180) at water.Lockable$Unlock.atomic(Lockable.java:176) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.H2O$H2OCountedCompleter.compute(H2O.java:698) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 06-30 18:01:54.570 172.17.2.154:56789 3951 # Session INFO: Method: POST , URI: /99/Rapids, route: /99/Rapids, parms: {ast=(, (gput py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (cbind %FALSE 'py99fef45f-7d80-41b3-88df-e65c578e2677' 'py1844e0f1-6095-4b69-9e03-8c3b3a9bd336' 'py82e77fc3-c2d4-4093-b77f-c983ced3e0c4' 'py554158f7-9142-469d-bc9b-2d1479b2b118' 'pyce700595-02fb-455b-a13d-6d4ea0f978f3' 'pyc37f39b1-ba43-4eed-bb0d-9f9e6bc19032' 'pyfa208086-445e-4507-bff5-b42a50f6b1ed' 'pybf318908-1f6b-4625-9a3c-2cd1a24affd9' 'pyf97296f1-6bb5-485f-b77b-ad224f498648' 'py74884c42-f4f2-443e-b015-9c736069f699')) (colnames= %py25ead33e-ba45-4f4a-ae45-04ed5f7dbab3 (: #0 #9) (slist "Days" "start station name" "Month" "DayOfWeek" "Humidity Fraction" "Rain (mm)" "Temperature (C)" "WC1" "Dew Point (C)" "bikes")}

    JIRA | 1 year ago | Raymond Peck
    water.DException$DistributedException: from /172.17.2.154:56793; by class water.Lockable$Unlock; class java.lang.AssertionError: Trying to unlock null!

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. water.DException$DistributedException

      from /192.168.0.7:54321; java.lang.IllegalArgumentException: Model GBMBad2 is already in use. Unable to use it now. Consider using a different destination name.

      at water.Lockable$PriorWriteLock.atomic()
    2. water
      H2O$H2OCountedCompleter.compute
      1. water.Lockable$PriorWriteLock.atomic(Lockable.java:84)
      2. water.Lockable$PriorWriteLock.atomic(Lockable.java:73)
      3. water.TAtomic.atomic(TAtomic.java:19)
      4. water.Atomic.compute2(Atomic.java:57)
      5. water.DTask.dinvoke(DTask.java:78)
      6. water.RPC$RPCCall.compute2(RPC.java:276)
      7. water.H2O$H2OCountedCompleter.compute(H2O.java:712)
      7 frames
    3. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames