java.lang.IllegalArgumentException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Parse and big dataset, run a big model, say lambda search. While the first job is running change alpha value in the glm frame, and fire another job with same destination frame. Get- Got exception 'class java.lang.IllegalArgumentException', with msg 'class hex.glm.GLMModel glm-ccedb30b-839b-43ce-8395-3fdec800c9f2 is already in use. Unable to use it now. Consider using a different destination name.' java.lang.IllegalArgumentException: class hex.glm.GLMModel glm-ccedb30b-839b-43ce-8395-3fdec800c9f2 is already in use. Unable to use it now. Consider using a different destination name. at water.Lockable$PriorWriteLock.atomic(Lockable.java:109) at water.Lockable$PriorWriteLock.atomic(Lockable.java:98) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.write_lock(Lockable.java:59) at water.Lockable.delete_and_lock(Lockable.java:66) at hex.glm.GLM.init(GLM.java:315) at hex.glm.GLM$GLMDriver.compute2(GLM.java:625) at water.H2O$H2OCountedCompleter.compute(H2O.java:682) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) which is good. But when the original (1st job) gets done (it shows a status as done ) Get- Error calling GET /3/Models/glm-ccedb30b-839b-43ce-8395-3fdec800c9f2 with opts null Object 'glm-ccedb30b-839b-43ce-8395-3fdec800c9f2' not found for argument: key TOGGLE STACK TRACE Object 'glm-ccedb30b-839b-43ce-8395-3fdec800c9f2' not found for argument: key (water.exceptions.H2OKeyNotFoundArgumentException) water.api.ModelsHandler.getFromDKV(ModelsHandler.java:105) water.api.ModelsHandler.fetch(ModelsHandler.java:124) sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) water.api.Handler.handle(Handler.java:57) water.api.RequestServer.handle(RequestServer.java:675) water.api.RequestServer.serve(RequestServer.java:611) water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:436) java.lang.Thread.run(Thread.java:744) Flow steps are in the attached file.
    via by Nidhi Mehta,
  • This looks like a test failure. The test will randomly have a name collision on model names. Or is this a cleanup problem? ./h2o-py/tests/testdir_algos/kmeans/pyunit_random_attack_medium.py Failure example: http://172.16.2.161:8080/job/h2o_master_DEV_win8_pyunit_medium_large/1248/artifact/h2o-py/tests/results/java_0_0.out.txt 11-01 09:37:46.444 172.17.6.15:56789 2404 FJ-0-5 INFO: {"_model_id":{"name":"my_model","type":"Key"},"_train":{"name":"ozone.hex","type":"Key"},"_valid":null,"_nfolds":0,"_keep_cross_validation_predictions":false,"_fold_assignment":"AUTO","_distribution":"AUTO","_tweedie_power":1.5,"_ignored_columns":["wind"],"_ignore_const_cols":true,"_weights_column":null,"_offset_column":null,"_fold_column":null,"_score_each_iteration":false,"_stopping_rounds":0,"_stopping_metric":"AUTO","_stopping_tolerance":0.001,"_response_column":null,"_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_hit_ratio_k":10,"_max_confusion_matrix_size":20,"_checkpoint":null,"_k":20,"_max_iterations":1000,"_standardize":false,"_seed":8718,"_init":"Furthest","_user_points":null,"_pred_indicator":false} 11-01 09:37:46.444 172.17.6.15:56789 2404 FJ-0-5 INFO: Dropping ignored columns: [wind] java.lang.IllegalArgumentException: class hex.kmeans.KMeansModel my_model is already in use. Unable to use it now. Consider using a different destination name. at water.Lockable$PriorWriteLock.atomic(Lockable.java:109) at water.Lockable$PriorWriteLock.atomic(Lockable.java:98) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.write_lock(Lockable.java:59) at water.Lockable.delete_and_lock(Lockable.java:66) at hex.kmeans.KMeans$KMeansDriver.compute2(KMeans.java:269) at water.H2O$H2OCountedCompleter.compute(H2O.java:1065) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) barrier onExCompletion for hex.kmeans.KMeans$KMeansDriver@60442d4a java.lang.AssertionError: Can't unlock: Not locked! at water.Lockable$Unlock.atomic(Lockable.java:187) at water.Lockable$Unlock.atomic(Lockable.java:177) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.unlock(Lockable.java:172) at water.Lockable.unlock(Lockable.java:168) at hex.kmeans.KMeans$KMeansDriver.compute2(KMeans.java:332) at water.H2O$H2OCountedCompleter.compute(H2O.java:1065) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    via by Brandon Hill,
  • This looks like a test failure. The test will randomly have a name collision on model names. Or is this a cleanup problem? ./h2o-py/tests/testdir_algos/kmeans/pyunit_random_attack_medium.py Failure example: http://172.16.2.161:8080/job/h2o_master_DEV_win8_pyunit_medium_large/1248/artifact/h2o-py/tests/results/java_0_0.out.txt 11-01 09:37:46.444 172.17.6.15:56789 2404 FJ-0-5 INFO: {"_model_id":{"name":"my_model","type":"Key"},"_train":{"name":"ozone.hex","type":"Key"},"_valid":null,"_nfolds":0,"_keep_cross_validation_predictions":false,"_fold_assignment":"AUTO","_distribution":"AUTO","_tweedie_power":1.5,"_ignored_columns":["wind"],"_ignore_const_cols":true,"_weights_column":null,"_offset_column":null,"_fold_column":null,"_score_each_iteration":false,"_stopping_rounds":0,"_stopping_metric":"AUTO","_stopping_tolerance":0.001,"_response_column":null,"_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_hit_ratio_k":10,"_max_confusion_matrix_size":20,"_checkpoint":null,"_k":20,"_max_iterations":1000,"_standardize":false,"_seed":8718,"_init":"Furthest","_user_points":null,"_pred_indicator":false} 11-01 09:37:46.444 172.17.6.15:56789 2404 FJ-0-5 INFO: Dropping ignored columns: [wind] java.lang.IllegalArgumentException: class hex.kmeans.KMeansModel my_model is already in use. Unable to use it now. Consider using a different destination name. at water.Lockable$PriorWriteLock.atomic(Lockable.java:109) at water.Lockable$PriorWriteLock.atomic(Lockable.java:98) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.write_lock(Lockable.java:59) at water.Lockable.delete_and_lock(Lockable.java:66) at hex.kmeans.KMeans$KMeansDriver.compute2(KMeans.java:269) at water.H2O$H2OCountedCompleter.compute(H2O.java:1065) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) barrier onExCompletion for hex.kmeans.KMeans$KMeansDriver@60442d4a java.lang.AssertionError: Can't unlock: Not locked! at water.Lockable$Unlock.atomic(Lockable.java:187) at water.Lockable$Unlock.atomic(Lockable.java:177) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.unlock(Lockable.java:172) at water.Lockable.unlock(Lockable.java:168) at hex.kmeans.KMeans$KMeansDriver.compute2(KMeans.java:332) at water.H2O$H2OCountedCompleter.compute(H2O.java:1065) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    via by Brandon Hill,
  • Cancel the job after say 30 sec, and fire another one with the same key name get Got exception 'class java.lang.IllegalArgumentException', with msg 'class hex.pca.PCAModel pca-de15afbe-4ac8-4908-8e4d-f173bda2799f is already in use. Unable to use it now. Consider using a different destination name.' java.lang.IllegalArgumentException: class hex.pca.PCAModel pca-de15afbe-4ac8-4908-8e4d-f173bda2799f is already in use. Unable to use it now. Consider using a different destination name. at water.Lockable$PriorWriteLock.atomic(Lockable.java:109) at water.Lockable$PriorWriteLock.atomic(Lockable.java:98) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.write_lock(Lockable.java:59) at water.Lockable.delete_and_lock(Lockable.java:66) at hex.pca.PCA$PCADriver.compute2(PCA.java:198) at water.H2O$H2OCountedCompleter.compute(H2O.java:666) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) buildModel 'pca', {"model_id":"pca-de15afbe-4ac8-4908-8e4d-f173bda2799f","training_frame":"Key_Frame__DatasetB_675gene.hex","ignore_const_cols":true,"score_each_iteration":false,"transform":"NONE","k":"50","max_iterations":1000,"seed":1432760733996723000}
    via by Nidhi Mehta,
  • Here I do back to back postfiles (or overlapped actually on different nodes) of the same file (iris) with different key on the postfile for the source, and different destination_key for the parsed result. I had thought that was enough to guarantee they were independent, since the visible created keys were different names here's the commands.log showing what I'm doing 2014-03-29 15:16:51.917781 -- Start http://10.71.0.101:54321/2/PostFile.json?key=iris2.csv_1 #/home/0xdiag/h2o/smalldata/iris/iris2.csv 2014-03-29 15:16:51.917919 -- Start http://10.71.0.100:54321/2/PostFile.json?key=iris2.csv_0 #/home/0xdiag/h2o/smalldata/iris/iris2.csv 2014-03-29 15:16:51.918169 -- Start http://10.71.0.100:54321/2/PostFile.json?key=iris2.csv_2 #/home/0xdiag/h2o/smalldata/iris/iris2.csv 2014-03-29 15:16:51.970297 -- Start http://10.71.0.101:54321/2/Parse2.json?destination_key=iris2.csv_2.hex&source_key=iris2.csv_2 2014-03-29 15:16:51.985916 -- Start http://10.71.0.100:54321/2/Parse2.json?destination_key=iris2.csv_1.hex&source_key=iris2.csv_1 2014-03-29 15:16:51.991497 -- Start http://10.71.0.101:54321/2/Parse2.json?destination_key=iris2.csv_0.hex&source_key=iris2.csv_0 I get this in stdout. It's possible h2o has some internal state it's protecting, that's locked to the actual filename. So either there's a real lock, or this is incorrectly detecting a collison when there shouldn't be one. java.lang.IllegalArgumentException: Dataset iris2.csv_0.hex is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:57) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:59) + at water.Lockable.delete_and_lock(Lockable.java:63) + at water.fvec.ParseDataset2.forkParseDataset(ParseDataset2.java:58) + at water.api.Parse2.serve(Parse2.java:43) + at water.api.Request.serveGrid(Request.java:133) + at water.api.Request.serve(Request.java:110) + at water.api.RequestServer.serve(RequestServer.java:327) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + at java.lang.Thread.run(Thread.java:744) If I create hard links with different names (pointing to the same file) I can do this test successfully (with covtype.data and covtype20x.data also) with up to 10 parallel uploads/parses overlapping. (all with different names for keys as above, with the links providing different src file names also)
    via by Kevin Normoyle,
  • On master -Build git hash:31e95f76d45a7e7848375361d16fc9fe9e68b6ff pp = h2o.importFile(h,"/Users/nidhimehta/h2o/smalldata/logreg/prostate.csv",key = "pp") |================================================================================| 100% > pp$GLEASON =as.factor(pp$GLEASON) > aa = h2o.prcomp(pp[,3:9],key="aa",standardize=T,retx=T) |================================================================================| 100% |================================================================================| 100% > dd = h2o.importFile(h,"/Users/nidhimehta/h2o/smalldata/logreg/prostate.csv",key = "dd") |================================================================================| 100% > h2o.predict(aa,newdata=dd) | | 0% Polling fails: <simpleError in .h2o.__poll(client, job_key): Illegal argument: Incompatible column: 'GLEASON', expected (trained on) categorical, was passed a numeric> |================================================================================| 100% IP Address: 10.0.0.3 Port : 54321 Parsed Data Key: PCAPredict_76abb42620ba4fbf971f3926bfc6fb2e data frame with 0 columns and 0 rows > aa = h2o.prcomp(pp[,3:9],key="aa",standardize=T,retx=T,max_pc=2) Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://10.0.0.3:54321/2/Exec2.json returned the following error: Frame is already locked by job null. > 14:59:25.548 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset PCAPredict_76abb42620ba4fbf971f3926bfc6fb2e is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.Lockable.delete_and_lock(Lockable.java:64) + at water.exec.Env.remove_and_unlock(Env.java:371) + at water.api.Exec2.serve(Exec2.java:74) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744) 14:59:28.111 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.11 = pp[,c(3,4,5,6,7,8,9)] 14:59:28.111 # Session ERRR WATER: + java.lang.AssertionError: Frame is already locked by job null. + at water.Lockable.set_read_lock(Lockable.java:209) + at water.Lockable.access$400(Lockable.java:23) + at water.Lockable$ReadLock.atomic(Lockable.java:143) + at water.Lockable$ReadLock.atomic(Lockable.java:136) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.read_lock(Lockable.java:131) + at water.exec.Exec2.exec(Exec2.java:81) + at water.api.Exec2.serve(Exec2.java:37) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744) 14:59:28.112 # Session ERRR WATER: + java.lang.AssertionError: Frame is already locked by job null. + at water.Lockable.set_read_lock(Lockable.java:209) + at water.Lockable.access$400(Lockable.java:23) + at water.Lockable$ReadLock.atomic(Lockable.java:143) + at water.Lockable$ReadLock.atomic(Lockable.java:136) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.read_lock(Lockable.java:131) + at water.exec.Exec2.exec(Exec2.java:81) + at water.api.Exec2.serve(Exec2.java:37) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744)
    via by Nidhi Mehta,
  • On master -Build git hash:31e95f76d45a7e7848375361d16fc9fe9e68b6ff pp = h2o.importFile(h,"/Users/nidhimehta/h2o/smalldata/logreg/prostate.csv",key = "pp") |================================================================================| 100% > pp$GLEASON =as.factor(pp$GLEASON) > aa = h2o.prcomp(pp[,3:9],key="aa",standardize=T,retx=T) |================================================================================| 100% |================================================================================| 100% > dd = h2o.importFile(h,"/Users/nidhimehta/h2o/smalldata/logreg/prostate.csv",key = "dd") |================================================================================| 100% > h2o.predict(aa,newdata=dd) | | 0% Polling fails: <simpleError in .h2o.__poll(client, job_key): Illegal argument: Incompatible column: 'GLEASON', expected (trained on) categorical, was passed a numeric> |================================================================================| 100% IP Address: 10.0.0.3 Port : 54321 Parsed Data Key: PCAPredict_76abb42620ba4fbf971f3926bfc6fb2e data frame with 0 columns and 0 rows > aa = h2o.prcomp(pp[,3:9],key="aa",standardize=T,retx=T,max_pc=2) Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://10.0.0.3:54321/2/Exec2.json returned the following error: Frame is already locked by job null. > 14:59:25.548 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset PCAPredict_76abb42620ba4fbf971f3926bfc6fb2e is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.Lockable.delete_and_lock(Lockable.java:64) + at water.exec.Env.remove_and_unlock(Env.java:371) + at water.api.Exec2.serve(Exec2.java:74) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744) 14:59:28.111 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.11 = pp[,c(3,4,5,6,7,8,9)] 14:59:28.111 # Session ERRR WATER: + java.lang.AssertionError: Frame is already locked by job null. + at water.Lockable.set_read_lock(Lockable.java:209) + at water.Lockable.access$400(Lockable.java:23) + at water.Lockable$ReadLock.atomic(Lockable.java:143) + at water.Lockable$ReadLock.atomic(Lockable.java:136) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.read_lock(Lockable.java:131) + at water.exec.Exec2.exec(Exec2.java:81) + at water.api.Exec2.serve(Exec2.java:37) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744) 14:59:28.112 # Session ERRR WATER: + java.lang.AssertionError: Frame is already locked by job null. + at water.Lockable.set_read_lock(Lockable.java:209) + at water.Lockable.access$400(Lockable.java:23) + at water.Lockable$ReadLock.atomic(Lockable.java:143) + at water.Lockable$ReadLock.atomic(Lockable.java:136) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.read_lock(Lockable.java:131) + at water.exec.Exec2.exec(Exec2.java:81) + at water.api.Exec2.serve(Exec2.java:37) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:490) + at water.api.Exec2.serveGrid(Exec2.java:81) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:502) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:425) + at java.lang.Thread.run(Thread.java:744)
    via by Nidhi Mehta,
  • (not shown here, but a prior parse got a stack trace, and apparently left a key locked) I'm transitioning to using remove_all()...so I'm just noting this in case we want remove_key to remove even if a key is locked (say due to an error) or whether we want a -force param details the test suite is running multiple tests on a single cloud...so I'm not sure what's going on now with the new behavior around locked keys. I used to remove all keys between tests. I probably have to update to the remove_all that forces key removal? this is an interesting message though. ...we probably don't wnat to cause exception on this? does this just send exception info to the browser, with no bad side effect on h2o (no crash? from /192.168.1.172:54355; java.lang.IllegalArgumentException: Dataset nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:83) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:72) /var/lib/jenkins/jobs/h2o_release_tests/workspace/py/testdir_release/c7/test_c7_rel.py check_sandbox_for_errors: Errors in sandbox stdout or stderr (or R stdout/stderr). Could have occurred at any prior time 10:01:12.429 # Session ERRR WATER: + water.DException$DistributedException: from /192.168.1.177:54355; java.lang.NullPointerException 10:01:12.435 # Session INFO HTTPD: GET /Remove.json key=nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz 10:01:12.437 # Session ERRR WATER: + water.DException$DistributedException: from /192.168.1.172:54355; java.lang.IllegalArgumentException: Dataset nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz/file_100.dat.gz is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:83) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:72) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:57) + at water.DTask.dinvoke(DTask.java:78) + at water.RPC$RPCCall.compute2(RPC.java:276) + at water.H2O$H2OCountedCompleter.compute(H2O.java:712) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 10:01:12.444 # Session INFO HTTPD: GET /Remove.json key=nfs://home/0xcustomer/home-0xdiag-datasets/manyfiles-nflx-gz
    via by Kevin Normoyle,
  • There is an error in unlocking the dataset after the previous model fails. The error in the previous model is a javaOutOfMemory and the following model is successfully created, despite the locked job error. The only issue is that that R unit test would fail overall. The test is too long to run, so the script for repo is attached. Output from logs: 05:22:21.748 # Session INFO HTTPD: POST /2/Exec2.json str=arcene.train.full = arcene.train.full 05:22:21.879 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset arcene.train.full is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:482) + at water.api.Exec2.serveGrid(Exec2.java:78) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:484) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:424) + at java.lang.Thread.run(Thread.java:745) Output from test: [2014-08-14 17:37:37] [ERROR] : Error: Test failed: 'Testing memory performance of Strong Rules' Not expected: http://127.0.0.1:54321/2/Exec2.json returned the following error: Frame is already locked by job null. 1: withWarnings(test(conn)) 2: withCallingHandlers(expr, warning = wHandler) 3: test(conn) 4: h2o.glm(x = c(1:7000), y = "arcene.train.label", data = arcene.train.full, family = "binomial", lambda_search = T, alpha = 1, nfolds = 0, use_all_factor_levels = T) 5: .h2o.get.glm(data@h2o, as.character(res$destination_key), return_all_lambda) 6: h2o.getFrame(h2o, pre$json$glm_model$dataKey) 7: .h2o.exec2(expr = key, h2o = h2o, dest_key = key) 8: .h2o.__exec2_dest_key(h2o, expr, dest_key) 9: .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) 10: stop(paste(myURL, " returned the following error:\n", .h2o.__formatError(res$error))) 11: .handleSimpleError(function (e) { e$calls <- head(sys.calls()[-seq_len(frame + 7)], -2) signalCondition(e) }, "http://127.0.0.1:54321/2/Exec2.json returned the following error:\n Frame is already locked by job null.\n", quote(.h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr))).
    via by Ariel Rao,
  • it's not necessarily a good thing to do, but we should support it [load a dataset; call it d] d$col2000 <- min(d$col2000) Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://127.0.0.1:54321/2/Exec2.json returned the following error: 2 better yet… once you've done this, stuff breaks everywhere > d$col2000 <- min(d$col2000) + d$col2000 Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://127.0.0.1:54321/2/Exec2.json returned the following error: Dataset Last.value.4 is already in use. Unable to use it now. Consider using a different destination name. > head(d) Error in data.frame(col1 = 0.542521227151155, col2 = 0.267292166827247, : arguments imply differing number of rows: 1, 0 > 04:23:10.668 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.8 = bad_2k.hex[,2000] = c(0.000574313336983323) 04:23:10.747 # Session ERRR WATER:▫ + java.lang.ArrayIndexOutOfBoundsException: 2 + ▸ at water.fvec.Vec.elem2ChunkIdx(Vec.java:407) + ▸ at water.fvec.Vec.chunk(Vec.java:505) + ▸ at water.fvec.Vec.at(Vec.java:513) + ▸ at water.fvec.Frame.toString(Frame.java:475) + ▸ at water.api.Exec2.serve(Exec2.java:50) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) 04:23:10.748 # Session ERRR WATER:▫ + java.lang.ArrayIndexOutOfBoundsException: 2 + ▸ at water.fvec.Vec.elem2ChunkIdx(Vec.java:407) + ▸ at water.fvec.Vec.chunk(Vec.java:505) + ▸ at water.fvec.Vec.at(Vec.java:513) + ▸ at water.fvec.Frame.toString(Frame.java:475) + ▸ at water.api.Exec2.serve(Exec2.java:50) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) 04:23:51.514 # Session INFO HTTPD: POST /Inspect.json key=bad_2k.hex 04:23:51.633 # Session INFO HTTPD: POST /Inspect.json key=bad_2k.hex max_column_display=2147483647 04:23:51.753 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.9 = bad_2k.hex[,2000] 04:23:51.768 # Session ERRR WATER:▫ + java.lang.IllegalArgumentException: Dataset Last.value.4 is already in use. Unable to use it now. Consider using a different destination name. + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + ▸ at water.TAtomic.atomic(TAtomic.java:19) + ▸ at water.Atomic.compute2(Atomic.java:57) + ▸ at water.Atomic.fork(Atomic.java:42) + ▸ at water.Atomic.invoke(Atomic.java:34) + ▸ at water.Lockable.write_lock(Lockable.java:59) + ▸ at water.exec.Env.remove_and_unlock(Env.java:326) + ▸ at water.api.Exec2.serve(Exec2.java:62) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724)
    via by Earl Hathaway,
    • java.lang.IllegalArgumentException: class hex.glm.GLMModel glm-ccedb30b-839b-43ce-8395-3fdec800c9f2 is already in use. Unable to use it now. Consider using a different destination name. at water.Lockable$PriorWriteLock.atomic(Lockable.java:109) at water.Lockable$PriorWriteLock.atomic(Lockable.java:98) at water.TAtomic.atomic(TAtomic.java:17) at water.Atomic.compute2(Atomic.java:55) at water.Atomic.fork(Atomic.java:39) at water.Atomic.invoke(Atomic.java:31) at water.Lockable.write_lock(Lockable.java:59) at water.Lockable.delete_and_lock(Lockable.java:66) at hex.glm.GLM.init(GLM.java:315) at hex.glm.GLM$GLMDriver.compute2(GLM.java:625) at water.H2O$H2OCountedCompleter.compute(H2O.java:682) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    No Bugmate found.