java.lang.IllegalArgumentException: Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name.

JIRA | Tom Kraljevic | 2 years ago
  1. 0

    vramanathan00@gmail.com reports On Sep 9, 2014, at 11:08 AM, vramanathan00@gmail.com wrote: Here's the use case Single server/instance h2o 1. I import dataset using REST API 2. Kick off training (DL) using REST API 3. Launch R -> -> data.train = h2o.getFrame(conn, '...') I get this exception..Is n't getFrame(..) read only or it's trying to update something while reading? Would this also prevent running multiple models on the same data frame?..THANKS java.lang.IllegalArgumentException: Dataset .... is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:481) + at water.api.Exec2.serveGrid(Exec2.java:78) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:479) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:424) + at java.lang.Thread.run(Thread.java:724) h2ostream, August 27, 2014 Thanks Tom..Regarding logs, I'm running long running job on large data set..I'll see if I can reproduce & resend logs.. In the interim, here are the steps 1) Launch H2o (single instance) 2) Import Files: curl .../ImportFiles2.json 3) Parse fies: curl ../Parse2.json 4) kick off deep learning: curl ../DeepLearning.json .....while that's running..it runs for long... 5) Start R session 6) connect to h2o 7) h2o.getFrame("train.hex"...) 8) summary(train.hex) ------------------------------------- On Wednesday, August 27, 2014 12:59:04 PM UTC-7, Tom Kraljevic wrote: ah, ok. the R api is synchronous, so this specific thing is not really tested today. the locking stuff is intended to prevent concurrent things from damaging each other. if you give us the high level list of steps (or logs), we can try to repro them and see if there is a bug, but based on the info i think this is the expected behavior. (dataset probably wont matter.) the stack trace might be a little ugly and misleading though. thanks tom Sent from my iPhone On Aug 27, 2014, at 12:43 PM, vramanathan00@gmail.com wrote: I started the training job directly using REST API.. I tried to access the data (essentially, .getFrame(...)) using R, while the job is still going.. Concurrent access is allowed correct? On Wednesday, August 27, 2014 12:29:57 PM UTC-7, Tom Kraljevic wrote: by chance, did you start a job from the web ui and then try to access it from r before it completed? this error looks kind of consistent with that. Sent from my iPhone On Aug 27, 2014, at 11:23 AM, vramanathan00@gmail.com wrote: Trying to access from R..while building model.. 11:19:13.699 # Session INFO WATER: ],"consensus":true,"locked":true 11:19:13.706 # Session INFO HTTPD: POST /2/Exec2.json str=segmentaa.hex = segmentaa.hex 11:19:17.247 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:481) --

    JIRA | 2 years ago | Tom Kraljevic
    java.lang.IllegalArgumentException: Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name.
  2. 0

    vramanathan00@gmail.com reports On Sep 9, 2014, at 11:08 AM, vramanathan00@gmail.com wrote: Here's the use case Single server/instance h2o 1. I import dataset using REST API 2. Kick off training (DL) using REST API 3. Launch R -> -> data.train = h2o.getFrame(conn, '...') I get this exception..Is n't getFrame(..) read only or it's trying to update something while reading? Would this also prevent running multiple models on the same data frame?..THANKS java.lang.IllegalArgumentException: Dataset .... is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:481) + at water.api.Exec2.serveGrid(Exec2.java:78) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:479) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:424) + at java.lang.Thread.run(Thread.java:724) h2ostream, August 27, 2014 Thanks Tom..Regarding logs, I'm running long running job on large data set..I'll see if I can reproduce & resend logs.. In the interim, here are the steps 1) Launch H2o (single instance) 2) Import Files: curl .../ImportFiles2.json 3) Parse fies: curl ../Parse2.json 4) kick off deep learning: curl ../DeepLearning.json .....while that's running..it runs for long... 5) Start R session 6) connect to h2o 7) h2o.getFrame("train.hex"...) 8) summary(train.hex) ------------------------------------- On Wednesday, August 27, 2014 12:59:04 PM UTC-7, Tom Kraljevic wrote: ah, ok. the R api is synchronous, so this specific thing is not really tested today. the locking stuff is intended to prevent concurrent things from damaging each other. if you give us the high level list of steps (or logs), we can try to repro them and see if there is a bug, but based on the info i think this is the expected behavior. (dataset probably wont matter.) the stack trace might be a little ugly and misleading though. thanks tom Sent from my iPhone On Aug 27, 2014, at 12:43 PM, vramanathan00@gmail.com wrote: I started the training job directly using REST API.. I tried to access the data (essentially, .getFrame(...)) using R, while the job is still going.. Concurrent access is allowed correct? On Wednesday, August 27, 2014 12:29:57 PM UTC-7, Tom Kraljevic wrote: by chance, did you start a job from the web ui and then try to access it from r before it completed? this error looks kind of consistent with that. Sent from my iPhone On Aug 27, 2014, at 11:23 AM, vramanathan00@gmail.com wrote: Trying to access from R..while building model.. 11:19:13.699 # Session INFO WATER: ],"consensus":true,"locked":true 11:19:13.706 # Session INFO HTTPD: POST /2/Exec2.json str=segmentaa.hex = segmentaa.hex 11:19:17.247 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:481) --

    JIRA | 2 years ago | Tom Kraljevic
    java.lang.IllegalArgumentException: Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name.
  3. 0

    There is an error in unlocking the dataset after the previous model fails. The error in the previous model is a javaOutOfMemory and the following model is successfully created, despite the locked job error. The only issue is that that R unit test would fail overall. The test is too long to run, so the script for repo is attached. Output from logs: 05:22:21.748 # Session INFO HTTPD: POST /2/Exec2.json str=arcene.train.full = arcene.train.full 05:22:21.879 # Session ERRR WATER: env.remove_and_unlock() failed + java.lang.IllegalArgumentException: Dataset arcene.train.full is already in use. Unable to use it now. Consider using a different destination name. + at water.Lockable$PriorWriteLock.atomic(Lockable.java:85) + at water.Lockable$PriorWriteLock.atomic(Lockable.java:74) + at water.TAtomic.atomic(TAtomic.java:19) + at water.Atomic.compute2(Atomic.java:58) + at water.Atomic.fork(Atomic.java:42) + at water.Atomic.invoke(Atomic.java:34) + at water.Lockable.write_lock(Lockable.java:60) + at water.exec.Env.remove_and_unlock(Env.java:349) + at water.api.Exec2.serve(Exec2.java:71) + at water.api.Request.serveGrid(Request.java:165) + at water.Request2.superServeGrid(Request2.java:482) + at water.api.Exec2.serveGrid(Exec2.java:78) + at water.api.Request.serve(Request.java:142) + at water.api.RequestServer.serve(RequestServer.java:484) + at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:424) + at java.lang.Thread.run(Thread.java:745) Output from test: [2014-08-14 17:37:37] [ERROR] : Error: Test failed: 'Testing memory performance of Strong Rules' Not expected: http://127.0.0.1:54321/2/Exec2.json returned the following error: Frame is already locked by job null. 1: withWarnings(test(conn)) 2: withCallingHandlers(expr, warning = wHandler) 3: test(conn) 4: h2o.glm(x = c(1:7000), y = "arcene.train.label", data = arcene.train.full, family = "binomial", lambda_search = T, alpha = 1, nfolds = 0, use_all_factor_levels = T) 5: .h2o.get.glm(data@h2o, as.character(res$destination_key), return_all_lambda) 6: h2o.getFrame(h2o, pre$json$glm_model$dataKey) 7: .h2o.exec2(expr = key, h2o = h2o, dest_key = key) 8: .h2o.__exec2_dest_key(h2o, expr, dest_key) 9: .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) 10: stop(paste(myURL, " returned the following error:\n", .h2o.__formatError(res$error))) 11: .handleSimpleError(function (e) { e$calls <- head(sys.calls()[-seq_len(frame + 7)], -2) signalCondition(e) }, "http://127.0.0.1:54321/2/Exec2.json returned the following error:\n Frame is already locked by job null.\n", quote(.h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr))).

    JIRA | 2 years ago | Ariel Rao
    java.lang.IllegalArgumentException: Dataset arcene.train.full is already in use. Unable to use it now. Consider using a different destination name.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    it's not necessarily a good thing to do, but we should support it [load a dataset; call it d] d$col2000 <- min(d$col2000) Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://127.0.0.1:54321/2/Exec2.json returned the following error: 2 better yet… once you've done this, stuff breaks everywhere > d$col2000 <- min(d$col2000) + d$col2000 Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://127.0.0.1:54321/2/Exec2.json returned the following error: Dataset Last.value.4 is already in use. Unable to use it now. Consider using a different destination name. > head(d) Error in data.frame(col1 = 0.542521227151155, col2 = 0.267292166827247, : arguments imply differing number of rows: 1, 0 > 04:23:10.668 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.8 = bad_2k.hex[,2000] = c(0.000574313336983323) 04:23:10.747 # Session ERRR WATER:▫ + java.lang.ArrayIndexOutOfBoundsException: 2 + ▸ at water.fvec.Vec.elem2ChunkIdx(Vec.java:407) + ▸ at water.fvec.Vec.chunk(Vec.java:505) + ▸ at water.fvec.Vec.at(Vec.java:513) + ▸ at water.fvec.Frame.toString(Frame.java:475) + ▸ at water.api.Exec2.serve(Exec2.java:50) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) 04:23:10.748 # Session ERRR WATER:▫ + java.lang.ArrayIndexOutOfBoundsException: 2 + ▸ at water.fvec.Vec.elem2ChunkIdx(Vec.java:407) + ▸ at water.fvec.Vec.chunk(Vec.java:505) + ▸ at water.fvec.Vec.at(Vec.java:513) + ▸ at water.fvec.Frame.toString(Frame.java:475) + ▸ at water.api.Exec2.serve(Exec2.java:50) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) 04:23:51.514 # Session INFO HTTPD: POST /Inspect.json key=bad_2k.hex 04:23:51.633 # Session INFO HTTPD: POST /Inspect.json key=bad_2k.hex max_column_display=2147483647 04:23:51.753 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.9 = bad_2k.hex[,2000] 04:23:51.768 # Session ERRR WATER:▫ + java.lang.IllegalArgumentException: Dataset Last.value.4 is already in use. Unable to use it now. Consider using a different destination name. + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + ▸ at water.TAtomic.atomic(TAtomic.java:19) + ▸ at water.Atomic.compute2(Atomic.java:57) + ▸ at water.Atomic.fork(Atomic.java:42) + ▸ at water.Atomic.invoke(Atomic.java:34) + ▸ at water.Lockable.write_lock(Lockable.java:59) + ▸ at water.exec.Env.remove_and_unlock(Env.java:326) + ▸ at water.api.Exec2.serve(Exec2.java:62) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724)

    JIRA | 3 years ago | Earl Hathaway
    java.lang.IllegalArgumentException: Dataset Last.value.4 is already in use. Unable to use it now. Consider using a different destination name.
  6. 0

    say you load a dataset. it didn't load correctly because of a bad column name. I edited the file then reran the same R code to load the data. d <- h2o.importFile(l, '/Users/earl/work/bad.2k.csv') Now a number of operations emit this (exception below) edit: it gets worse. even if I reload the data and specify a different key, eg d <- h2o.importFile(l, '/Users/earl/work/bad.2k.csv', 'fml') the key is set properly, but operations such as > min(d$col2000) Error in .h2o.__remoteSend(client, .h2o.__PAGE_EXEC2, str = expr) : http://127.0.0.1:54321/2/Exec2.json returned the following error: Dataset Last.value.38 is already in use. Unable to use it now. Consider using a different destination name. create the below exception as well. To get exec2 to work again you have to bounce the whole cluster. + java.lang.IllegalArgumentException: Dataset bad_2k.hex is already in use. Unable to use it now. Consider using a different destination name. + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + ▸ at water.TAtomic.atomic(TAtomic.java:19) + ▸ at water.Atomic.compute2(Atomic.java:57) + ▸ at water.Atomic.fork(Atomic.java:42) + ▸ at water.Atomic.invoke(Atomic.java:34) + ▸ at water.Lockable.write_lock(Lockable.java:59) + ▸ at water.exec.Env.remove_and_unlock(Env.java:326) + ▸ at water.api.Exec2.serve(Exec2.java:62) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) 04:29:39.990 # Session INFO HTTPD: POST /Inspect.json key=bad_2k2.hex 04:29:40.230 # Session INFO HTTPD: POST /Inspect.json key=bad_2k2.hex max_column_display=2147483647 04:29:40.350 # Session INFO HTTPD: POST /2/Exec2.json str=Last.value.38 = bad_2k2.hex[,2000] 04:29:40.366 # Session ERRR WATER:▫ + java.lang.IllegalArgumentException: Dataset bad_2k.hex is already in use. Unable to use it now. Consider using a different destination name. + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + ▸ at water.TAtomic.atomic(TAtomic.java:19) + ▸ at water.Atomic.compute2(Atomic.java:57) + ▸ at water.Atomic.fork(Atomic.java:42) + ▸ at water.Atomic.invoke(Atomic.java:34) + ▸ at water.Lockable.write_lock(Lockable.java:59) + ▸ at water.exec.Env.remove_and_unlock(Env.java:326) + ▸ at water.api.Exec2.serve(Exec2.java:62) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724) ================ new exception 04:33:21.503 # Session ERRR WATER:▫ + java.lang.IllegalArgumentException: Dataset Last.value.38 is already in use. Unable to use it now. Consider using a different destination name. + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:84) + ▸ at water.Lockable$PriorWriteLock.atomic(Lockable.java:73) + ▸ at water.TAtomic.atomic(TAtomic.java:19) + ▸ at water.Atomic.compute2(Atomic.java:57) + ▸ at water.Atomic.fork(Atomic.java:42) + ▸ at water.Atomic.invoke(Atomic.java:34) + ▸ at water.Lockable.write_lock(Lockable.java:59) + ▸ at water.exec.Env.remove_and_unlock(Env.java:326) + ▸ at water.api.Exec2.serve(Exec2.java:62) + ▸ at water.api.Request.serveGrid(Request.java:129) + ▸ at water.Request2.superServeGrid(Request2.java:472) + ▸ at water.api.Exec2.serveGrid(Exec2.java:71) + ▸ at water.api.Request.serve(Request.java:108) + ▸ at water.api.RequestServer.serve(RequestServer.java:315) + ▸ at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:421) + ▸ at java.lang.Thread.run(Thread.java:724)

    JIRA | 3 years ago | Earl Hathaway
    java.lang.IllegalArgumentException: Dataset Last.value.38 is already in use. Unable to use it now. Consider using a different destination name.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Dataset segmentab.hex is already in use. Unable to use it now. Consider using a different destination name.

      at water.Lockable$PriorWriteLock.atomic()
    2. water
      Lockable.write_lock
      1. water.Lockable$PriorWriteLock.atomic(Lockable.java:85)
      2. water.Lockable$PriorWriteLock.atomic(Lockable.java:74)
      3. water.TAtomic.atomic(TAtomic.java:19)
      4. water.Atomic.compute2(Atomic.java:58)
      5. water.Atomic.fork(Atomic.java:42)
      6. water.Atomic.invoke(Atomic.java:34)
      7. water.Lockable.write_lock(Lockable.java:60)
      7 frames
    3. water.exec
      Env.remove_and_unlock
      1. water.exec.Env.remove_and_unlock(Env.java:349)
      1 frame
    4. water.api
      Request.serveGrid
      1. water.api.Exec2.serve(Exec2.java:71)
      2. water.api.Request.serveGrid(Request.java:165)
      2 frames
    5. water
      Request2.superServeGrid
      1. water.Request2.superServeGrid(Request2.java:481)
      1 frame