java.lang.AssertionError

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Build git branch: rel-jacobi Build git hash: 82c33a399506a133d6f0485c1a11cdac1466fd91 Build project version: 2.2.0.6 Built by: 'jenkins' On 1 machine with heap size =12g Model finishes fine in ~ 30 min On cust_churn test/train dataset 11:04:04.812 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 validation=churn_test.hex max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 On 2 machines (162/163) with heap size=20G 12:18:25.453 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 After ~45 min, Get- java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:44:47.408 FJ-9-156 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:44:47.409 FJ-9-156 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:44:47.409 FJ-9-156 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Exception in thread "Multi-UDP-R" java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive0(Native Method) 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) 12:44:56.636 #ti-UDP-R INFO WATER: at java.net.DatagramSocket.receive(DatagramSocket.java:725) 12:44:56.636 #ti-UDP-R INFO WATER: at water.MultiReceiverThread.run(MultiReceiverThread.java:64) java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 12:45:04.386 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) 12:45:13.234 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) 12:45:19.383 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:157) 12:45:23.960 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:153) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:45:23.960 FJ-9-167 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:45:23.961 FJ-9-167 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:45:23.961 FJ-9-167 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 12:45:23.962 #:54321-0 INFO WATER: at water.MemoryManager.set_goals(MemoryManager.java:99) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc(MemoryManager.java:254) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc1(MemoryManager.java:259) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:818) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:815) 12:45:23.965 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:689) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getA(AutoBuffer.java:699) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getAA(AutoBuffer.java:711) 12:45:23.966 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel.read(DTree.java) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.DRF$DRFModel.read(DRF.java) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:683) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.TreeVotes.read(TreeVotes.java) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.response(RPC.java:470) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.tcp_ack(RPC.java:438) 12:45:23.966 #:54321-0 INFO WATER: at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)
    via by Nidhi Mehta,
  • -on rel turnrbull1 -launched h2o with 50g heap size -data, script etc on mr0xg8 /nidhi/automl_dl/ -ran following script dl grid with xval. -hit following java lang assertion error. -went to flow, exported all models that were built fine -deleted data and all models. the cluster status still showed 43gb occupied. -to conform, tried parsing airlines_all.csv (12g). got error as to cannot parse as the cluster size is zero (screenshots attached) script - {code:java} aa = read.csv("/home/nidhi/auto_ml_dl/auto_ml_fileList.txt",header=F) i=79 print(paste("/home/nidhi/auto_sklearn_csv/", aa[i,1],sep='')) data = h2o.importFile(paste("/home/nidhi/auto_sklearn_csv/", aa[i,1],sep=''),destination_frame = "data",header = T) dim(data) response = as.character(aa[i,2]) response_indx = which(names(data)==response) data[,response_indx] = as.factor(data[,response_indx]) data = h2o.assign(data,key = "data") m = length(h2o.levels(data[,response_indx])) if (m > 2){ family ="multinomial" }else{ family = "binomial" } id_col_idx = which(names(data) %in% c("ID","Id")) if(length(id_col_idx)>0){ #remove id column data = h2o.assign(data[,-id_col_idx],key = "data") } myY = response myX = setdiff(names(data),response) griables1 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50), c(200), c(500)), hidden_dropout_ratios=list(rep(0,1), rep(0.1,1),rep(.2,1),rep(.3,1),rep(.4,1), rep(0.5,1)), stopping_rounds = 3 ) griables2 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50,50), c(200,200), c(500,500)), hidden_dropout_ratios=list(rep(0,2), rep(0.1,2),rep(.2,2),rep(.3,2),rep(.4,2), rep(0.5,2)), stopping_rounds = 3 ) griables3 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50,50,50), c(200,200,200), c(500,500,500)), hidden_dropout_ratios=list(rep(0,3), rep(0.1,3),rep(.2,3),rep(.3,3),rep(.4,3), rep(0.5,3)), stopping_rounds = 3 ) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables1,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables2,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables3,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) {code} {code:java} [1] "dataset-79" [1] "/home/nidhi/auto_sklearn_csv/KDDCup09_appetency.arff.txt" |======================================================================| 100% |======================================================================| 100% |======================================================================| 100% |======================================================= | 79% java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 at water.MemoryManager.set_goals(MemoryManager.java:94) at water.MemoryManager.malloc(MemoryManager.java:246) at water.MemoryManager.malloc(MemoryManager.java:206) at water.MemoryManager.arrayCopyOfRange(MemoryManager.java:272) at water.AutoBuffer.expandByteBuffer(AutoBuffer.java:704) at water.AutoBuffer.putA4f(AutoBuffer.java:1340) at hex.deeplearning.Storage$DenseRowMatrix$Icer.write121(Storage$DenseRowMatrix$Icer.java) at hex.deeplearning.Storage$DenseRowMatrix$Icer.write(Storage$DenseRowMatrix$Icer.java) at water.Iced.write(Iced.java:61) at water.AutoBuffer.put(AutoBuffer.java:756) at water.AutoBuffer.putA(AutoBuffer.java:868) at hex.deeplearning.DeepLearningModelInfo$Icer.write120(DeepLearningModelInfo$Icer.java) at hex.deeplearning.DeepLearningModelInfo$Icer.write(DeepLearningModelInfo$Icer.java) at water.Iced.write(Iced.java:61) at water.AutoBuffer.put(AutoBuffer.java:756) at hex.deeplearning.DeepLearningModel$Icer.write110(DeepLearningModel$Icer.java) at hex.deeplearning.DeepLearningModel$Icer.write(DeepLearningModel$Icer.java) at water.Iced.write(Iced.java:61) at water.Iced.asBytes(Iced.java:42) at water.Value.<init>(Value.java:317) at water.Value.<init>(Value.java:312) at water.DKV.put(DKV.java:55) at hex.deeplearning.DeepLearningModel.putMeAsBestModel(DeepLearningModel.java:916) at hex.deeplearning.DeepLearningModel.doScoring(DeepLearningModel.java:499) at hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:436) at hex.deeplearning.DeepLearning$DeepLearningDriver.buildModel(DeepLearning.java:306) at hex.deeplearning.DeepLearning$DeepLearningDriver.computeImpl(DeepLearning.java:211) at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:169) at hex.deeplearning.DeepLearning$DeepLearningDriver.compute2(DeepLearning.java:204) at water.H2O$H2OCountedCompleter.compute(H2O.java:1217) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Error: java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 {code}
    via by Nidhi Mehta,
    • java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 at water.MemoryManager.set_goals(MemoryManager.java:99) at water.MemoryManager.malloc(MemoryManager.java:254) at water.MemoryManager.malloc1(MemoryManager.java:259) at water.AutoBuffer.getA1(AutoBuffer.java:818) at water.AutoBuffer.getA1(AutoBuffer.java:815) at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) at water.AutoBuffer.get(AutoBuffer.java:689) at water.AutoBuffer.getA(AutoBuffer.java:699) at water.AutoBuffer.getAA(AutoBuffer.java:711) at hex.gbm.DTree$TreeModel.read(DTree.java) at hex.drf.DRF$DRFModel.read(DRF.java) at water.AutoBuffer.get(AutoBuffer.java:683) at hex.drf.TreeVotes.read(TreeVotes.java) at water.RPC.response(RPC.java:470) at water.RPC.tcp_ack(RPC.java:438) at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)
    No Bugmate found.