java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152

JIRA | Nidhi Mehta | 3 years ago
  1. 0

    Build git branch: rel-jacobi Build git hash: 82c33a399506a133d6f0485c1a11cdac1466fd91 Build project version: 2.2.0.6 Built by: 'jenkins' On 1 machine with heap size =12g Model finishes fine in ~ 30 min On cust_churn test/train dataset 11:04:04.812 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 validation=churn_test.hex max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 On 2 machines (162/163) with heap size=20G 12:18:25.453 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 After ~45 min, Get- java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:44:47.408 FJ-9-156 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:44:47.409 FJ-9-156 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:44:47.409 FJ-9-156 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Exception in thread "Multi-UDP-R" java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive0(Native Method) 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) 12:44:56.636 #ti-UDP-R INFO WATER: at java.net.DatagramSocket.receive(DatagramSocket.java:725) 12:44:56.636 #ti-UDP-R INFO WATER: at water.MultiReceiverThread.run(MultiReceiverThread.java:64) java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 12:45:04.386 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) 12:45:13.234 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) 12:45:19.383 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:157) 12:45:23.960 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:153) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:45:23.960 FJ-9-167 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:45:23.961 FJ-9-167 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:45:23.961 FJ-9-167 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 12:45:23.962 #:54321-0 INFO WATER: at water.MemoryManager.set_goals(MemoryManager.java:99) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc(MemoryManager.java:254) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc1(MemoryManager.java:259) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:818) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:815) 12:45:23.965 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:689) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getA(AutoBuffer.java:699) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getAA(AutoBuffer.java:711) 12:45:23.966 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel.read(DTree.java) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.DRF$DRFModel.read(DRF.java) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:683) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.TreeVotes.read(TreeVotes.java) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.response(RPC.java:470) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.tcp_ack(RPC.java:438) 12:45:23.966 #:54321-0 INFO WATER: at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)

    JIRA | 3 years ago | Nidhi Mehta
    java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152
  2. 0

    -on rel turnrbull1 -launched h2o with 50g heap size -data, script etc on mr0xg8 /nidhi/automl_dl/ -ran following script dl grid with xval. -hit following java lang assertion error. -went to flow, exported all models that were built fine -deleted data and all models. the cluster status still showed 43gb occupied. -to conform, tried parsing airlines_all.csv (12g). got error as to cannot parse as the cluster size is zero (screenshots attached) script - {code:java} aa = read.csv("/home/nidhi/auto_ml_dl/auto_ml_fileList.txt",header=F) i=79 print(paste("/home/nidhi/auto_sklearn_csv/", aa[i,1],sep='')) data = h2o.importFile(paste("/home/nidhi/auto_sklearn_csv/", aa[i,1],sep=''),destination_frame = "data",header = T) dim(data) response = as.character(aa[i,2]) response_indx = which(names(data)==response) data[,response_indx] = as.factor(data[,response_indx]) data = h2o.assign(data,key = "data") m = length(h2o.levels(data[,response_indx])) if (m > 2){ family ="multinomial" }else{ family = "binomial" } id_col_idx = which(names(data) %in% c("ID","Id")) if(length(id_col_idx)>0){ #remove id column data = h2o.assign(data[,-id_col_idx],key = "data") } myY = response myX = setdiff(names(data),response) griables1 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50), c(200), c(500)), hidden_dropout_ratios=list(rep(0,1), rep(0.1,1),rep(.2,1),rep(.3,1),rep(.4,1), rep(0.5,1)), stopping_rounds = 3 ) griables2 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50,50), c(200,200), c(500,500)), hidden_dropout_ratios=list(rep(0,2), rep(0.1,2),rep(.2,2),rep(.3,2),rep(.4,2), rep(0.5,2)), stopping_rounds = 3 ) griables3 = list(activation = "RectifierWithDropout", ## all you need epochs=10000,## early stopping takes care of epochs - no need to tune! adaptive_rate =TRUE, rho = c(.9,.95,.99), epsilon = c(1e-6,1e-7,1e-8,1e-9), input_dropout_ratio = c(0,.05,.1,.15,.2), hidden = list(c(50,50,50), c(200,200,200), c(500,500,500)), hidden_dropout_ratios=list(rep(0,3), rep(0.1,3),rep(.2,3),rep(.3,3),rep(.4,3), rep(0.5,3)), stopping_rounds = 3 ) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables1,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables2,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) gg = h2o.grid(algorithm = "deeplearning",grid_id = as.character(aa[i,1]), x=myX, y=myY, training_frame=data, hyper_params= griables3,nfolds=3, search_criteria =list(strategy = "RandomDiscrete",stopping_metric = "AUTO", stopping_tolerance = 0.0001, stopping_rounds = 3, max_models = 10)) {code} {code:java} [1] "dataset-79" [1] "/home/nidhi/auto_sklearn_csv/KDDCup09_appetency.arff.txt" |======================================================================| 100% |======================================================================| 100% |======================================================================| 100% |======================================================= | 79% java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 at water.MemoryManager.set_goals(MemoryManager.java:94) at water.MemoryManager.malloc(MemoryManager.java:246) at water.MemoryManager.malloc(MemoryManager.java:206) at water.MemoryManager.arrayCopyOfRange(MemoryManager.java:272) at water.AutoBuffer.expandByteBuffer(AutoBuffer.java:704) at water.AutoBuffer.putA4f(AutoBuffer.java:1340) at hex.deeplearning.Storage$DenseRowMatrix$Icer.write121(Storage$DenseRowMatrix$Icer.java) at hex.deeplearning.Storage$DenseRowMatrix$Icer.write(Storage$DenseRowMatrix$Icer.java) at water.Iced.write(Iced.java:61) at water.AutoBuffer.put(AutoBuffer.java:756) at water.AutoBuffer.putA(AutoBuffer.java:868) at hex.deeplearning.DeepLearningModelInfo$Icer.write120(DeepLearningModelInfo$Icer.java) at hex.deeplearning.DeepLearningModelInfo$Icer.write(DeepLearningModelInfo$Icer.java) at water.Iced.write(Iced.java:61) at water.AutoBuffer.put(AutoBuffer.java:756) at hex.deeplearning.DeepLearningModel$Icer.write110(DeepLearningModel$Icer.java) at hex.deeplearning.DeepLearningModel$Icer.write(DeepLearningModel$Icer.java) at water.Iced.write(Iced.java:61) at water.Iced.asBytes(Iced.java:42) at water.Value.<init>(Value.java:317) at water.Value.<init>(Value.java:312) at water.DKV.put(DKV.java:55) at hex.deeplearning.DeepLearningModel.putMeAsBestModel(DeepLearningModel.java:916) at hex.deeplearning.DeepLearningModel.doScoring(DeepLearningModel.java:499) at hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:436) at hex.deeplearning.DeepLearning$DeepLearningDriver.buildModel(DeepLearning.java:306) at hex.deeplearning.DeepLearning$DeepLearningDriver.computeImpl(DeepLearning.java:211) at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:169) at hex.deeplearning.DeepLearning$DeepLearningDriver.compute2(DeepLearning.java:204) at water.H2O$H2OCountedCompleter.compute(H2O.java:1217) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Error: java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656 {code}

    JIRA | 1 month ago | Nidhi Mehta
    java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=47721742336 heapUsedGC=48054034656
  3. 0
    Check if you use the right path
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    This is the assertion failure: java.lang.AssertionError: Expected :[i love movie, i am happy today that I got this working.] Actual :[i am happy today that I got this working.] <Click to see difference> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at co.cask.cdap.apps.sentiment.TwitterSentimentAppTest.testSentimentService(TwitterSentimentAppTest.java:82) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

    Cask Community Issue Tracker | 1 year ago | Ali Anwar
    java.lang.AssertionError: Expected :[i love movie, i am happy today that I got this working.] Actual :[i am happy today that I got this working.] <Click to see difference>
  6. 0

    strange AssertError when using fest regexp assert

    Stack Overflow | 3 years ago | ligi
    java.lang.AssertionError: &lt;'0 I: bar 1 I: foo '&gt; should match the regular expression:&lt;'.*bar.*'&gt;

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.AssertionError

      I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152

      at water.MemoryManager.set_goals()
    2. water
      AutoBuffer.getA1
      1. water.MemoryManager.set_goals(MemoryManager.java:99)
      2. water.MemoryManager.malloc(MemoryManager.java:254)
      3. water.MemoryManager.malloc1(MemoryManager.java:259)
      4. water.AutoBuffer.getA1(AutoBuffer.java:818)
      5. water.AutoBuffer.getA1(AutoBuffer.java:815)
      5 frames
    3. hex.gbm
      DTree$TreeModel$CompressedTree.read
      1. hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java)
      1 frame
    4. water
      AutoBuffer.getAA
      1. water.AutoBuffer.get(AutoBuffer.java:689)
      2. water.AutoBuffer.getA(AutoBuffer.java:699)
      3. water.AutoBuffer.getAA(AutoBuffer.java:711)
      3 frames
    5. hex.gbm
      DTree$TreeModel.read
      1. hex.gbm.DTree$TreeModel.read(DTree.java)
      1 frame
    6. hex.drf
      DRF$DRFModel.read
      1. hex.drf.DRF$DRFModel.read(DRF.java)
      1 frame
    7. water
      AutoBuffer.get
      1. water.AutoBuffer.get(AutoBuffer.java:683)
      1 frame
    8. hex.drf
      TreeVotes.read
      1. hex.drf.TreeVotes.read(TreeVotes.java)
      1 frame
    9. water
      TCPReceiverThread$TCPReaderThread.run
      1. water.RPC.response(RPC.java:470)
      2. water.RPC.tcp_ack(RPC.java:438)
      3. water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)
      3 frames