java.lang.AssertionError

JIRA | Hank Roark | 12 months ago
  1. 0

    I was doing a hyper parameter search over GBM (I would expected 108 models to build). 59 of the models build successfully. Sometime building model 60 this failure occurred. It even looks like 6 of 10 of the cross validation models were built. Logs attached. It would be nice to just fix the bug. With that said, it would also be nice to be able to restart a grid search that fails for some reason, but don't repeat building models for parameters that completed successfully (in this case, the system would keep models 1-59 and just add models 60-108 onto the same grid object). Also note that the gbm.grid object is not available in R. Having a handle on the object, even if partially completed, would be good. {noformat} H2O Build git branch master H2O Build git hash 1a955973fbb9beeca0ea08783e79e0584149f68d H2O Build git describe jenkins-master-3300 H2O Build project version 3.7.0.3300 H2O Built by jenkins H2O Built on 2015-12-17 23:32:31 Flow version 0.4.8 {noformat} {noformat} > gbm.hyper.params <- list(ntrees=c(100,300,1000), learn_rate=c(0.1,0.03,0.01), col_sample_rate_per_tree=c(1,0.7), max_depth=c(2,5,10), sample_rate=c(1,0.7) ) > gbm.grid <- h2o.grid(algorithm = "gbm", grid="gbm.grid", x=x.columns, y=y.column.h2o, training_frame=h2o.train, nfolds=10, distribution="multinomial", hyper_params = gbm.hyper.params) |=============================================================================================== | 56% Got exception 'class java.lang.AssertionError', with msg 'null' java.lang.AssertionError at hex.tree.DTree$UndecidedNode.scoreCols(DTree.java:301) at hex.tree.DTree$UndecidedNode.<init>(DTree.java:274) at hex.tree.DTree$DecidedNode.makeUndecidedNode(DTree.java:422) at hex.tree.DTree$DecidedNode.<init>(DTree.java:493) at hex.tree.SharedTree.makeDecided(SharedTree.java:491) at hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:438) at jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425) at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383) at water.MRTask.compute2(MRTask.java:689) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1060) at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1056) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Error: 'null' > gbm.grid Error: object 'gbm.grid' not found {noformat} This appears at the tail of the log {noformat} 12-21 02:08:47.594 127.0.0.1:54321 90094 FJ-1-9 INFO: 96. tree was built in 00:00:00.149 (Wall: 21-Dec 02:08:47.594) 12-21 02:08:47.750 127.0.0.1:54321 90094 FJ-1-9 INFO: 97. tree was built in 00:00:00.156 (Wall: 21-Dec 02:08:47.750) 12-21 02:08:47.894 127.0.0.1:54321 90094 FJ-3-13 ERRR: Error: Missing chunk 6 for $04ffb4000000ffffffff14b16c1fc73c32bab142df427378eb$P 12-21 02:08:47.896 127.0.0.1:54321 90094 FJ-3-15 ERRR: Error: Missing chunk 0 for $04ff85000000ffffffff14b16c1fc73c32bab142df427378eb$P {noformat}

    JIRA | 12 months ago | Hank Roark
    java.lang.AssertionError
  2. 0

    {code} parseFiles paths: ["/users/arno/./trainCSV.csv"] destination_frame: "trainCSV.hex" parse_type: "CSV" separator: 44 number_columns: 22 single_quotes: false column_names: ["Store","DayOfWeek","Sales","Customers","Open","Promo","SchoolHoliday","StoreType","Assortment","CompetitionDistance","CompetitionOpenSinceMonth","CompetitionOpenSinceYear","Promo2","Promo2SinceWeek","Promo2SinceYear","PromoInterval","month","year","day","logSales","foldId","ymd"] column_types: ["Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric"] delete_on_done: true check_header: 1 chunk_size: 32768 {code} {code} buildModel 'gbm', {"model_id":"gbm-9403e728-d56e-4877-af7c-5d76251b50f7","training_frame":"trainCSV.hex","nfolds":"5","response_column":"logSales","ignored_columns":["Sales","Customers"],"ignore_const_cols":true,"ntrees":"3000","max_depth":"10","min_rows":10,"nbins":20,"nbins_cats":1024,"seed":-5945912630645972000,"learn_rate":"0.02","distribution":"AUTO","sample_rate":".9","col_sample_rate":"0.7","col_sample_rate_per_tree":0.7,"score_each_iteration":false,"fold_assignment":"AUTO","balance_classes":false,"max_confusion_matrix_size":20,"max_hit_ratio_k":10,"r2_stopping":0.999999,"stopping_rounds":0,"stopping_metric":"AUTO","stopping_tolerance":0.001,"build_tree_one_node":false,"tweedie_power":1.5,"checkpoint":"","keep_cross_validation_predictions":false,"class_sampling_factors":[],"max_after_balance_size":5,"nbins_top_level":1024} {code} {code} onExCompletion for hex.tree.SharedTree$ScoreBuildOneTree@6e0a357f java.lang.AssertionError at hex.tree.DTree$UndecidedNode.scoreCols(DTree.java:301) at hex.tree.DTree$UndecidedNode.<init>(DTree.java:274) at hex.tree.DTree$DecidedNode.makeUndecidedNode(DTree.java:422) at hex.tree.DTree$DecidedNode.<init>(DTree.java:493) at hex.tree.SharedTree.makeDecided(SharedTree.java:491) at hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:438) at jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425) at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383) at water.MRTask.compute2(MRTask.java:689) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1060) at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1056) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) {code}

    JIRA | 12 months ago | Arno Candel
    java.lang.AssertionError
  3. 0

    {code} parseFiles paths: ["/users/arno/./trainCSV.csv"] destination_frame: "trainCSV.hex" parse_type: "CSV" separator: 44 number_columns: 22 single_quotes: false column_names: ["Store","DayOfWeek","Sales","Customers","Open","Promo","SchoolHoliday","StoreType","Assortment","CompetitionDistance","CompetitionOpenSinceMonth","CompetitionOpenSinceYear","Promo2","Promo2SinceWeek","Promo2SinceYear","PromoInterval","month","year","day","logSales","foldId","ymd"] column_types: ["Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric","Numeric"] delete_on_done: true check_header: 1 chunk_size: 32768 {code} {code} buildModel 'gbm', {"model_id":"gbm-9403e728-d56e-4877-af7c-5d76251b50f7","training_frame":"trainCSV.hex","nfolds":"5","response_column":"logSales","ignored_columns":["Sales","Customers"],"ignore_const_cols":true,"ntrees":"3000","max_depth":"10","min_rows":10,"nbins":20,"nbins_cats":1024,"seed":-5945912630645972000,"learn_rate":"0.02","distribution":"AUTO","sample_rate":".9","col_sample_rate":"0.7","col_sample_rate_per_tree":0.7,"score_each_iteration":false,"fold_assignment":"AUTO","balance_classes":false,"max_confusion_matrix_size":20,"max_hit_ratio_k":10,"r2_stopping":0.999999,"stopping_rounds":0,"stopping_metric":"AUTO","stopping_tolerance":0.001,"build_tree_one_node":false,"tweedie_power":1.5,"checkpoint":"","keep_cross_validation_predictions":false,"class_sampling_factors":[],"max_after_balance_size":5,"nbins_top_level":1024} {code} {code} onExCompletion for hex.tree.SharedTree$ScoreBuildOneTree@6e0a357f java.lang.AssertionError at hex.tree.DTree$UndecidedNode.scoreCols(DTree.java:301) at hex.tree.DTree$UndecidedNode.<init>(DTree.java:274) at hex.tree.DTree$DecidedNode.makeUndecidedNode(DTree.java:422) at hex.tree.DTree$DecidedNode.<init>(DTree.java:493) at hex.tree.SharedTree.makeDecided(SharedTree.java:491) at hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:438) at jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425) at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383) at water.MRTask.compute2(MRTask.java:689) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1060) at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1056) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) {code}

    JIRA | 12 months ago | Arno Candel
    java.lang.AssertionError
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I was doing a hyper parameter search over GBM (I would expected 108 models to build). 59 of the models build successfully. Sometime building model 60 this failure occurred. It even looks like 6 of 10 of the cross validation models were built. Logs attached. It would be nice to just fix the bug. With that said, it would also be nice to be able to restart a grid search that fails for some reason, but don't repeat building models for parameters that completed successfully (in this case, the system would keep models 1-59 and just add models 60-108 onto the same grid object). Also note that the gbm.grid object is not available in R. Having a handle on the object, even if partially completed, would be good. {noformat} H2O Build git branch master H2O Build git hash 1a955973fbb9beeca0ea08783e79e0584149f68d H2O Build git describe jenkins-master-3300 H2O Build project version 3.7.0.3300 H2O Built by jenkins H2O Built on 2015-12-17 23:32:31 Flow version 0.4.8 {noformat} {noformat} > gbm.hyper.params <- list(ntrees=c(100,300,1000), learn_rate=c(0.1,0.03,0.01), col_sample_rate_per_tree=c(1,0.7), max_depth=c(2,5,10), sample_rate=c(1,0.7) ) > gbm.grid <- h2o.grid(algorithm = "gbm", grid="gbm.grid", x=x.columns, y=y.column.h2o, training_frame=h2o.train, nfolds=10, distribution="multinomial", hyper_params = gbm.hyper.params) |=============================================================================================== | 56% Got exception 'class java.lang.AssertionError', with msg 'null' java.lang.AssertionError at hex.tree.DTree$UndecidedNode.scoreCols(DTree.java:301) at hex.tree.DTree$UndecidedNode.<init>(DTree.java:274) at hex.tree.DTree$DecidedNode.makeUndecidedNode(DTree.java:422) at hex.tree.DTree$DecidedNode.<init>(DTree.java:493) at hex.tree.SharedTree.makeDecided(SharedTree.java:491) at hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:438) at jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425) at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383) at water.MRTask.compute2(MRTask.java:689) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1060) at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1056) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Error: 'null' > gbm.grid Error: object 'gbm.grid' not found {noformat} This appears at the tail of the log {noformat} 12-21 02:08:47.594 127.0.0.1:54321 90094 FJ-1-9 INFO: 96. tree was built in 00:00:00.149 (Wall: 21-Dec 02:08:47.594) 12-21 02:08:47.750 127.0.0.1:54321 90094 FJ-1-9 INFO: 97. tree was built in 00:00:00.156 (Wall: 21-Dec 02:08:47.750) 12-21 02:08:47.894 127.0.0.1:54321 90094 FJ-3-13 ERRR: Error: Missing chunk 6 for $04ffb4000000ffffffff14b16c1fc73c32bab142df427378eb$P 12-21 02:08:47.896 127.0.0.1:54321 90094 FJ-3-15 ERRR: Error: Missing chunk 0 for $04ff85000000ffffffff14b16c1fc73c32bab142df427378eb$P {noformat}

    JIRA | 12 months ago | Hank Roark
    java.lang.AssertionError
  6. 0

    {code} library(h2o) h2o.init() df <- h2o.createFrame(rows = 100000,cols=100,categorical_fraction = 0,seed=1234) df summary(df) predictors <- 1:ncol(df) df$C100 <- df$C100 * 1e100 df$response <- df[,4]*df[,3]-df[,57]+4*df[,67] # this works fine #dl <- h2o.deeplearning(x=predictors,y="response",training_frame = df) #dl # this fails gbm <- h2o.gbm(x=predictors,y="response",training_frame = df) gbm {code} {code} 03-09 13:52:38.177 172.16.2.20:54321 42759 FJ-1-15 INFO: ============================================================== 03-09 13:52:38.456 172.16.2.20:54321 42759 FJ-1-15 INFO: 9. tree was built in 00:00:00.276 (Wall: 09-Mar 13:52:38.456) 03-09 13:52:38.457 172.16.2.20:54321 42759 FJ-1-15 INFO: ============================================================== 03-09 13:52:38.708 172.16.2.20:54321 42759 FJ-1-15 INFO: 10. tree was built in 00:00:00.249 (Wall: 09-Mar 13:52:38.708) 03-09 13:52:38.709 172.16.2.20:54321 42759 FJ-1-15 INFO: ============================================================== 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: java.lang.AssertionError 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.DHistogram.<init>(DHistogram.java:107) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.DHistogram.make(DHistogram.java:197) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.DTree$Split.split(DTree.java:245) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.DTree$DecidedNode.<init>(DTree.java:492) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.SharedTree.makeDecided(SharedTree.java:429) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:378) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.MRTask.compute2(MRTask.java:648) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.MRTask.compute2(MRTask.java:577) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.MRTask.compute2(MRTask.java:577) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.MRTask.compute2(MRTask.java:577) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1184) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1180) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 03-09 13:52:43.100 172.16.2.20:54321 42759 FJ-1-15 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) {code}

    JIRA | 9 months ago | Arno Candel
    java.lang.AssertionError

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.AssertionError

      No message provided

      at hex.tree.DTree$UndecidedNode.scoreCols()
    2. hex.tree
      SharedTree$ScoreBuildOneTree.onCompletion
      1. hex.tree.DTree$UndecidedNode.scoreCols(DTree.java:301)
      2. hex.tree.DTree$UndecidedNode.<init>(DTree.java:274)
      3. hex.tree.DTree$DecidedNode.makeUndecidedNode(DTree.java:422)
      4. hex.tree.DTree$DecidedNode.<init>(DTree.java:493)
      5. hex.tree.SharedTree.makeDecided(SharedTree.java:491)
      6. hex.tree.SharedTree$ScoreBuildOneTree.onCompletion(SharedTree.java:438)
      6 frames
    3. jsr166y
      CountedCompleter.tryComplete
      1. jsr166y.CountedCompleter.__tryComplete(CountedCompleter.java:425)
      2. jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:383)
      2 frames
    4. water
      H2O$H2OCountedCompleter.compute1
      1. water.MRTask.compute2(MRTask.java:689)
      2. water.H2O$H2OCountedCompleter.compute1(H2O.java:1060)
      2 frames
    5. hex.tree
      ScoreBuildHistogram$Icer.compute1
      1. hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java)
      1 frame
    6. water
      H2O$H2OCountedCompleter.compute
      1. water.H2O$H2OCountedCompleter.compute(H2O.java:1056)
      1 frame
    7. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames