java.lang.AssertionError

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • H2O Build git hash d651ad244a9510d286c6b6f0bc4a8d23b2f485ed Running a gbm grid search (with cross validation) on a couple of datasets and hit a few NPEs and java assertion error due to leaked frames and missing chunks. Could not repro the errors when ran them again. here are the stack traces- {code:java} [1] "/home/nidhi/auto_sklearn_csv/dataset_22_mfeat-zernike.arff.txt" Hyper-parameter: col_sample_rate_per_tree, 0.6 Hyper-parameter: learn_rate, 0.5 Hyper-parameter: max_depth, 1 Hyper-parameter: min_rows, 100 Hyper-parameter: min_split_improvement, 1e-04 Hyper-parameter: ntrees, 10000 Hyper-parameter: sample_rate, 0.8 [2016-09-05 07:06:27] failure_details: DistributedException from /172.17.2.208:28555 [2016-09-05 07:06:27] failure_stack_traces: DistributedException from /172.17.2.208:28555, caused by java.lang.NullPointerException at water.MRTask.getResult(MRTask.java:477) at water.MRTask.getResult(MRTask.java:485) at water.MRTask.doAll(MRTask.java:389) at water.MRTask.doAll(MRTask.java:383) at hex.tree.gbm.GBM$GBMDriver.buildNextKTrees(GBM.java:444) at hex.tree.SharedTree$Driver.scoreAndBuildTrees(SharedTree.java:344) at hex.tree.SharedTree$Driver.computeImpl(SharedTree.java:284) at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:168) at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Caused by: java.lang.NullPointerException at water.fvec.NewChunk.chunkD(NewChunk.java:1395) at water.fvec.NewChunk.compress2(NewChunk.java:1114) at water.fvec.NewChunk.compress(NewChunk.java:1013) at water.fvec.NewChunk.new_close(NewChunk.java:876) at water.fvec.Chunk.close(Chunk.java:561) at water.MRTask.compute2(MRTask.java:650) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) at hex.tree.gbm.GBM$GBMDriver$ComputePredAndRes$Icer.compute1(GBM$GBMDriver$ComputePredAndRes$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) ... 5 more {code} {code:java} ##when rerun runs fine [1] "/home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt" |======================================================================| 100% |====== | 8% java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 cnt=0.000000, [0.000000 - 1.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [1.000000 - 2.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [2.000000 - 3.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [3.000000 - 4.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [4.000000 - 5.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [5.000000 - 6.000000], mean/var= 0.00/ 0.00, Error: java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.bin(DHistogram.java:172) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.updateSharedHistosAndReset(DHistogram.java:617) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.accum_all(ScoreBuildHistogram.java:256) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.map(ScoreBuildHistogram.java:123) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.MRTask.compute2(MRTask.java:630) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-9 ERRR: Error: Missing chunk 9 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-21 ERRR: Error: Missing chunk 15 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt ################ {code} {code:java} ##COULD NOT REPRO ON MASTER 18 fails check nohupR.out rollup error 08-31 03:59:44.789 172.16.2.188:23555 8819 FJ-2-171 INFO: {"_train":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_train","type":"Key"},"_valid":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_valid","type":"Key"},"_nfolds":0,"_keep_cross_validation_predictions":false,"_keep_cross_validation_fold_assignment":false,"_parallelize_cross_validation":true,"_auto_rebalance":true,"_seed":4563563636689,"_fold_assignment":"AUTO","_categorical_encoding":"AUTO","_distribution":"AUTO","_tweedie_power":1.5,"_quantile_alpha":0.5,"_huber_alpha":0.9,"_ignored_columns":[""],"_ignore_const_cols":true,"_weights_column":"__internal_cv_weights__","_offset_column":null,"_fold_column":null,"_is_cv_model":true,"_score_each_iteration":false,"_max_runtime_secs":0.0,"_stopping_rounds":2,"_stopping_metric":"AUTO","_stopping_tolerance":0.001,"_response_column":"class","_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_confusion_matrix_size":20,"_checkpoint":null,"_pretrained_autoencoder":null,"_ntrees":10000,"_max_depth":3,"_min_rows":1.0,"_nbins":20,"_nbins_cats":1024,"_min_split_improvement":1.0E-5,"_histogram_type":"AUTO","_r2_stopping":1.7976931348623157E308,"_nbins_top_level":1024,"_build_tree_one_node":false,"_score_tree_interval":20,"_initial_score_interval":4000,"_score_interval":4000,"_sample_rate":1.0,"_sample_rate_per_class":null,"_col_sample_rate_change_per_level":1.0,"_col_sample_rate_per_tree":1.0,"_learn_rate":0.01,"_learn_rate_annealing":1.0,"_col_sample_rate":1.0,"_max_abs_leafnode_pred":1.7976931348623157E308} 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: Grid search: model builder for parameters hex.tree.gbm.GBMModel$GBMParameters@127558f failed! Exception: java.lang.NullPointerExceptionjava.lang.NullPointerException 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.bufB(NewChunk.java:1455) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress2(NewChunk.java:1209) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress(NewChunk.java:1013) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.new_close(NewChunk.java:876) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.Chunk.close(Chunk.java:561) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.MRTask.compute2(MRTask.java:650) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Windowed averages (window size 2) of grid's best 3 logloss metrics: [0.20842527609394138, 0.18711026295815913, 0.16014838264852577] 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Checking convergence with logloss metric: 0.20842527609394138 --> 0.16014838264852577 (still improving). 08-31 03:59:44.902 172.16.2.188:23555 8819 FJ-1-43 INFO: Creating 5 cross-validation splits with random number seed: 4563563636689 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: java.lang.AssertionError: null vec: $04ff1e000000ffffffff$nfs://home/nidhi/auto_sklearn_csv/dataset_16_mfeat-karhunen.arff.txt 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:113) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:83) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.cv_makeFramesAndBuilders(ModelBuilder.java:353) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.computeCrossValidation(ModelBuilder.java:262) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.trainModelNested(ModelBuilder.java:212) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.startBuildModel(GridSearch.java:329) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.buildModel(GridSearch.java:311) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.gridSearch(GridSearch.java:215) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.access$000(GridSearch.java:69) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch$1.compute2(GridSearch.java:136) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 08:33:42.526 172.16.2.188:23555 8819 #47664-12 INFO: GET /3/Frames, parms: {} 08-31 08:33:42.606 172.16.2.188:23555 8819 #47664-12 WARN: Leaked frame: Frame data points to one or more deleted vecs. ####################### {code}
    via by Nidhi Mehta,
  • I'm trying to use RF and GBM in R to build a regression model. I'm using cross-validation for testing. For one of the numeric features, I've got an error that the value is not in the expected range: {code:java} java.lang.AssertionError: Coldata 0.919531598844283 out of range hotelPriceChangeMean:0.9196702296720011-1.1848896269111302 step=0.013260969861956457 nbins=20 isInt=0 cnt=209.000000, [0.919670 - 0.932931], mean/var= -0.35/ 0.00, cnt=43.000000, [0.932931 - 0.946192], mean/var= -0.35/ 0.00, cnt=71.000000, [0.946192 - 0.959453], mean/var= -0.34/ 0.00, cnt=0.000000, [0.959453 - 0.972714], mean/var= 0.00/ 0.00, cnt=0.000000, [0.972714 - 0.985975], mean/var= 0.00/ 0.00, cnt=0.000000, [0.985975 - 0.999236], mean/var= 0.00/ 0.00, cnt=0.000000, [0.999236 - 1.012497], mean/var= 0.00/ 0.00, cnt=0.000000, [1.012497 - 1.025758], mean/var= 0.00/ 0.00, cnt=0.000000, [1.025758 - 1.039019], mean/var= 0.00/ 0.00, cnt=0.000000, [1.039019 - 1.052280], mean/var= 0.00/ 0.00, cnt=0.000000, [1.052280 - 1.065541], mean/var= 0.00/ 0.00, cnt=0.000000, [1.065541 - 1.078802], mean/var= 0.00/ 0.00, cnt=0.000000, [1.078802 - 1.092063], mean/var= 0.00/ 0.00, cnt=0.000000, [1.092063 - 1.105324], mean/var= 0.00/ 0.00, cnt=0.000000, [1.105324 - 1.118585], mean/var= 0.00/ 0.00, cnt=0.000000, [1.118585 - 1.131846], mean/var= 0.00/ 0.00, cnt=0.000000, [1.131846 - 1.145107], mean/var= 0.00/ 0.00, cnt=0.000000, [1.145107 - 1.158368], mean/var= 0.00/ 0.00, cnt=0.000000, [1.158368 - 1.171629], mean/var= 0.00/ 0.00, cnt=0.000000, [1.171629 - 1.184890], mean/var= 0.00/ 0.00, at hex.tree.DHistogram.bin(DHistogram.java:209) at hex.tree.DHistogram.updateHisto(DHistogram.java:416) at hex.tree.ScoreBuildHistogram2$ComputeHistoThread.computeChunk(ScoreBuildHistogram2.java:331) at hex.tree.ScoreBuildHistogram2$ComputeHistoThread.map(ScoreBuildHistogram2.java:306) at water.LocalMR.compute2(LocalMR.java:84) at water.LocalMR.compute2(LocalMR.java:76) at water.LocalMR.compute2(LocalMR.java:76) at water.LocalMR.compute2(LocalMR.java:76) at water.H2O$H2OCountedCompleter.compute(H2O.java:1220) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Error: java.lang.AssertionError: Coldata 0.919531598844283 out of range hotelPriceChangeMean:0.9196702296720011-1.1848896269111302 step=0.013260969861956457 nbins=20 isInt=0 {code} *H2O version:* 3.10.3.3 *R version:* 3.3.2 *Java:* java version "1.7.0_121" OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.14.04.3) OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
    via by Adam Harasimowicz,
    • java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.bin(DHistogram.java:172) at hex.tree.DHistogram.updateSharedHistosAndReset(DHistogram.java:617) at hex.tree.ScoreBuildHistogram.accum_all(ScoreBuildHistogram.java:256) at hex.tree.ScoreBuildHistogram.map(ScoreBuildHistogram.java:123) at water.MRTask.compute2(MRTask.java:630) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
    No Bugmate found.