java.lang.AssertionError: Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv

JIRA | Nidhi Mehta | 2 years ago
  1. 0

    On Master H2O Build git hash 33176195e3bb572df463a578a31af73b27ae04e4 After running a few py demos, when try to build a model using flow, and invoke a build model, get - 05-04 12:16:35.313 192.168.1.14:54321 27994 # Session INFO: Method: GET , URI: /3/Frames, route: /3/Frames, parms: {} Exception in thread "NanoHTTPD Session" java.lang.AssertionError: Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv at water.fvec.RollupStats.get(RollupStats.java:271) at water.fvec.RollupStats.get(RollupStats.java:292) at water.fvec.Vec.rollupStats(Vec.java:622) at water.fvec.Vec.byteSize(Vec.java:591) at water.fvec.Frame.byteSize(Frame.java:386) at water.api.FrameV3.<init>(FrameV3.java:210) at water.api.FramesBase.fillFromImpl(FramesBase.java:66) at water.api.FramesHandler.list(FramesHandler.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at water.api.Handler.handle(Handler.java:57) at water.api.RequestServer.handle(RequestServer.java:675) at water.api.RequestServer.serve(RequestServer.java:611) at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:436) at java.lang.Thread.run(Thread.java:744) is reproable. parse some data to h2o from R. run some models. Run some pydemos. close pybook Let the cluster remain idle for some time. Go to flow try to invoke build model say kmeans. Get the rollup stats error

    JIRA | 2 years ago | Nidhi Mehta
    java.lang.AssertionError: Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv
  2. 0

    On Master H2O Build git hash 33176195e3bb572df463a578a31af73b27ae04e4 After running a few py demos, when try to build a model using flow, and invoke a build model, get - 05-04 12:16:35.313 192.168.1.14:54321 27994 # Session INFO: Method: GET , URI: /3/Frames, route: /3/Frames, parms: {} Exception in thread "NanoHTTPD Session" java.lang.AssertionError: Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv at water.fvec.RollupStats.get(RollupStats.java:271) at water.fvec.RollupStats.get(RollupStats.java:292) at water.fvec.Vec.rollupStats(Vec.java:622) at water.fvec.Vec.byteSize(Vec.java:591) at water.fvec.Frame.byteSize(Frame.java:386) at water.api.FrameV3.<init>(FrameV3.java:210) at water.api.FramesBase.fillFromImpl(FramesBase.java:66) at water.api.FramesHandler.list(FramesHandler.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at water.api.Handler.handle(Handler.java:57) at water.api.RequestServer.handle(RequestServer.java:675) at water.api.RequestServer.serve(RequestServer.java:611) at water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:436) at java.lang.Thread.run(Thread.java:744) is reproable. parse some data to h2o from R. run some models. Run some pydemos. close pybook Let the cluster remain idle for some time. Go to flow try to invoke build model say kmeans. Get the rollup stats error

    JIRA | 2 years ago | Nidhi Mehta
    java.lang.AssertionError: Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv
  3. 0

    H2O Build git hash d651ad244a9510d286c6b6f0bc4a8d23b2f485ed Running a gbm grid search (with cross validation) on a couple of datasets and hit a few NPEs and java assertion error due to leaked frames and missing chunks. Could not repro the errors when ran them again. here are the stack traces- {code:java} [1] "/home/nidhi/auto_sklearn_csv/dataset_22_mfeat-zernike.arff.txt" Hyper-parameter: col_sample_rate_per_tree, 0.6 Hyper-parameter: learn_rate, 0.5 Hyper-parameter: max_depth, 1 Hyper-parameter: min_rows, 100 Hyper-parameter: min_split_improvement, 1e-04 Hyper-parameter: ntrees, 10000 Hyper-parameter: sample_rate, 0.8 [2016-09-05 07:06:27] failure_details: DistributedException from /172.17.2.208:28555 [2016-09-05 07:06:27] failure_stack_traces: DistributedException from /172.17.2.208:28555, caused by java.lang.NullPointerException at water.MRTask.getResult(MRTask.java:477) at water.MRTask.getResult(MRTask.java:485) at water.MRTask.doAll(MRTask.java:389) at water.MRTask.doAll(MRTask.java:383) at hex.tree.gbm.GBM$GBMDriver.buildNextKTrees(GBM.java:444) at hex.tree.SharedTree$Driver.scoreAndBuildTrees(SharedTree.java:344) at hex.tree.SharedTree$Driver.computeImpl(SharedTree.java:284) at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:168) at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Caused by: java.lang.NullPointerException at water.fvec.NewChunk.chunkD(NewChunk.java:1395) at water.fvec.NewChunk.compress2(NewChunk.java:1114) at water.fvec.NewChunk.compress(NewChunk.java:1013) at water.fvec.NewChunk.new_close(NewChunk.java:876) at water.fvec.Chunk.close(Chunk.java:561) at water.MRTask.compute2(MRTask.java:650) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) at hex.tree.gbm.GBM$GBMDriver$ComputePredAndRes$Icer.compute1(GBM$GBMDriver$ComputePredAndRes$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) ... 5 more {code} {code:java} ##when rerun runs fine [1] "/home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt" |======================================================================| 100% |====== | 8% java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 cnt=0.000000, [0.000000 - 1.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [1.000000 - 2.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [2.000000 - 3.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [3.000000 - 4.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [4.000000 - 5.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [5.000000 - 6.000000], mean/var= 0.00/ 0.00, Error: java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.bin(DHistogram.java:172) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.updateSharedHistosAndReset(DHistogram.java:617) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.accum_all(ScoreBuildHistogram.java:256) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.map(ScoreBuildHistogram.java:123) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.MRTask.compute2(MRTask.java:630) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-9 ERRR: Error: Missing chunk 9 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-21 ERRR: Error: Missing chunk 15 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt ################ {code} {code:java} ##COULD NOT REPRO ON MASTER 18 fails check nohupR.out rollup error 08-31 03:59:44.789 172.16.2.188:23555 8819 FJ-2-171 INFO: {"_train":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_train","type":"Key"},"_valid":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_valid","type":"Key"},"_nfolds":0,"_keep_cross_validation_predictions":false,"_keep_cross_validation_fold_assignment":false,"_parallelize_cross_validation":true,"_auto_rebalance":true,"_seed":4563563636689,"_fold_assignment":"AUTO","_categorical_encoding":"AUTO","_distribution":"AUTO","_tweedie_power":1.5,"_quantile_alpha":0.5,"_huber_alpha":0.9,"_ignored_columns":[""],"_ignore_const_cols":true,"_weights_column":"__internal_cv_weights__","_offset_column":null,"_fold_column":null,"_is_cv_model":true,"_score_each_iteration":false,"_max_runtime_secs":0.0,"_stopping_rounds":2,"_stopping_metric":"AUTO","_stopping_tolerance":0.001,"_response_column":"class","_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_confusion_matrix_size":20,"_checkpoint":null,"_pretrained_autoencoder":null,"_ntrees":10000,"_max_depth":3,"_min_rows":1.0,"_nbins":20,"_nbins_cats":1024,"_min_split_improvement":1.0E-5,"_histogram_type":"AUTO","_r2_stopping":1.7976931348623157E308,"_nbins_top_level":1024,"_build_tree_one_node":false,"_score_tree_interval":20,"_initial_score_interval":4000,"_score_interval":4000,"_sample_rate":1.0,"_sample_rate_per_class":null,"_col_sample_rate_change_per_level":1.0,"_col_sample_rate_per_tree":1.0,"_learn_rate":0.01,"_learn_rate_annealing":1.0,"_col_sample_rate":1.0,"_max_abs_leafnode_pred":1.7976931348623157E308} 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: Grid search: model builder for parameters hex.tree.gbm.GBMModel$GBMParameters@127558f failed! Exception: java.lang.NullPointerExceptionjava.lang.NullPointerException 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.bufB(NewChunk.java:1455) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress2(NewChunk.java:1209) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress(NewChunk.java:1013) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.new_close(NewChunk.java:876) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.Chunk.close(Chunk.java:561) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.MRTask.compute2(MRTask.java:650) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Windowed averages (window size 2) of grid's best 3 logloss metrics: [0.20842527609394138, 0.18711026295815913, 0.16014838264852577] 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Checking convergence with logloss metric: 0.20842527609394138 --> 0.16014838264852577 (still improving). 08-31 03:59:44.902 172.16.2.188:23555 8819 FJ-1-43 INFO: Creating 5 cross-validation splits with random number seed: 4563563636689 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: java.lang.AssertionError: null vec: $04ff1e000000ffffffff$nfs://home/nidhi/auto_sklearn_csv/dataset_16_mfeat-karhunen.arff.txt 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:113) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:83) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.cv_makeFramesAndBuilders(ModelBuilder.java:353) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.computeCrossValidation(ModelBuilder.java:262) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.trainModelNested(ModelBuilder.java:212) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.startBuildModel(GridSearch.java:329) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.buildModel(GridSearch.java:311) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.gridSearch(GridSearch.java:215) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.access$000(GridSearch.java:69) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch$1.compute2(GridSearch.java:136) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 08:33:42.526 172.16.2.188:23555 8819 #47664-12 INFO: GET /3/Frames, parms: {} 08-31 08:33:42.606 172.16.2.188:23555 8819 #47664-12 WARN: Leaked frame: Frame data points to one or more deleted vecs. ####################### {code}

    JIRA | 3 months ago | Nidhi Mehta
    java.lang.AssertionError: null vec: $04ff1e000000ffffffff$nfs://home/nidhi/auto_sklearn_csv/dataset_16_mfeat-karhunen.arff.txt
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    H2O Build git hash d651ad244a9510d286c6b6f0bc4a8d23b2f485ed Running a gbm grid search (with cross validation) on a couple of datasets and hit a few NPEs and java assertion error due to leaked frames and missing chunks. Could not repro the errors when ran them again. here are the stack traces- {code:java} [1] "/home/nidhi/auto_sklearn_csv/dataset_22_mfeat-zernike.arff.txt" Hyper-parameter: col_sample_rate_per_tree, 0.6 Hyper-parameter: learn_rate, 0.5 Hyper-parameter: max_depth, 1 Hyper-parameter: min_rows, 100 Hyper-parameter: min_split_improvement, 1e-04 Hyper-parameter: ntrees, 10000 Hyper-parameter: sample_rate, 0.8 [2016-09-05 07:06:27] failure_details: DistributedException from /172.17.2.208:28555 [2016-09-05 07:06:27] failure_stack_traces: DistributedException from /172.17.2.208:28555, caused by java.lang.NullPointerException at water.MRTask.getResult(MRTask.java:477) at water.MRTask.getResult(MRTask.java:485) at water.MRTask.doAll(MRTask.java:389) at water.MRTask.doAll(MRTask.java:383) at hex.tree.gbm.GBM$GBMDriver.buildNextKTrees(GBM.java:444) at hex.tree.SharedTree$Driver.scoreAndBuildTrees(SharedTree.java:344) at hex.tree.SharedTree$Driver.computeImpl(SharedTree.java:284) at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:168) at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Caused by: java.lang.NullPointerException at water.fvec.NewChunk.chunkD(NewChunk.java:1395) at water.fvec.NewChunk.compress2(NewChunk.java:1114) at water.fvec.NewChunk.compress(NewChunk.java:1013) at water.fvec.NewChunk.new_close(NewChunk.java:876) at water.fvec.Chunk.close(Chunk.java:561) at water.MRTask.compute2(MRTask.java:650) at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) at hex.tree.gbm.GBM$GBMDriver$ComputePredAndRes$Icer.compute1(GBM$GBMDriver$ComputePredAndRes$Icer.java) at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) ... 5 more {code} {code:java} ##when rerun runs fine [1] "/home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt" |======================================================================| 100% |====== | 8% java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 cnt=0.000000, [0.000000 - 1.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [1.000000 - 2.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [2.000000 - 3.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [3.000000 - 4.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [4.000000 - 5.000000], mean/var= 0.00/ 0.00, cnt=0.000000, [5.000000 - 6.000000], mean/var= 0.00/ 0.00, Error: java.lang.AssertionError: Coldata -2.0 out of range glycogen:0.0-6.0 step=1.0 nbins=6 isInt=1 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.bin(DHistogram.java:172) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.DHistogram.updateSharedHistosAndReset(DHistogram.java:617) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.accum_all(ScoreBuildHistogram.java:256) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram.map(ScoreBuildHistogram.java:123) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.MRTask.compute2(MRTask.java:630) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-1-57 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-9 ERRR: Error: Missing chunk 9 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt 09-01 11:39:15.863 172.16.2.189:43555 22281 FJ-3-21 ERRR: Error: Missing chunk 15 for $04ff2e020080ffffffff$nfs://home/nidhi/auto_sklearn_csv/oh10.wc.arff.txt ################ {code} {code:java} ##COULD NOT REPRO ON MASTER 18 fails check nohupR.out rollup error 08-31 03:59:44.789 172.16.2.188:23555 8819 FJ-2-171 INFO: {"_train":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_train","type":"Key"},"_valid":{"name":"dataset_16_mfeat-karhunen.arff.txt_model_8_cv_5_valid","type":"Key"},"_nfolds":0,"_keep_cross_validation_predictions":false,"_keep_cross_validation_fold_assignment":false,"_parallelize_cross_validation":true,"_auto_rebalance":true,"_seed":4563563636689,"_fold_assignment":"AUTO","_categorical_encoding":"AUTO","_distribution":"AUTO","_tweedie_power":1.5,"_quantile_alpha":0.5,"_huber_alpha":0.9,"_ignored_columns":[""],"_ignore_const_cols":true,"_weights_column":"__internal_cv_weights__","_offset_column":null,"_fold_column":null,"_is_cv_model":true,"_score_each_iteration":false,"_max_runtime_secs":0.0,"_stopping_rounds":2,"_stopping_metric":"AUTO","_stopping_tolerance":0.001,"_response_column":"class","_balance_classes":false,"_max_after_balance_size":5.0,"_class_sampling_factors":null,"_max_confusion_matrix_size":20,"_checkpoint":null,"_pretrained_autoencoder":null,"_ntrees":10000,"_max_depth":3,"_min_rows":1.0,"_nbins":20,"_nbins_cats":1024,"_min_split_improvement":1.0E-5,"_histogram_type":"AUTO","_r2_stopping":1.7976931348623157E308,"_nbins_top_level":1024,"_build_tree_one_node":false,"_score_tree_interval":20,"_initial_score_interval":4000,"_score_interval":4000,"_sample_rate":1.0,"_sample_rate_per_class":null,"_col_sample_rate_change_per_level":1.0,"_col_sample_rate_per_tree":1.0,"_learn_rate":0.01,"_learn_rate_annealing":1.0,"_col_sample_rate":1.0,"_max_abs_leafnode_pred":1.7976931348623157E308} 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: Grid search: model builder for parameters hex.tree.gbm.GBMModel$GBMParameters@127558f failed! Exception: java.lang.NullPointerExceptionjava.lang.NullPointerException 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.bufB(NewChunk.java:1455) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress2(NewChunk.java:1209) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.compress(NewChunk.java:1013) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.NewChunk.new_close(NewChunk.java:876) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.fvec.Chunk.close(Chunk.java:561) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.MRTask.compute2(MRTask.java:650) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute1(H2O.java:1201) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at hex.tree.ScoreBuildHistogram$Icer.compute1(ScoreBuildHistogram$Icer.java) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at water.H2O$H2OCountedCompleter.compute(H2O.java:1197) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:914) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:979) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 WARN: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Windowed averages (window size 2) of grid's best 3 logloss metrics: [0.20842527609394138, 0.18711026295815913, 0.16014838264852577] 08-31 03:59:44.858 172.16.2.188:23555 8819 FJ-1-43 INFO: Checking convergence with logloss metric: 0.20842527609394138 --> 0.16014838264852577 (still improving). 08-31 03:59:44.902 172.16.2.188:23555 8819 FJ-1-43 INFO: Creating 5 cross-validation splits with random number seed: 4563563636689 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: java.lang.AssertionError: null vec: $04ff1e000000ffffffff$nfs://home/nidhi/auto_sklearn_csv/dataset_16_mfeat-karhunen.arff.txt 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:113) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.fvec.Frame.<init>(Frame.java:83) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.cv_makeFramesAndBuilders(ModelBuilder.java:353) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.computeCrossValidation(ModelBuilder.java:262) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.ModelBuilder.trainModelNested(ModelBuilder.java:212) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.startBuildModel(GridSearch.java:329) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.buildModel(GridSearch.java:311) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.gridSearch(GridSearch.java:215) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch.access$000(GridSearch.java:69) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at hex.grid.GridSearch$1.compute2(GridSearch.java:136) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at water.H2O$H2OCountedCompleter.compute(H2O.java:1198) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 08-31 03:59:44.910 172.16.2.188:23555 8819 FJ-1-43 ERRR: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 08-31 08:33:42.526 172.16.2.188:23555 8819 #47664-12 INFO: GET /3/Frames, parms: {} 08-31 08:33:42.606 172.16.2.188:23555 8819 #47664-12 WARN: Leaked frame: Frame data points to one or more deleted vecs. ####################### {code}

    JIRA | 3 months ago | Nidhi Mehta
    java.lang.AssertionError: null vec: $04ff1e000000ffffffff$nfs://home/nidhi/auto_sklearn_csv/dataset_16_mfeat-karhunen.arff.txt
  6. 0

    Seen once recently: http://mr-0xb1:8080/job/h2o_master_DEV_gradle_build/24722/testReport/junit/water.fvec/FVecTest/testRollups/ Jul 28, 2015 12:44:08 PM From the Java log: 1) testRollups(water.fvec.FVecTest) java.lang.AssertionError: should have thrown IAE since we're requesting rollups while changing the Vec (got Vec.Writer) at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at water.fvec.FVecTest.testRollups(FVecTest.java:204) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at water.TestUtil$2$TimerStatement.evaluate(TestUtil.java:95) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at org.junit.runner.JUnitCore.run(JUnitCore.java:138) at org.junit.runner.JUnitCore.run(JUnitCore.java:117) at water.junit.H2OTestRunner.run(H2OTestRunner.java:44) at water.junit.H2OTestRunner.main(H2OTestRunner.java:53)

    JIRA | 1 year ago | Brandon Hill
    java.lang.AssertionError: should have thrown IAE since we're requesting rollups while changing the Vec (got Vec.Writer)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.AssertionError

      Rollups not possible, because Vec was deleted: $04ff16000000ffffffff$nfs://Users/nidhimehta/Desktop/german/tst.csv

      at water.fvec.RollupStats.get()
    2. water.fvec
      Frame.byteSize
      1. water.fvec.RollupStats.get(RollupStats.java:271)
      2. water.fvec.RollupStats.get(RollupStats.java:292)
      3. water.fvec.Vec.rollupStats(Vec.java:622)
      4. water.fvec.Vec.byteSize(Vec.java:591)
      5. water.fvec.Frame.byteSize(Frame.java:386)
      5 frames
    3. water.api
      FramesHandler.list
      1. water.api.FrameV3.<init>(FrameV3.java:210)
      2. water.api.FramesBase.fillFromImpl(FramesBase.java:66)
      3. water.api.FramesHandler.list(FramesHandler.java:89)
      3 frames
    4. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:606)
      4 frames
    5. water.api
      RequestServer.serve
      1. water.api.Handler.handle(Handler.java:57)
      2. water.api.RequestServer.handle(RequestServer.java:675)
      3. water.api.RequestServer.serve(RequestServer.java:611)
      3 frames
    6. water
      NanoHTTPD$HTTPSession.run
      1. water.NanoHTTPD$HTTPSession.run(NanoHTTPD.java:436)
      1 frame
    7. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:744)
      1 frame