java.lang.AssertionError: invalid weight NaN

JIRA | Kevin Normoyle | 2 years ago
  1. 0

    happens with one of the random cases in http://test.0xdata.com/job/h2o.nightly/837/testReport/junit/test_GLM2_tweedie_rand2/Basic/test_GLM2_tweedie_rand2/ cd testdir_single_jvm python test_GLM2_tweedie_rand2.py probably will fail if make a single test with these params just using covtype.data 2/GLM2 parameters: {'family': 'tweedie', 'cols': None, 'n_folds': None, 'use_all_factor_levels': None, 'higher_accuracy': None, 'ignored_cols_by_name': None, 'source': u'covtype.hex', 'lambda_search': None, 'destination_key': None, 'standardize': 0, 'beta_eps': None, 'max_iter': None, 'lambda_min_ratio': None, 'alpha': 0.2, 'response': 54, 'beta_epsilon': None, 'nlambdas': None, 'tweedie_variance_power': 1.9999999, 'ignored_cols': None, 'prior': None, 'parallelism': None, 'strong_rules_enabled': None, 'lambda': 0.0001} { "Request2": 0, "_beta": null, "_proximalPenalty": 0.0, "_runAllLambdas": "true", "_wgiven": null, "alpha": [ 0.2 ], "beta_epsilon": 0.0001, "description": "GLM2", "destination_key": "GLMModel__94ce7d98e9fcad5463a8b66959b945d8", "end_time": 0, "exception": null, "family": "tweedie", "job_key": "GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3", "lambda": [ 0.0001 ], "lambda_max": "NaN", "link": "tweedie", "max_iter": 100, "n_folds": 0, "response": {}, "response_info": { "h2o": "pytest-jenkins-25944", "node": "/10.137.71.117:54321", "redirect_url": "/2/GLMProgress.json?job_key=GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3&destination_key=GLMModel__94ce7d98e9fcad5463a8b66959b945d8", "status": "redirect", "time": 36 }, "source": { "_key": "covtype.hex" }, "standardize": "false", "start_time": 1403690504671, "state": "RUNNING", "tweedie_link_power": -0.9999998999999999, "tweedie_variance_power": 1.9999999, "variable_importances": "true" } at top of poll_url, timeoutSec: 180 redirect http://10.137.71.117:54321/2/GLMProgress.json?job_key=GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3&destination_key=GLMModel__94ce7d98e9fcad5463a8b66959b945d8 rjson: { "error": "Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN'\njava.lang.AssertionError: invalid weight NaN\n\tat hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279)\n\tat hex.FrameTask.map(FrameTask.java:581)\n\tat water.MRTask2.compute2(MRTask2.java:404)\n\tat water.MRTask2.compute2(MRTask2.java:358)\n\tat water.MRTask2.compute2(MRTask2.java:358)\n\tat water.H2O$H2OCountedCompleter.compute(H2O.java:711)\n\tat jsr166y.CountedCompleter.exec(CountedCompleter.java:429)\n\tat jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)\n\tat jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)\n\tat jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)\n\tat jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)\n", "response": { "h2o": "pytest-jenkins-25944", "node": "/10.137.71.117:54321", "status": "error", "time": 0 } } rjson error in poll_url: Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN' java.lang.AssertionError: invalid weight NaN at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) at hex.FrameTask.map(FrameTask.java:581) at water.MRTask2.compute2(MRTask2.java:404) at water.MRTask2.compute2(MRTask2.java:358) at water.MRTask2.compute2(MRTask2.java:358) at water.H2O$H2OCountedCompleter.compute(H2O.java:711) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: invalid weight NaN 03:01:46.163 FJ-0-7 INFO WATER: at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) 03:01:46.164 FJ-0-7 INFO WATER: at hex.FrameTask.map(FrameTask.java:581) 03:01:46.165 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:404) 03:01:46.167 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:358) 03:01:46.168 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:358) 03:01:46.168 FJ-0-7 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:711) 03:01:46.169 FJ-0-7 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 03:01:46.169 FJ-0-7 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 03:01:46.171 FJ-0-7 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 03:01:46.171 FJ-0-7 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 03:01:46.172 FJ-0-7 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 03:01:46.173 FJ-0-7 ERRR WATER: Job GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3(GLM2) failed. 03:01:46.173 FJ-0-7 ERRR WATER: Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN' + java.lang.AssertionError: invalid weight NaN + at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) + at hex.FrameTask.map(FrameTask.java:581) + at water.MRTask2.compute2(MRTask2.java:404) + at water.MRTask2.compute2(MRTask2.java:358) + at water.MRTask2.compute2(MRTask2.java:358) + at water.H2O$H2OCountedCompleter.compute(H2O.java:711) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

    JIRA | 2 years ago | Kevin Normoyle
    java.lang.AssertionError: invalid weight NaN
  2. 0

    happens with one of the random cases in http://test.0xdata.com/job/h2o.nightly/837/testReport/junit/test_GLM2_tweedie_rand2/Basic/test_GLM2_tweedie_rand2/ cd testdir_single_jvm python test_GLM2_tweedie_rand2.py probably will fail if make a single test with these params just using covtype.data 2/GLM2 parameters: {'family': 'tweedie', 'cols': None, 'n_folds': None, 'use_all_factor_levels': None, 'higher_accuracy': None, 'ignored_cols_by_name': None, 'source': u'covtype.hex', 'lambda_search': None, 'destination_key': None, 'standardize': 0, 'beta_eps': None, 'max_iter': None, 'lambda_min_ratio': None, 'alpha': 0.2, 'response': 54, 'beta_epsilon': None, 'nlambdas': None, 'tweedie_variance_power': 1.9999999, 'ignored_cols': None, 'prior': None, 'parallelism': None, 'strong_rules_enabled': None, 'lambda': 0.0001} { "Request2": 0, "_beta": null, "_proximalPenalty": 0.0, "_runAllLambdas": "true", "_wgiven": null, "alpha": [ 0.2 ], "beta_epsilon": 0.0001, "description": "GLM2", "destination_key": "GLMModel__94ce7d98e9fcad5463a8b66959b945d8", "end_time": 0, "exception": null, "family": "tweedie", "job_key": "GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3", "lambda": [ 0.0001 ], "lambda_max": "NaN", "link": "tweedie", "max_iter": 100, "n_folds": 0, "response": {}, "response_info": { "h2o": "pytest-jenkins-25944", "node": "/10.137.71.117:54321", "redirect_url": "/2/GLMProgress.json?job_key=GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3&destination_key=GLMModel__94ce7d98e9fcad5463a8b66959b945d8", "status": "redirect", "time": 36 }, "source": { "_key": "covtype.hex" }, "standardize": "false", "start_time": 1403690504671, "state": "RUNNING", "tweedie_link_power": -0.9999998999999999, "tweedie_variance_power": 1.9999999, "variable_importances": "true" } at top of poll_url, timeoutSec: 180 redirect http://10.137.71.117:54321/2/GLMProgress.json?job_key=GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3&destination_key=GLMModel__94ce7d98e9fcad5463a8b66959b945d8 rjson: { "error": "Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN'\njava.lang.AssertionError: invalid weight NaN\n\tat hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279)\n\tat hex.FrameTask.map(FrameTask.java:581)\n\tat water.MRTask2.compute2(MRTask2.java:404)\n\tat water.MRTask2.compute2(MRTask2.java:358)\n\tat water.MRTask2.compute2(MRTask2.java:358)\n\tat water.H2O$H2OCountedCompleter.compute(H2O.java:711)\n\tat jsr166y.CountedCompleter.exec(CountedCompleter.java:429)\n\tat jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)\n\tat jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)\n\tat jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)\n\tat jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)\n", "response": { "h2o": "pytest-jenkins-25944", "node": "/10.137.71.117:54321", "status": "error", "time": 0 } } rjson error in poll_url: Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN' java.lang.AssertionError: invalid weight NaN at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) at hex.FrameTask.map(FrameTask.java:581) at water.MRTask2.compute2(MRTask2.java:404) at water.MRTask2.compute2(MRTask2.java:358) at water.MRTask2.compute2(MRTask2.java:358) at water.H2O$H2OCountedCompleter.compute(H2O.java:711) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: invalid weight NaN 03:01:46.163 FJ-0-7 INFO WATER: at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) 03:01:46.164 FJ-0-7 INFO WATER: at hex.FrameTask.map(FrameTask.java:581) 03:01:46.165 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:404) 03:01:46.167 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:358) 03:01:46.168 FJ-0-7 INFO WATER: at water.MRTask2.compute2(MRTask2.java:358) 03:01:46.168 FJ-0-7 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:711) 03:01:46.169 FJ-0-7 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 03:01:46.169 FJ-0-7 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 03:01:46.171 FJ-0-7 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) 03:01:46.171 FJ-0-7 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 03:01:46.172 FJ-0-7 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 03:01:46.173 FJ-0-7 ERRR WATER: Job GLM2Job__aa13929f6a9bd9be7b79c0933e9141f3(GLM2) failed. 03:01:46.173 FJ-0-7 ERRR WATER: Got exception 'class java.lang.AssertionError', with msg 'invalid weight NaN' + java.lang.AssertionError: invalid weight NaN + at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279) + at hex.FrameTask.map(FrameTask.java:581) + at water.MRTask2.compute2(MRTask2.java:404) + at water.MRTask2.compute2(MRTask2.java:358) + at water.MRTask2.compute2(MRTask2.java:358) + at water.H2O$H2OCountedCompleter.compute(H2O.java:711) + at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) + at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) + at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) + at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) + at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

    JIRA | 2 years ago | Kevin Normoyle
    java.lang.AssertionError: invalid weight NaN
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Test Failure in MultiLayerTest.testDbnFaces

    GitHub | 2 years ago | smarthi
    java.lang.AssertionError: Invalid output on labels. Must not contain nan or infinite numbers.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.AssertionError

      invalid weight NaN

      at hex.glm.GLMTask$GLMIterationTask.processRow()
    2. hex.glm
      GLMTask$GLMIterationTask.processRow
      1. hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:279)
      1 frame
    3. hex
      FrameTask.map
      1. hex.FrameTask.map(FrameTask.java:581)
      1 frame
    4. water
      H2O$H2OCountedCompleter.compute
      1. water.MRTask2.compute2(MRTask2.java:404)
      2. water.MRTask2.compute2(MRTask2.java:358)
      3. water.MRTask2.compute2(MRTask2.java:358)
      4. water.H2O$H2OCountedCompleter.compute(H2O.java:711)
      4 frames
    5. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames