java.lang.IllegalArgumentException: Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5

JIRA | Arno Candel | 1 year ago
  1. 0

    h1. Problem H2O Deep Learning triggers an internal limitation of H2O on the max. size of an object in the distributed K-V store (that is the core of H2O). This limit is 256MB, and once the DL model hits that size, this condition occurs. The reason is that the Deep Learning model is currently stored as one large piece, instead of splitting it up into partial pieces. Cutting it into one piece per hidden layer won't solve this issue either, so we would have to cut a single matrix into multiple pieces to address this issue, which is somewhat cumbersome to implement. That said, a model of that size is also going to take a long time to train. Note: The memory limit has nothing to do with the number of rows of the training data (just the # columns, as that affects the first hidden layer matrix size), nor the RAM or max. allowed heap memory (that is checked separately). It also has nothing to do with the number of nodes, threads, etc. It's purely a function of the model complexity, see the next section. h2. What affects the model size? It's mainly the number of total weights and biases, multiplied by an overhead factor of x1, x2 or x3, depending on whether momentum_start==0 && momentum_stable==0 (x1), momentum > 0 (x2) or adaptive learning rate (x3) is used. Then there's some small overhead for model metrics, statistics, counters, etc. The total weights is directly given by the fully connected layers: The number of input columns (after automatic one-hot encoding of categoricals) The size of the hidden layers The number of output neurons (#classes) h2. Failing example (~25M floats * 3 for ADADELTA > 256MB) {noformat} library(h2o) h2o.init() h2o.deeplearning(x=1:4,y=5,as.h2o(iris),hidden=c(5000,5000)) {noformat} h2. Working example (~25M floats * 1 without ADADELTA and no momentum < 256MB) {noformat} library(h2o) h2o.init() h2o.deeplearning(x=1:4,y=5,as.h2o(iris),hidden=c(5000,5000), adaptive_rate=F) {noformat} h2. Output: java.lang.IllegalArgumentException: Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5 at hex.deeplearning.DeepLearningModel.<init>(DeepLearningModel.java:424) at hex.deeplearning.DeepLearning$DeepLearningDriver.buildModel(DeepLearning.java:201) at hex.deeplearning.DeepLearning$DeepLearningDriver.compute2(DeepLearning.java:171) at water.H2O$H2OCountedCompleter.compute(H2O.java:1005) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) barrier onExCompletion for hex.deeplearning.DeepLearning$DeepLearningDriver@5205f0fd h1. Solution The current solution is to reduce the number of hidden neurons, or to reduce the number of (especially categorical) features.

    JIRA | 1 year ago | Arno Candel
    java.lang.IllegalArgumentException: Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5
  2. 0

    h1. Problem H2O Deep Learning triggers an internal limitation of H2O on the max. size of an object in the distributed K-V store (that is the core of H2O). This limit is 256MB, and once the DL model hits that size, this condition occurs. The reason is that the Deep Learning model is currently stored as one large piece, instead of splitting it up into partial pieces. Cutting it into one piece per hidden layer won't solve this issue either, so we would have to cut a single matrix into multiple pieces to address this issue, which is somewhat cumbersome to implement. That said, a model of that size is also going to take a long time to train. Note: The memory limit has nothing to do with the number of rows of the training data (just the # columns, as that affects the first hidden layer matrix size), nor the RAM or max. allowed heap memory (that is checked separately). It also has nothing to do with the number of nodes, threads, etc. It's purely a function of the model complexity, see the next section. h2. What affects the model size? It's mainly the number of total weights and biases, multiplied by an overhead factor of x1, x2 or x3, depending on whether momentum_start==0 && momentum_stable==0 (x1), momentum > 0 (x2) or adaptive learning rate (x3) is used. Then there's some small overhead for model metrics, statistics, counters, etc. The total weights is directly given by the fully connected layers: The number of input columns (after automatic one-hot encoding of categoricals) The size of the hidden layers The number of output neurons (#classes) h2. Failing example (~25M floats * 3 for ADADELTA > 256MB) {noformat} library(h2o) h2o.init() h2o.deeplearning(x=1:4,y=5,as.h2o(iris),hidden=c(5000,5000)) {noformat} h2. Working example (~25M floats * 1 without ADADELTA and no momentum < 256MB) {noformat} library(h2o) h2o.init() h2o.deeplearning(x=1:4,y=5,as.h2o(iris),hidden=c(5000,5000), adaptive_rate=F) {noformat} h2. Output: java.lang.IllegalArgumentException: Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5 at hex.deeplearning.DeepLearningModel.<init>(DeepLearningModel.java:424) at hex.deeplearning.DeepLearning$DeepLearningDriver.buildModel(DeepLearning.java:201) at hex.deeplearning.DeepLearning$DeepLearningDriver.compute2(DeepLearning.java:171) at water.H2O$H2OCountedCompleter.compute(H2O.java:1005) at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) barrier onExCompletion for hex.deeplearning.DeepLearning$DeepLearningDriver@5205f0fd h1. Solution The current solution is to reduce the number of hidden neurons, or to reduce the number of (especially categorical) features.

    JIRA | 1 year ago | Arno Candel
    java.lang.IllegalArgumentException: Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5
  3. 0

    Request header too large while sending image in REST response

    Stack Overflow | 2 months ago | seriousgeek
    java.lang.IllegalArgumentException: Request header is too large
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Can not Update Gujarati Data[jsp servlet] using mysql

    Stack Overflow | 3 months ago | Aneri Bhatt
    java.lang.IllegalArgumentException: Request header is too large
  6. 0

    h3. Summary When adding 200+ (I tested with 300) issue types to an Issue Type Scheme, clicking on Save will lead to a blank page. The issue types are not added to the scheme and cannot be added at all h3. Steps to Reproduce # Create a new Issue Type Scheme # Create 300 issue types in JIRA #* I used the following shell script {code}#!/bin/bash counter=1 while [ $counter -lt 300 ]; do curl -u admin:admin -X POST --data '{"name": "'$counter'","description": "description","type": "standard"}' -H "Content-Type: application/json" http://localhost:8080/jira/rest/api/2/issuetype echo $counter let counter=counter+1 done echo All done {code} # Edit the Issue Type Scheme and add all the available Issue Types to the scheme # Click the *Save* button at the bottom of the page h3. Expected Results # All issue types are added successfully h3. Actual Results # You will be lead to a blank page with a super long URL {code}http://localhost:8080/jira/secure/admin/ConfigureOptionSchemes.jspa?selectedOptions=10007&selectedOptions=10006&selectedOptions=10008&selectedOptions=10002&selectedOptions=10004&selectedOptions=10003&selectedOptions=10118&selectedOptions=10119&selectedOptions=10120&selectedOptions=10121&selectedOptions=10122&selectedOptions=10123&selectedOptions=10124&selectedOptions=10125&selectedOptions=10126&selectedOptions=10127&selectedOptions=10128&selectedOptions=10129&selectedOptions=10130&selectedOptions=10131&selectedOptions=10132&selectedOptions=10133&selectedOptions=10134&selectedOptions=10135&selectedOptions=10136&selectedOptions=10137&selectedOptions=10138&selectedOptions=10139&selectedOptions=10140&selectedOptions=10141&selectedOptions=10142&selectedOptions=10143&selectedOptions=10144&selectedOptions=10145&selectedOptions=10146&selectedOptions=10147&selectedOptions=10148&selectedOptions=10149&selectedOptions=10150&selectedOptions=10151&selectedOptions=10152&selectedOptions=10153&selectedOptions=10154&selectedOptions=10155&selectedOptions=10156&selectedOptions=10157&selectedOptions=10158&selectedOptions=10159&selectedOptions=10160&selectedOptions=10161&selectedOptions=10162&selectedOptions=10163&selectedOptions=10164&selectedOptions=10165&selectedOptions=10166&selectedOptions=10167&selectedOptions=10168&selectedOptions=10169&selectedOptions=10170&selectedOptions=10171&selectedOptions=10172&selectedOptions=10173&selectedOptions=10174&selectedOptions=10175&selectedOptions=10176&selectedOptions=10177&selectedOptions=10178&selectedOptions=10179&selectedOptions=10180&selectedOptions=10181&selectedOptions=10182&selectedOptions=10183&selectedOptions=10184&selectedOptions=10185&selectedOptions=10186&selectedOptions=10187&selectedOptions=10188&selectedOptions=10189&selectedOptions=10190&selectedOptions=10191&selectedOptions=10192&selectedOptions=10193&selectedOptions=10194&selectedOptions=10195&selectedOptions=10196&selectedOptions=10197&selectedOptions=10198&selectedOptions=10199&selectedOptions=10200&selectedOptions=10201&selectedOptions=10202&selectedOptions=10203&selectedOptions=10204&selectedOptions=10205&selectedOptions=10206&selectedOptions=10207&selectedOptions=10208&selectedOptions=10209&selectedOptions=10210&selectedOptions=10211&selectedOptions=10212&selectedOptions=10213&selectedOptions=10214&selectedOptions=10215&selectedOptions=10216&selectedOptions=10217&selectedOptions=10218&selectedOptions=10219&selectedOptions=10220&selectedOptions=10221&selectedOptions=10222&selectedOptions=10223&selectedOptions=10224&selectedOptions=10225&selectedOptions=10226&selectedOptions=10227&selectedOptions=10228&selectedOptions=10229&selectedOptions=10230&selectedOptions=10231&selectedOptions=10232&selectedOptions=10233&selectedOptions=10234&selectedOptions=10235&selectedOptions=10236&selectedOptions=10237&selectedOptions=10238&selectedOptions=10239&selectedOptions=10240&selectedOptions=10241&selectedOptions=10242&selectedOptions=10243&selectedOptions=10244&selectedOptions=10245&selectedOptions=10246&selectedOptions=10247&selectedOptions=10248&selectedOptions=10249&selectedOptions=10250&selectedOptions=10251&selectedOptions=10252&selectedOptions=10253&selectedOptions=10254&selectedOptions=10255&selectedOptions=10256&selectedOptions=10257&selectedOptions=10258&selectedOptions=10259&selectedOptions=10260&selectedOptions=10261&selectedOptions=10262&selectedOptions=10263&selectedOptions=10264&selectedOptions=10265&selectedOptions=10266&selectedOptions=10267&selectedOptions=10268&selectedOptions=10269&selectedOptions=10270&selectedOptions=10271&selectedOptions=10272&selectedOptions=10273&selectedOptions=10274&selectedOptions=10275&selectedOptions=10276&selectedOptions=10277&selectedOptions=10278&selectedOptions=10279&selectedOptions=10280&selectedOptions=10281&selectedOptions=10282&selectedOptions=10283&selectedOptions=10284&selectedOptions=10285&selectedOptions=10286&selectedOptions=10287&selectedOptions=10288&selectedOptions=10289&selectedOptions=10290&selectedOptions=10291&selectedOptions=10292&selectedOptions=10293&selectedOptions=10294&selectedOptions=10295&selectedOptions=10296&selectedOptions=10297&selectedOptions=10298&selectedOptions=10299&selectedOptions=10300&selectedOptions=10301&selectedOptions=10302&selectedOptions=10303&selectedOptions=10304&selectedOptions=10305&selectedOptions=10306&selectedOptions=10307&selectedOptions=10308&selectedOptions=10309&selectedOptions=10310&selectedOptions=10311&selectedOptions=10312&selectedOptions=10313&selectedOptions=10314&selectedOptions=10315&selectedOptions=10316&selectedOptions=10317&selectedOptions=10318&selectedOptions=10319&selectedOptions=10320&selectedOptions=10321&selectedOptions=10322&selectedOptions=10323&selectedOptions=10324&selectedOptions=10325&selectedOptions=10326&selectedOptions=10327&selectedOptions=10328&selectedOptions=10329&selectedOptions=10330&selectedOptions=10331&selectedOptions=10332&selectedOptions=10333&selectedOptions=10334&selectedOptions=10335&selectedOptions=10336&selectedOptions=10337&selectedOptions=10338&selectedOptions=10339&selectedOptions=10340&selectedOptions=10341&selectedOptions=10342&selectedOptions=10343&selectedOptions=10344&selectedOptions=10345&selectedOptions=10346&selectedOptions=10347&selectedOptions=10348&selectedOptions=10349&selectedOptions=10350&selectedOptions=10351&selectedOptions=10352&selectedOptions=10353&selectedOptions=10354&selectedOptions=10355&selectedOptions=10356&selectedOptions=10357&selectedOptions=10358&selectedOptions=10359&selectedOptions=10360&selectedOptions=10361&selectedOptions=10362&selectedOptions=10363&selectedOptions=10364&selectedOptions=10365&selectedOptions=10366&selectedOptions=10367&selectedOptions=10368&selectedOptions=10369&selectedOptions=10370&selectedOptions=10371&selectedOptions=10372&selectedOptions=10373&selectedOptions=10374&selectedOptions=10375&selectedOptions=10376&selectedOptions=10377&selectedOptions=10378&selectedOptions=10379&selectedOptions=10380&selectedOptions=10381&selectedOptions=10382&selectedOptions=10383&selectedOptions=10384&selectedOptions=10385&selectedOptions=10386&selectedOptions=10387&selectedOptions=10388&selectedOptions=10389&selectedOptions=10390&selectedOptions=10391&selectedOptions=10392&selectedOptions=10393&selectedOptions=10394&selectedOptions=10395&selectedOptions=10396&selectedOptions=10397&selectedOptions=10398&selectedOptions=10399&selectedOptions=10400&selectedOptions=10401&selectedOptions=10402&selectedOptions=10403&selectedOptions=10404&selectedOptions=10405&selectedOptions=10406&selectedOptions=10407&selectedOptions=10408&selectedOptions=10409&selectedOptions=10410&selectedOptions=10411&selectedOptions=10412&selectedOptions=10413&selectedOptions=10414&selectedOptions=10415&selectedOptions=10416&selectedOptions=10417&selectedOptions=10418&selectedOptions=10419&selectedOptions=10420&selectedOptions=10421&selectedOptions=10422&selectedOptions=10423&selectedOptions=10424&selectedOptions=10425&selectedOptions=10426&selectedOptions=10427&selectedOptions=10428&selectedOptions=10429&selectedOptions=10430&selectedOptions=10431&selectedOptions=10432&selectedOptions=10433&selectedOptions=10434&selectedOptions=10435&selectedOptions=10436&selectedOptions=10437&selectedOptions=10438&selectedOptions=10439&selectedOptions=10440&selectedOptions=10441&selectedOptions=10442&selectedOptions=10450&selectedOptions=10443&selectedOptions=10444&selectedOptions=10445&selectedOptions=10446&selectedOptions=10447&selectedOptions=10448&selectedOptions=10449&selectedOptions=10000&selectedOptions=10001&selectedOptions=10005&selectedOptions=10100&selectedOptions=10101&selectedOptions=10102&selectedOptions=10103&selectedOptions=10104&selectedOptions=10105&selectedOptions=10106&selectedOptions=10107&selectedOptions=10108&selectedOptions=10109&selectedOptions=10110&selectedOptions=10111&selectedOptions=10112&selectedOptions=10113&selectedOptions=10114&selectedOptions=10115&selectedOptions=10116&selectedOptions=10117{code} # Following may be thrown in the logs (I wasn't able to consistently see this error being thrown) {noformat}24-Aug-2016 19:36:18.768 INFO [http-nio-7191-exec-9] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.AbstractNioInputBuffer.parseHeaders(AbstractNioInputBuffer.java:392) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1024) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1502) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1458) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745){noformat} h3. Notes # I seem to be able to add up to 200+ but once the limit is reached, I won't be able to add any more h3.Workaround No known workaround

    Atlassian JIRA | 3 months ago | Woo Yit Wei [Atlassian]
    java.lang.IllegalArgumentException: Request header is too large

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Model is too large For more information visit: http://jira.h2o.ai/browse/TN-5

      at hex.deeplearning.DeepLearningModel.<init>()
    2. hex.deeplearning
      DeepLearning$DeepLearningDriver.compute2
      1. hex.deeplearning.DeepLearningModel.<init>(DeepLearningModel.java:424)
      2. hex.deeplearning.DeepLearning$DeepLearningDriver.buildModel(DeepLearning.java:201)
      3. hex.deeplearning.DeepLearning$DeepLearningDriver.compute2(DeepLearning.java:171)
      3 frames
    3. water
      H2O$H2OCountedCompleter.compute
      1. water.H2O$H2OCountedCompleter.compute(H2O.java:1005)
      1 frame
    4. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
      4. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      5. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      5 frames