java.lang.OutOfMemoryError: GC overhead limit exceeded

JIRA | Nidhi Mehta | 3 years ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Build git branch: rel-jacobi Build git hash: 82c33a399506a133d6f0485c1a11cdac1466fd91 Build project version: 2.2.0.6 Built by: 'jenkins' On 1 machine with heap size =12g Model finishes fine in ~ 30 min On cust_churn test/train dataset 11:04:04.812 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 validation=churn_test.hex max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 On 2 machines (162/163) with heap size=20G 12:18:25.453 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 After ~45 min, Get- java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:44:47.408 FJ-9-156 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:44:47.409 FJ-9-156 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:44:47.409 FJ-9-156 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Exception in thread "Multi-UDP-R" java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive0(Native Method) 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) 12:44:56.636 #ti-UDP-R INFO WATER: at java.net.DatagramSocket.receive(DatagramSocket.java:725) 12:44:56.636 #ti-UDP-R INFO WATER: at water.MultiReceiverThread.run(MultiReceiverThread.java:64) java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 12:45:04.386 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) 12:45:13.234 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) 12:45:19.383 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:157) 12:45:23.960 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:153) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:45:23.960 FJ-9-167 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:45:23.961 FJ-9-167 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:45:23.961 FJ-9-167 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 12:45:23.962 #:54321-0 INFO WATER: at water.MemoryManager.set_goals(MemoryManager.java:99) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc(MemoryManager.java:254) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc1(MemoryManager.java:259) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:818) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:815) 12:45:23.965 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:689) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getA(AutoBuffer.java:699) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getAA(AutoBuffer.java:711) 12:45:23.966 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel.read(DTree.java) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.DRF$DRFModel.read(DRF.java) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:683) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.TreeVotes.read(TreeVotes.java) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.response(RPC.java:470) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.tcp_ack(RPC.java:438) 12:45:23.966 #:54321-0 INFO WATER: at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)

    JIRA | 3 years ago | Nidhi Mehta
    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at hex.gbm.DTree$TreeModel$CompressedTree.score()
    2. hex.gbm
      DTree$TreeModel.score0
      1. hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781)
      2. hex.gbm.DTree$TreeModel.score0(DTree.java:606)
      2 frames
    3. hex.drf
      TreeVotes.map
      1. hex.drf.TreeVotes.map(TreeVotes.java:57)
      1 frame
    4. water
      H2O$H2OCountedCompleter.compute
      1. water.MRTask2.compute2(MRTask2.java:353)
      2. water.H2O$H2OCountedCompleter.compute(H2O.java:712)
      2 frames
    5. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904)
      4. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977)
      5. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      6. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      6 frames