java.lang.OutOfMemoryError: GC overhead limit exceeded

JIRA | Nidhi Mehta | 3 years ago
  1. 0

    Build git branch: rel-jacobi Build git hash: 82c33a399506a133d6f0485c1a11cdac1466fd91 Build project version: 2.2.0.6 Built by: 'jenkins' On 1 machine with heap size =12g Model finishes fine in ~ 30 min On cust_churn test/train dataset 11:04:04.812 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 validation=churn_test.hex max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 On 2 machines (162/163) with heap size=20G 12:18:25.453 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 After ~45 min, Get- java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:44:47.408 FJ-9-156 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:44:47.409 FJ-9-156 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:44:47.409 FJ-9-156 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Exception in thread "Multi-UDP-R" java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive0(Native Method) 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) 12:44:56.636 #ti-UDP-R INFO WATER: at java.net.DatagramSocket.receive(DatagramSocket.java:725) 12:44:56.636 #ti-UDP-R INFO WATER: at water.MultiReceiverThread.run(MultiReceiverThread.java:64) java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 12:45:04.386 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) 12:45:13.234 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) 12:45:19.383 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:157) 12:45:23.960 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:153) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:45:23.960 FJ-9-167 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:45:23.961 FJ-9-167 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:45:23.961 FJ-9-167 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 12:45:23.962 #:54321-0 INFO WATER: at water.MemoryManager.set_goals(MemoryManager.java:99) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc(MemoryManager.java:254) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc1(MemoryManager.java:259) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:818) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:815) 12:45:23.965 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:689) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getA(AutoBuffer.java:699) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getAA(AutoBuffer.java:711) 12:45:23.966 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel.read(DTree.java) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.DRF$DRFModel.read(DRF.java) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:683) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.TreeVotes.read(TreeVotes.java) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.response(RPC.java:470) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.tcp_ack(RPC.java:438) 12:45:23.966 #:54321-0 INFO WATER: at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)

    JIRA | 3 years ago | Nidhi Mehta
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  2. 0

    OutOfMemoryError exception when reading Avro files on GCS

    Stack Overflow | 2 years ago | Jason Chou
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  3. 0

    [platform] Unhandled event loop exception

    Eclipse Bugzilla | 2 years ago | error-reports-inbox
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Java Out of Memory Problems | Wowza Support

    wowza.com | 1 year ago
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  6. 0

    JDBC MySQL import GC Overhead Limit Exceeded

    Stack Overflow | 3 years ago | BradStevenson
    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at hex.gbm.DTree$TreeModel$CompressedTree.score()
    2. hex.gbm
      DTree$TreeModel.score0
      1. hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781)
      2. hex.gbm.DTree$TreeModel.score0(DTree.java:606)
      2 frames
    3. hex.drf
      TreeVotes.map
      1. hex.drf.TreeVotes.map(TreeVotes.java:57)
      1 frame
    4. water
      H2O$H2OCountedCompleter.compute
      1. water.MRTask2.compute2(MRTask2.java:353)
      2. water.H2O$H2OCountedCompleter.compute(H2O.java:712)
      2 frames
    5. jsr166y
      ForkJoinWorkerThread.run
      1. jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
      2. jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
      3. jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904)
      4. jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977)
      5. jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
      6. jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
      6 frames