java.lang.OutOfMemoryError: GC overhead limit exceeded

JIRA | Nidhi Mehta | 3 years ago
  1. 0

    Build git branch: rel-jacobi Build git hash: 82c33a399506a133d6f0485c1a11cdac1466fd91 Build project version: 2.2.0.6 Built by: 'jenkins' On 1 machine with heap size =12g Model finishes fine in ~ 30 min On cust_churn test/train dataset 11:04:04.812 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 validation=churn_test.hex max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 On 2 machines (162/163) with heap size=20G 12:18:25.453 # Session INFO HTTPD: GET /2/DRF.html source=churn_train.hex mtries=-1 score_each_iteration=0 max_depth=999 response=churn nbins=20 seed=-1 classification=1 build_tree_per_node=1 ignored_cols=98,103,104,120,121,122,124,132,138,139,140,141,146,147,149,152,153,154,155,156,157,159,160,161,170,172 ntrees=800 sample_rate=0.6666666865348816 min_rows=1 importance=1 After ~45 min, Get- java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:44:47.408 FJ-9-156 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:44:47.408 FJ-9-156 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:44:47.409 FJ-9-156 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:44:47.409 FJ-9-156 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:44:47.409 FJ-9-156 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) Exception in thread "Multi-UDP-R" java.lang.OutOfMemoryError: GC overhead limit exceeded 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive0(Native Method) 12:44:56.635 #ti-UDP-R INFO WATER: at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) 12:44:56.636 #ti-UDP-R INFO WATER: at java.net.DatagramSocket.receive(DatagramSocket.java:725) 12:44:56.636 #ti-UDP-R INFO WATER: at water.MultiReceiverThread.run(MultiReceiverThread.java:64) java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 12:45:04.386 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) 12:45:13.234 FJ-9-167 INFO WATER: at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) 12:45:19.383 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:157) 12:45:23.960 FJ-9-167 INFO WATER: at water.AutoBuffer.<init>(AutoBuffer.java:153) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.score(DTree.java:781) 12:45:23.960 FJ-9-167 INFO WATER: at hex.gbm.DTree$TreeModel.score0(DTree.java:606) 12:45:23.960 FJ-9-167 INFO WATER: at hex.drf.TreeVotes.map(TreeVotes.java:57) 12:45:23.961 FJ-9-167 INFO WATER: at water.MRTask2.compute2(MRTask2.java:353) 12:45:23.961 FJ-9-167 INFO WATER: at water.H2O$H2OCountedCompleter.compute(H2O.java:712) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.CountedCompleter.exec(CountedCompleter.java:429) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.popAndExecAll(ForkJoinPool.java:904) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:977) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) 12:45:23.961 FJ-9-167 INFO WATER: at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) java.lang.AssertionError: I am really confused about the heap usage; MEM_MAX=19088801792 heapUsed=20816379152 12:45:23.962 #:54321-0 INFO WATER: at water.MemoryManager.set_goals(MemoryManager.java:99) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc(MemoryManager.java:254) 12:45:23.965 #:54321-0 INFO WATER: at water.MemoryManager.malloc1(MemoryManager.java:259) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:818) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.getA1(AutoBuffer.java:815) 12:45:23.965 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel$CompressedTree.read(DTree.java) 12:45:23.965 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:689) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getA(AutoBuffer.java:699) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.getAA(AutoBuffer.java:711) 12:45:23.966 #:54321-0 INFO WATER: at hex.gbm.DTree$TreeModel.read(DTree.java) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.DRF$DRFModel.read(DRF.java) 12:45:23.966 #:54321-0 INFO WATER: at water.AutoBuffer.get(AutoBuffer.java:683) 12:45:23.966 #:54321-0 INFO WATER: at hex.drf.TreeVotes.read(TreeVotes.java) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.response(RPC.java:470) 12:45:23.966 #:54321-0 INFO WATER: at water.RPC.tcp_ack(RPC.java:438) 12:45:23.966 #:54321-0 INFO WATER: at water.TCPReceiverThread$TCPReaderThread.run(TCPReceiverThread.java:86)

    JIRA | 3 years ago | Nidhi Mehta
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  2. 0

    OutOfMemory Error in 126.330 while indexing files

    YouTrack | 4 years ago
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  3. 0

    java.lang.OutOfMemoryError: getNewTla&nbsp; in coherence production cluster!

    Oracle Community | 4 years ago | 923561
    java.lang.OutOfMemoryError: java/net/Inet4Address, size 24B
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    java.lang.OutOfMemoryError: heap allocation failed

    Oracle Community | 7 years ago | 843790
    java.lang.OutOfMemoryError: heap allocation failed
  6. 0

    OutOfMemoryError exception when reading Avro files on GCS

    Stack Overflow | 2 years ago | Jason Chou
    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at java.net.PlainDatagramSocketImpl.receive0()
    2. Java RT
      DatagramSocket.receive
      1. java.net.PlainDatagramSocketImpl.receive0(Native Method)
      2. java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
      3. java.net.DatagramSocket.receive(DatagramSocket.java:725)
      3 frames
    3. water
      MultiReceiverThread.run
      1. water.MultiReceiverThread.run(MultiReceiverThread.java:64)
      1 frame