java.lang.OutOfMemoryError: GC overhead limit exceeded

JIRA | Simon Oxwell | 1 year ago
  1. 0

    We're seeing PuppetDB regularly run out of memory every 3-4 weeks still. I did report something similar back in PDB-1484, but that was when we had troubles with 'replace facts' failing (PDB-1448). We're getting this in our logs, as a first sign: {noformat} 2015-07-02 08:05:05,553 ERROR [c.p.p.command] [08271ea0-ad7c-42c7-a0e5-3810fa668118] [replace catalog] Fatal error on attemp t 1 java.lang.OutOfMemoryError: GC overhead limit exceeded at clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57) ~[puppetdb.jar:na] at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208) ~[puppetdb.jar:na] at clojure.lang.APersistentMap.cons(APersistentMap.java:36) ~[puppetdb.jar:na] at clojure.lang.RT.conj(RT.java:562) ~[puppetdb.jar:na] at clojure.core$conj.invoke(core.clj:83) ~[puppetdb.jar:na] at schema.utils$result_builder$conjer__1981.invoke(utils.clj:139) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.MapEntry$fn__2721.invoke(core.clj:675) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.Either$fn__2541.invoke(core.clj:469) ~[na:na] at schema.core.MapEntry$fn__2715.invoke(core.clj:665) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at clojure.core$comp$fn__4192.invoke(core.clj:2403) ~[puppetdb.jar:na] at schema.core$check.invoke(core.clj:155) ~[na:na] at schema.core$validate.invoke(core.clj:160) ~[na:na] {noformat} and Java dumps a hprof file. We've had a look at that and seen: {noformat} 5,169 instances of "org.postgresql.jdbc4.Jdbc4PreparedStatement", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 584,559,496 (76.68%) bytes. 3,174 instances of "org.postgresql.core.v3.SimpleQuery", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 82,194,760 (10.78%) bytes. {noformat} Our heap size is 750M, and we have 108 puppet agents, so our heap size should be plenty (the rule of thumb suggest 256MB heap). The dashboard says we have 88,142 resources, out agents check in every 15 minutes. Last time, Kenneth suggested we tune statement-cache-size, which we have not done, as the default of 1000 seemed fine - but we seem to have five times that number. Are we seeing a possible leak around the prepared statements?

    JIRA | 1 year ago | Simon Oxwell
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  2. 0

    We're seeing PuppetDB regularly run out of memory every 3-4 weeks still. I did report something similar back in PDB-1484, but that was when we had troubles with 'replace facts' failing (PDB-1448). We're getting this in our logs, as a first sign: {noformat} 2015-07-02 08:05:05,553 ERROR [c.p.p.command] [08271ea0-ad7c-42c7-a0e5-3810fa668118] [replace catalog] Fatal error on attemp t 1 java.lang.OutOfMemoryError: GC overhead limit exceeded at clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57) ~[puppetdb.jar:na] at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208) ~[puppetdb.jar:na] at clojure.lang.APersistentMap.cons(APersistentMap.java:36) ~[puppetdb.jar:na] at clojure.lang.RT.conj(RT.java:562) ~[puppetdb.jar:na] at clojure.core$conj.invoke(core.clj:83) ~[puppetdb.jar:na] at schema.utils$result_builder$conjer__1981.invoke(utils.clj:139) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.MapEntry$fn__2721.invoke(core.clj:675) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.Either$fn__2541.invoke(core.clj:469) ~[na:na] at schema.core.MapEntry$fn__2715.invoke(core.clj:665) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at clojure.core$comp$fn__4192.invoke(core.clj:2403) ~[puppetdb.jar:na] at schema.core$check.invoke(core.clj:155) ~[na:na] at schema.core$validate.invoke(core.clj:160) ~[na:na] {noformat} and Java dumps a hprof file. We've had a look at that and seen: {noformat} 5,169 instances of "org.postgresql.jdbc4.Jdbc4PreparedStatement", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 584,559,496 (76.68%) bytes. 3,174 instances of "org.postgresql.core.v3.SimpleQuery", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 82,194,760 (10.78%) bytes. {noformat} Our heap size is 750M, and we have 108 puppet agents, so our heap size should be plenty (the rule of thumb suggest 256MB heap). The dashboard says we have 88,142 resources, out agents check in every 15 minutes. Last time, Kenneth suggested we tune statement-cache-size, which we have not done, as the default of 1000 seemed fine - but we seem to have five times that number. Are we seeing a possible leak around the prepared statements?

    JIRA | 1 year ago | Simon Oxwell
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  3. 0

    OutOfMemoryError exception when reading Avro files on GCS

    Stack Overflow | 2 years ago | Jason Chou
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [platform] Unhandled event loop exception

    Eclipse Bugzilla | 2 years ago | error-reports-inbox
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  6. 0

    Java Out of Memory Problems | Wowza Support

    wowza.com | 1 year ago
    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at clojure.lang.PersistentArrayMap.create()
    2. Clojure
      core$conj.invoke
      1. clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57)[puppetdb.jar:na]
      2. clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208)[puppetdb.jar:na]
      3. clojure.lang.APersistentMap.cons(APersistentMap.java:36)[puppetdb.jar:na]
      4. clojure.lang.RT.conj(RT.java:562)[puppetdb.jar:na]
      5. clojure.core$conj.invoke(core.clj:83)[puppetdb.jar:na]
      5 frames
    3. schema
      core$map_walker$fn__2760$fn__2763$fn__2764.invoke
      1. schema.utils$result_builder$conjer__1981.invoke(utils.clj:139)[na:na]
      2. schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731)[na:na]
      2 frames
    4. Clojure
      core$reduce.invoke
      1. clojure.core.protocols$fn__6086.invoke(protocols.clj:143)[puppetdb.jar:na]
      2. clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19)[puppetdb.jar:na]
      3. clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)[puppetdb.jar:na]
      4. clojure.core.protocols$fn__6078.invoke(protocols.clj:54)[puppetdb.jar:na]
      5. clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)[puppetdb.jar:na]
      6. clojure.core$reduce.invoke(core.clj:6289)[puppetdb.jar:na]
      6 frames
    5. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    6. schema.core
      MapEntry$fn__2721.invoke
      1. schema.core.MapEntry$fn__2721.invoke(core.clj:675)[na:na]
      1 frame
    7. schema
      core$map_walker$fn__2760$fn__2763$fn__2764.invoke
      1. schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731)[na:na]
      1 frame
    8. Clojure
      core$reduce.invoke
      1. clojure.core.protocols$fn__6086.invoke(protocols.clj:143)[puppetdb.jar:na]
      2. clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19)[puppetdb.jar:na]
      3. clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)[puppetdb.jar:na]
      4. clojure.core.protocols$fn__6078.invoke(protocols.clj:54)[puppetdb.jar:na]
      5. clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)[puppetdb.jar:na]
      6. clojure.core$reduce.invoke(core.clj:6289)[puppetdb.jar:na]
      6 frames
    9. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    10. schema.core
      MapEntry$fn__2715.invoke
      1. schema.core.Either$fn__2541.invoke(core.clj:469)[na:na]
      2. schema.core.MapEntry$fn__2715.invoke(core.clj:665)[na:na]
      2 frames
    11. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    12. Clojure
      core$comp$fn__4192.invoke
      1. clojure.core$comp$fn__4192.invoke(core.clj:2403)[puppetdb.jar:na]
      1 frame
    13. schema
      core$validate.invoke
      1. schema.core$check.invoke(core.clj:155)[na:na]
      2. schema.core$validate.invoke(core.clj:160)[na:na]
      2 frames