java.lang.OutOfMemoryError: GC overhead limit exceeded

JIRA | Simon Oxwell | 2 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    We're seeing PuppetDB regularly run out of memory every 3-4 weeks still. I did report something similar back in PDB-1484, but that was when we had troubles with 'replace facts' failing (PDB-1448). We're getting this in our logs, as a first sign: {noformat} 2015-07-02 08:05:05,553 ERROR [c.p.p.command] [08271ea0-ad7c-42c7-a0e5-3810fa668118] [replace catalog] Fatal error on attemp t 1 java.lang.OutOfMemoryError: GC overhead limit exceeded at clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57) ~[puppetdb.jar:na] at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208) ~[puppetdb.jar:na] at clojure.lang.APersistentMap.cons(APersistentMap.java:36) ~[puppetdb.jar:na] at clojure.lang.RT.conj(RT.java:562) ~[puppetdb.jar:na] at clojure.core$conj.invoke(core.clj:83) ~[puppetdb.jar:na] at schema.utils$result_builder$conjer__1981.invoke(utils.clj:139) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.MapEntry$fn__2721.invoke(core.clj:675) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.Either$fn__2541.invoke(core.clj:469) ~[na:na] at schema.core.MapEntry$fn__2715.invoke(core.clj:665) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at clojure.core$comp$fn__4192.invoke(core.clj:2403) ~[puppetdb.jar:na] at schema.core$check.invoke(core.clj:155) ~[na:na] at schema.core$validate.invoke(core.clj:160) ~[na:na] {noformat} and Java dumps a hprof file. We've had a look at that and seen: {noformat} 5,169 instances of "org.postgresql.jdbc4.Jdbc4PreparedStatement", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 584,559,496 (76.68%) bytes. 3,174 instances of "org.postgresql.core.v3.SimpleQuery", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 82,194,760 (10.78%) bytes. {noformat} Our heap size is 750M, and we have 108 puppet agents, so our heap size should be plenty (the rule of thumb suggest 256MB heap). The dashboard says we have 88,142 resources, out agents check in every 15 minutes. Last time, Kenneth suggested we tune statement-cache-size, which we have not done, as the default of 1000 seemed fine - but we seem to have five times that number. Are we seeing a possible leak around the prepared statements?

    JIRA | 2 years ago | Simon Oxwell
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  2. 0

    We're seeing PuppetDB regularly run out of memory every 3-4 weeks still. I did report something similar back in PDB-1484, but that was when we had troubles with 'replace facts' failing (PDB-1448). We're getting this in our logs, as a first sign: {noformat} 2015-07-02 08:05:05,553 ERROR [c.p.p.command] [08271ea0-ad7c-42c7-a0e5-3810fa668118] [replace catalog] Fatal error on attemp t 1 java.lang.OutOfMemoryError: GC overhead limit exceeded at clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57) ~[puppetdb.jar:na] at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208) ~[puppetdb.jar:na] at clojure.lang.APersistentMap.cons(APersistentMap.java:36) ~[puppetdb.jar:na] at clojure.lang.RT.conj(RT.java:562) ~[puppetdb.jar:na] at clojure.core$conj.invoke(core.clj:83) ~[puppetdb.jar:na] at schema.utils$result_builder$conjer__1981.invoke(utils.clj:139) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.MapEntry$fn__2721.invoke(core.clj:675) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731) ~[na:na] at clojure.core.protocols$fn__6086.invoke(protocols.clj:143) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19) ~[puppetdb.jar:na] at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[puppetdb.jar:na] at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13) ~[puppetdb.jar:na] at clojure.core$reduce.invoke(core.clj:6289) ~[puppetdb.jar:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at schema.core.Either$fn__2541.invoke(core.clj:469) ~[na:na] at schema.core.MapEntry$fn__2715.invoke(core.clj:665) ~[na:na] at schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739) ~[na:na] at schema.core$map_walker$fn__2760.invoke(core.clj:726) ~[na:na] at clojure.core$comp$fn__4192.invoke(core.clj:2403) ~[puppetdb.jar:na] at schema.core$check.invoke(core.clj:155) ~[na:na] at schema.core$validate.invoke(core.clj:160) ~[na:na] {noformat} and Java dumps a hprof file. We've had a look at that and seen: {noformat} 5,169 instances of "org.postgresql.jdbc4.Jdbc4PreparedStatement", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 584,559,496 (76.68%) bytes. 3,174 instances of "org.postgresql.core.v3.SimpleQuery", loaded by "sun.misc.Launcher$AppClassLoader @ 0xd0288580" occupy 82,194,760 (10.78%) bytes. {noformat} Our heap size is 750M, and we have 108 puppet agents, so our heap size should be plenty (the rule of thumb suggest 256MB heap). The dashboard says we have 88,142 resources, out agents check in every 15 minutes. Last time, Kenneth suggested we tune statement-cache-size, which we have not done, as the default of 1000 seemed fine - but we seem to have five times that number. Are we seeing a possible leak around the prepared statements?

    JIRA | 2 years ago | Simon Oxwell
    java.lang.OutOfMemoryError: GC overhead limit exceeded

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at clojure.lang.PersistentArrayMap.create()
    2. Clojure
      core$conj.invoke
      1. clojure.lang.PersistentArrayMap.create(PersistentArrayMap.java:57)[puppetdb.jar:na]
      2. clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:208)[puppetdb.jar:na]
      3. clojure.lang.APersistentMap.cons(APersistentMap.java:36)[puppetdb.jar:na]
      4. clojure.lang.RT.conj(RT.java:562)[puppetdb.jar:na]
      5. clojure.core$conj.invoke(core.clj:83)[puppetdb.jar:na]
      5 frames
    3. schema
      core$map_walker$fn__2760$fn__2763$fn__2764.invoke
      1. schema.utils$result_builder$conjer__1981.invoke(utils.clj:139)[na:na]
      2. schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731)[na:na]
      2 frames
    4. Clojure
      core$reduce.invoke
      1. clojure.core.protocols$fn__6086.invoke(protocols.clj:143)[puppetdb.jar:na]
      2. clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19)[puppetdb.jar:na]
      3. clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)[puppetdb.jar:na]
      4. clojure.core.protocols$fn__6078.invoke(protocols.clj:54)[puppetdb.jar:na]
      5. clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)[puppetdb.jar:na]
      6. clojure.core$reduce.invoke(core.clj:6289)[puppetdb.jar:na]
      6 frames
    5. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    6. schema.core
      MapEntry$fn__2721.invoke
      1. schema.core.MapEntry$fn__2721.invoke(core.clj:675)[na:na]
      1 frame
    7. schema
      core$map_walker$fn__2760$fn__2763$fn__2764.invoke
      1. schema.core$map_walker$fn__2760$fn__2763$fn__2764.invoke(core.clj:731)[na:na]
      1 frame
    8. Clojure
      core$reduce.invoke
      1. clojure.core.protocols$fn__6086.invoke(protocols.clj:143)[puppetdb.jar:na]
      2. clojure.core.protocols$fn__6057$G__6052__6066.invoke(protocols.clj:19)[puppetdb.jar:na]
      3. clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)[puppetdb.jar:na]
      4. clojure.core.protocols$fn__6078.invoke(protocols.clj:54)[puppetdb.jar:na]
      5. clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)[puppetdb.jar:na]
      6. clojure.core$reduce.invoke(core.clj:6289)[puppetdb.jar:na]
      6 frames
    9. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:735)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    10. schema.core
      MapEntry$fn__2715.invoke
      1. schema.core.Either$fn__2541.invoke(core.clj:469)[na:na]
      2. schema.core.MapEntry$fn__2715.invoke(core.clj:665)[na:na]
      2 frames
    11. schema
      core$map_walker$fn__2760.invoke
      1. schema.core$map_walker$fn__2760$fn__2763.invoke(core.clj:739)[na:na]
      2. schema.core$map_walker$fn__2760.invoke(core.clj:726)[na:na]
      2 frames
    12. Clojure
      core$comp$fn__4192.invoke
      1. clojure.core$comp$fn__4192.invoke(core.clj:2403)[puppetdb.jar:na]
      1 frame
    13. schema
      core$validate.invoke
      1. schema.core$check.invoke(core.clj:155)[na:na]
      2. schema.core$validate.invoke(core.clj:160)[na:na]
      2 frames