java.sql.BatchUpdateException: Batch entry 4 ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE was aborted. Call getNextException to see the cause.

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • With current master, migration 29 fails when run against a stable database as depicted below. And while this is quite rough, it'll should demonstrate/test the problem: https://github.com/rbrw/puppetdb/tree/tmp/test-stable-db-migration You should be able to run it (after editing pg-setup-pdb to list your certs at the bottom) like this: {code:sh} PDB_ACCTEST_DB_PORT=12112 PATH=/usr/lib/postgresql/9.4/bin:"$PATH" \ acceptance/local/test-stable-migration {code} {quote} ERROR: insert or update on table "factsets" violates foreign key constraint "factsets_certname_fk" DETAIL: Key (certname)=(host-3) is not present in table "certnames". STATEMENT: ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE 2015-03-12 17:55:15,477 ERROR [main] [p.p.s.migrate] Caught SQLException during migration java.sql.BatchUpdateException: Batch entry 4 ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[postgresql-9.2-1003-jdbc4.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE] at clojure.java.jdbc$do_commands$fn__7845.invoke(jdbc.clj:188) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) [na:na] at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:187) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:na] at puppetlabs.puppetdb.scf.migrate$migrate_to_report_id_and_noop_column_and_drop_latest_reports.invoke(migrate.clj:1004) ~[na:na] {quote}
    via by Rob Browning,
  • With current master, migration 29 fails when run against a stable database as depicted below. And while this is quite rough, it'll should demonstrate/test the problem: https://github.com/rbrw/puppetdb/tree/tmp/test-stable-db-migration You should be able to run it (after editing pg-setup-pdb to list your certs at the bottom) like this: {code:sh} PDB_ACCTEST_DB_PORT=12112 PATH=/usr/lib/postgresql/9.4/bin:"$PATH" \ acceptance/local/test-stable-migration {code} {quote} ERROR: insert or update on table "factsets" violates foreign key constraint "factsets_certname_fk" DETAIL: Key (certname)=(host-3) is not present in table "certnames". STATEMENT: ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE 2015-03-12 17:55:15,477 ERROR [main] [p.p.s.migrate] Caught SQLException during migration java.sql.BatchUpdateException: Batch entry 4 ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[postgresql-9.2-1003-jdbc4.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE] at clojure.java.jdbc$do_commands$fn__7845.invoke(jdbc.clj:188) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) [na:na] at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:187) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:na] at puppetlabs.puppetdb.scf.migrate$migrate_to_report_id_and_noop_column_and_drop_latest_reports.invoke(migrate.clj:1004) ~[na:na] {quote}
    via by Rob Browning,
  • Hi, We're seeing an issue with our puppetdb since it was upgraded to 2.3.1, with some nodes not updating. We get errors like the following the the puppetdb.log {noformat} 015-04-23 14:45:10,223 ERROR [c.p.p.command] [b7ba3fa5-c61b-4461-a2b5-4317d00a92c8] [replace facts] Retrying after attempt 4, due to: java.sql.BatchUpdateException: Batch entry 0 DELETE FROM fact_values fv WHERE fv.id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in ((36,820),(36,44),(36,279),(36,67),(36,924),(36,452),(36,812),(36,105),(36,616),(36,139),(36,1019),(36,487),(36,231),(36,65),(36,15),(36,48),(36,88),(36,655),(36,118),(36,87),(36,230),(36,647),(36,493),(36,991),(36,857),(36,81),(36,2),(36,1),(36,79),(36,249),(36,911),(36,349),(36,63),(36,548),(36,141),(36,324),(36,1006),(36,40),(36,27),(36,612),(36,364),(36,530),(36,832),(36,339),(36,25),(36,526),(36,92),(36,797),(36,85),(36,642),(36,131),(36,86),(36,383),(36,58),(36,103),(36,13),(36,871),(36,68),(36,136),(36,536),(36,127),(36,624),(36,7),(36,128),(36,543),(36,519),(36,214),(36,4),(36,46),(36,482),(36,897),(36,49),(36,809),(36,704),(36,811))) was aborted. Call getNextException to see the cause. java.sql.BatchUpdateException: Batch entry 0 DELETE FROM fact_values fv WHERE fv.id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in ((36,820),(36,44),(36,279),(36,67),(36,924),(36,452),(36,812),(36,105),(36,616),(36,139),(36,1019),(36,487),(36,231),(36,65),(36,15),(36,48),(36,88),(36,655),(36,118),(36,87),(36,230),(36,647),(36,493),(36,991),(36,857),(36,81),(36,2),(36,1),(36,79),(36,249),(36,911),(36,349),(36,63),(36,548),(36,141),(36,324),(36,1006),(36,40),(36,27),(36,612),(36,364),(36,530),(36,832),(36,339),(36,25),(36,526),(36,92),(36,797),(36,85),(36,642),(36,131),(36,86),(36,383),(36,58),(36,103),(36,13),(36,871),(36,68),(36,136),(36,536),(36,127),(36,624),(36,7),(36,128),(36,543),(36,519),(36,214),(36,4),(36,46),(36,482),(36,897),(36,49),(36,809),(36,704),(36,811))) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc.internal$do_prepared_STAR_$fn__7259.invoke(internal.clj:356) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at clojure.java.jdbc.internal$do_prepared_STAR_.doInvoke(internal.clj:356) ~[na:na] at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:626) ~[puppetdb.jar:na] at clojure.java.jdbc$do_prepared.doInvoke(jdbc.clj:206) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:423) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$eval10818$delete_pending_value_id_orphans_BANG___10819$fn__10820.invoke(storage.clj:797) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10818$delete_pending_value_id_orphans_BANG___10819.invoke(storage.clj:777) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996$fn__11000$fn__11002.invoke(storage.clj:993) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996$fn__11000.invoke(storage.clj:956) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996.invoke(storage.clj:949) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198$fn__11202$fn__11204.invoke(storage.clj:1212) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198$fn__11202.invoke(storage.clj:1209) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198.invoke(storage.clj:1202) ~[na:na] at com.puppetlabs.puppetdb.command$eval13392$fn__13395$fn__13408.invoke(command.clj:381) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670$fn__7671$fn__7672.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670$fn__7671.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670.invoke(jdbc.clj:287) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649$fn__7650$fn__7651.invoke(jdbc.clj:259) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649$fn__7650.invoke(jdbc.clj:258) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649.invoke(jdbc.clj:250) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:286) ~[na:na] at com.puppetlabs.puppetdb.command$eval13392$fn__13395.invoke(command.clj:379) ~[na:na] at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$produce_message_handler$fn__13564.invoke(command.clj:647) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__13513$fn__13517.invoke(command.clj:554) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__13513.invoke(command.clj:553) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__13498$fn__13499.invoke(command.clj:507) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__13498.invoke(command.clj:506) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__13508.invoke(command.clj:529) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__13488.invoke(command.clj:467) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__13522.invoke(command.clj:569) [na:na] at com.puppetlabs.mq$create_message_listener$reify__11783.onMessage(mq.clj:270) [na:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] {noformat} and the corresponding error in the postgres.log: {noformat} RROR: update or delete on table "fact_values" violates foreign key constraint "fact_value_id_fk" on table "facts" DETAIL: Key (id)=(1524348) is still referenced from table "facts". STATEMENT: DELETE FROM fact_values fv WHERE fv.id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in ($76,$77,$78,$79,$80,$81,$82,$83,$84,$85,$86,$87,$88,$89,$90,$91,$92,$93,$94,$95,$96,$97,$98,$99,$100,$101,$102,$103,$104,$105,$106,$107,$108,$109,$110,$111,$112,$113,$114,$115,$116,$117,$118,$119,$120,$121,$122,$123,$124,$125,$126,$127,$128,$129,$130,$131,$132,$133,$134,$135,$136,$137,$138,$139,$140,$141,$142,$143,$144,$145,$146,$147,$148,$149,$150) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in (($151,$152),($153,$154),($155,$156),($157,$158),($159,$160),($161,$162),($163,$164),($165,$166),($167,$168),($169,$170),($171,$172),($173,$174),($175,$176),($177,$178),($179,$180),($181,$182),($183,$184),($185,$186),($187,$188),($189,$190),($191,$192),($193,$194),($195,$196),($197,$198),($199,$200),($201,$202),($203,$204),($205,$206),($207,$208),($209,$210),($211,$212),($213,$214),($215,$216),($217,$218),($219,$220),($221,$222),($223,$224),($225,$226),($227,$228),($229,$230),($231,$232),($233,$234),($235,$236),($237,$238),($239,$240),($241,$242),($243,$244),($245,$246),($247,$248),($249,$250),($251,$252),($253,$254),($255,$256),($257,$258),($259,$260),($261,$262),($263,$264),($265,$266),($267,$268),($269,$270),($271,$272),($273,$274),($275,$276),($277,$278),($279,$280),($281,$282),($283,$284),($285,$286),($287,$288),($289,$290),($291,$292),($293,$294),($295,$296),($297,$298),($299,$300))) {noformat} I've been able to get things moving along by performing {noformat} delete from facts where fact_value_id in ( <list of numbers from failing fact_values delete>) {noformat} and I've updated to 2.3.3, but new 'factsets' keep developing the same error. We're on CentOS 7 and postgres 9.2.7. We have about 110 puppet agents.
    via by Simon Oxwell,
  • Hi, We're seeing an issue with our puppetdb since it was upgraded to 2.3.1, with some nodes not updating. We get errors like the following the the puppetdb.log {noformat} 015-04-23 14:45:10,223 ERROR [c.p.p.command] [b7ba3fa5-c61b-4461-a2b5-4317d00a92c8] [replace facts] Retrying after attempt 4, due to: java.sql.BatchUpdateException: Batch entry 0 DELETE FROM fact_values fv WHERE fv.id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in ((36,820),(36,44),(36,279),(36,67),(36,924),(36,452),(36,812),(36,105),(36,616),(36,139),(36,1019),(36,487),(36,231),(36,65),(36,15),(36,48),(36,88),(36,655),(36,118),(36,87),(36,230),(36,647),(36,493),(36,991),(36,857),(36,81),(36,2),(36,1),(36,79),(36,249),(36,911),(36,349),(36,63),(36,548),(36,141),(36,324),(36,1006),(36,40),(36,27),(36,612),(36,364),(36,530),(36,832),(36,339),(36,25),(36,526),(36,92),(36,797),(36,85),(36,642),(36,131),(36,86),(36,383),(36,58),(36,103),(36,13),(36,871),(36,68),(36,136),(36,536),(36,127),(36,624),(36,7),(36,128),(36,543),(36,519),(36,214),(36,4),(36,46),(36,482),(36,897),(36,49),(36,809),(36,704),(36,811))) was aborted. Call getNextException to see the cause. java.sql.BatchUpdateException: Batch entry 0 DELETE FROM fact_values fv WHERE fv.id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in (90,180,2414,2667,2590,1264,2436,1561878,2568,1524348,90,2516,2436,2468,1561273,1561275,33,1210,1236,2678,90,33,2320,2478,2494,2452,90,2678,1551663,33,26,2660,1561269,896,33,578,33,1561277,1551664,90,4641,180,2590,90,83,578,1551663,83,2452,2451,1561876,1561877,180,1561272,1561876,2474,90,1551664,180,2516,1561277,2462,1561271,33,2568,26,2494,2667,1524348,2414,180,33,33,90,4616) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in ((36,820),(36,44),(36,279),(36,67),(36,924),(36,452),(36,812),(36,105),(36,616),(36,139),(36,1019),(36,487),(36,231),(36,65),(36,15),(36,48),(36,88),(36,655),(36,118),(36,87),(36,230),(36,647),(36,493),(36,991),(36,857),(36,81),(36,2),(36,1),(36,79),(36,249),(36,911),(36,349),(36,63),(36,548),(36,141),(36,324),(36,1006),(36,40),(36,27),(36,612),(36,364),(36,530),(36,832),(36,339),(36,25),(36,526),(36,92),(36,797),(36,85),(36,642),(36,131),(36,86),(36,383),(36,58),(36,103),(36,13),(36,871),(36,68),(36,136),(36,536),(36,127),(36,624),(36,7),(36,128),(36,543),(36,519),(36,214),(36,4),(36,46),(36,482),(36,897),(36,49),(36,809),(36,704),(36,811))) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc.internal$do_prepared_STAR_$fn__7259.invoke(internal.clj:356) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at clojure.java.jdbc.internal$do_prepared_STAR_.doInvoke(internal.clj:356) ~[na:na] at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:626) ~[puppetdb.jar:na] at clojure.java.jdbc$do_prepared.doInvoke(jdbc.clj:206) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:423) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$eval10818$delete_pending_value_id_orphans_BANG___10819$fn__10820.invoke(storage.clj:797) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10818$delete_pending_value_id_orphans_BANG___10819.invoke(storage.clj:777) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996$fn__11000$fn__11002.invoke(storage.clj:993) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996$fn__11000.invoke(storage.clj:956) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval10995$update_facts_BANG___10996.invoke(storage.clj:949) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198$fn__11202$fn__11204.invoke(storage.clj:1212) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198$fn__11202.invoke(storage.clj:1209) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$eval11197$replace_facts_BANG___11198.invoke(storage.clj:1202) ~[na:na] at com.puppetlabs.puppetdb.command$eval13392$fn__13395$fn__13408.invoke(command.clj:381) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670$fn__7671$fn__7672.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670$fn__7671.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__7670.invoke(jdbc.clj:287) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649$fn__7650$fn__7651.invoke(jdbc.clj:259) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649$fn__7650.invoke(jdbc.clj:258) ~[na:na] at com.puppetlabs.jdbc$eval7648$retry_sql_STAR___7649.invoke(jdbc.clj:250) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:286) ~[na:na] at com.puppetlabs.puppetdb.command$eval13392$fn__13395.invoke(command.clj:379) ~[na:na] at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$produce_message_handler$fn__13564.invoke(command.clj:647) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__13513$fn__13517.invoke(command.clj:554) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__13513.invoke(command.clj:553) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__13498$fn__13499.invoke(command.clj:507) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__13498.invoke(command.clj:506) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__13508.invoke(command.clj:529) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__13488.invoke(command.clj:467) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__13522.invoke(command.clj:569) [na:na] at com.puppetlabs.mq$create_message_listener$reify__11783.onMessage(mq.clj:270) [na:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] {noformat} and the corresponding error in the postgres.log: {noformat} RROR: update or delete on table "fact_values" violates foreign key constraint "fact_value_id_fk" on table "facts" DETAIL: Key (id)=(1524348) is still referenced from table "facts". STATEMENT: DELETE FROM fact_values fv WHERE fv.id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75) AND NOT EXISTS (SELECT 1 FROM facts f WHERE f.fact_value_id in ($76,$77,$78,$79,$80,$81,$82,$83,$84,$85,$86,$87,$88,$89,$90,$91,$92,$93,$94,$95,$96,$97,$98,$99,$100,$101,$102,$103,$104,$105,$106,$107,$108,$109,$110,$111,$112,$113,$114,$115,$116,$117,$118,$119,$120,$121,$122,$123,$124,$125,$126,$127,$128,$129,$130,$131,$132,$133,$134,$135,$136,$137,$138,$139,$140,$141,$142,$143,$144,$145,$146,$147,$148,$149,$150) AND f.fact_value_id = fv.id AND (f.factset_id, f.fact_path_id) NOT in (($151,$152),($153,$154),($155,$156),($157,$158),($159,$160),($161,$162),($163,$164),($165,$166),($167,$168),($169,$170),($171,$172),($173,$174),($175,$176),($177,$178),($179,$180),($181,$182),($183,$184),($185,$186),($187,$188),($189,$190),($191,$192),($193,$194),($195,$196),($197,$198),($199,$200),($201,$202),($203,$204),($205,$206),($207,$208),($209,$210),($211,$212),($213,$214),($215,$216),($217,$218),($219,$220),($221,$222),($223,$224),($225,$226),($227,$228),($229,$230),($231,$232),($233,$234),($235,$236),($237,$238),($239,$240),($241,$242),($243,$244),($245,$246),($247,$248),($249,$250),($251,$252),($253,$254),($255,$256),($257,$258),($259,$260),($261,$262),($263,$264),($265,$266),($267,$268),($269,$270),($271,$272),($273,$274),($275,$276),($277,$278),($279,$280),($281,$282),($283,$284),($285,$286),($287,$288),($289,$290),($291,$292),($293,$294),($295,$296),($297,$298),($299,$300))) {noformat} I've been able to get things moving along by performing {noformat} delete from facts where fact_value_id in ( <list of numbers from failing fact_values delete>) {noformat} and I've updated to 2.3.3, but new 'factsets' keep developing the same error. We're on CentOS 7 and postgres 9.2.7. We have about 110 puppet agents.
    via by Simon Oxwell,
  • Some of the the slaves in PDB-1124 (e.g 2) seem to hit this every time they attempt to gc the master database. {code} 2015-01-04 05:23:24,952 INFO [c.p.p.command] [b9d19827-c06b-4396-ac07-0afb6707e89f] [replace catalog] 2015-01-04 05:23:32,427 INFO [c.p.p.c.services] Starting sweep of stale reports (threshold: 14 days) 2015-01-04 05:23:32,439 INFO [c.p.p.c.services] Finished sweep of stale reports (threshold: 14 days) 2015-01-04 05:23:32,439 INFO [c.p.p.c.services] Starting database garbage collection 2015-01-04 05:23:35,208 ERROR [c.p.p.c.services] Error during garbage collection java.sql.BatchUpdateException: Batch entry 0 DELETE FROM resource_params_cache WHERE NOT EXISTS (SELECT * FROM catalog_resources cr WHERE cr.resource=resource_params_cache.resource) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:457) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc.internal$do_prepared_STAR_$fn__6350.invoke(internal.clj:356) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at clojure.java.jdbc.internal$do_prepared_STAR_.doInvoke(internal.clj:356) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:423) [puppetdb.jar:na] at clojure.java.jdbc$delete_rows.invoke(jdbc.clj:297) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$delete_unassociated_params_BANG_$fn__9696.invoke(storage.clj:740) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$delete_unassociated_params_BANG_.invoke(storage.clj:739) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn__9708$fn__9709.invoke(storage.clj:767) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn__9708.invoke(storage.clj:766) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_.invoke(storage.clj:765) ~[na:na] at com.puppetlabs.puppetdb.cli.services$garbage_collect_BANG_$fn__19190.invoke(services.clj:165) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761$fn__6762$fn__6763.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761$fn__6762.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761.invoke(jdbc.clj:287) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740$fn__6741$fn__6742.invoke(jdbc.clj:259) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740$fn__6741.invoke(jdbc.clj:258) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740.invoke(jdbc.clj:250) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:286) ~[na:na] at com.puppetlabs.puppetdb.cli.services$garbage_collect_BANG_.invoke(services.clj:164) ~[na:na] at com.puppetlabs.puppetdb.cli.services$perform_db_maintenance_BANG_.doInvoke(services.clj:177) [na:na] at clojure.lang.RestFn.applyTo(RestFn.java:139) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:626) [puppetdb.jar:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb$fn__19276.invoke(services.clj:347) [na:na] at clojure.lang.AFn.run(AFn.java:22) [puppetdb.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] {code} That is from host 2's puppetdb.log-20150111, which is in the ticket. This is also present in the master puppetdb logs for PE-7392 on 2014-12-25, but that is the only time it occurs for them.
    via by Wyatt Alt,
  • Hi Gang, I just upgraded to PuppetDB 2.3.1 from 2.3.0 in my development environment and PuppetDB crashed on restart. If you need more information let me know. The resulting log messages. 2015-04-01 08:44:51,294 INFO [o.e.j.u.log] Logging initialized @24441ms 2015-04-01 08:44:52,110 INFO [p.t.s.w.jetty9-core] Removing buggy security provider SunPKCS11-NSS version 1.7 2015-04-01 08:44:52,636 INFO [p.t.s.w.jetty9-service] Initializing web server(s). 2015-04-01 08:44:52,642 INFO [p.t.s.w.jetty9-service] Starting web server(s). 2015-04-01 08:44:52,818 INFO [p.t.s.w.jetty9-core] Starting web server. 2015-04-01 08:44:52,822 INFO [o.e.j.s.Server] jetty-9.2.z-SNAPSHOT 2015-04-01 08:44:52,874 INFO [o.e.j.s.ServerConnector] Started ServerConnector@66ace155{HTTP/1.1}{localhost:8080} 2015-04-01 08:44:53,150 INFO [o.e.j.s.ServerConnector] Started ServerConnector@7f17e429{SSL-HTTP/1.1}{0.0.0.0:8081} 2015-04-01 08:44:53,151 INFO [o.e.j.s.Server] Started @26300ms 2015-04-01 08:44:53,228 INFO [c.p.p.c.services] PuppetDB version 2.3.1 2015-04-01 08:44:53,362 INFO [c.p.p.s.migrate] Applying database migration version 28 2015-04-01 08:44:53,396 ERROR [c.p.p.s.migrate] Caught SQLException during migration java.sql.BatchUpdateException: Batch entry 5 DELETE FROM fact_paths t1 WHERE t1.id <> (SELECT MIN(t2.id) FROM fact_paths t2 WHERE t1.path = t2.path) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:457) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc$do_commands$fn__7301.invoke(jdbc.clj:188) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) [na:na] at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:187) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:3894) [puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.migrate$lift_fact_paths_into_facts.invoke(migrate.clj:968) ~[na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_$fn__20902$fn__20915.invoke(migrate.clj:1063) ~[na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_$fn__20902.invoke(migrate.clj:1062) [na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) [na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:1059) [na:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb$fn__21109.invoke(services.clj:292) [na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) [na:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb.invoke(services.clj:290) [na:na] at com.puppetlabs.puppetdb.cli.services$reify__21157$service_fnk__18232__auto___positional$reify__21168.start(services.clj:366) [na:na] at puppetlabs.trapperkeeper.services$eval18068$fn__18082$G__18058__18085.invoke(services.clj:10) [na:na] at puppetlabs.trapperkeeper.services$eval18068$fn__18082$G__18057__18089.invoke(services.clj:10) [na:na] at puppetlabs.trapperkeeper.internal$run_lifecycle_fn_BANG_.invoke(internal.clj:154) [na:na] at puppetlabs.trapperkeeper.internal$run_lifecycle_fns.invoke(internal.clj:182) [na:na] at puppetlabs.trapperkeeper.internal$build_app_STAR_$reify__18905.start(internal.clj:449) [na:na] at puppetlabs.trapperkeeper.internal$boot_services_STAR_$fn__18917.invoke(internal.clj:473) [na:na] at puppetlabs.trapperkeeper.internal$boot_services_STAR_.invoke(internal.clj:471) [na:na] at puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke(core.clj:113) [na:na] at puppetlabs.trapperkeeper.core$run.invoke(core.clj:144) [na:na] at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:159) [na:na] at clojure.lang.RestFn.applyTo(RestFn.java:137) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:373) [na:na] at clojure.lang.RestFn.invoke(RestFn.java:421) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:383) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:156) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at com.puppetlabs.puppetdb.core$run_command.invoke(core.clj:87) [na:na] at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:95) [na:na] at clojure.lang.RestFn.invoke(RestFn.java:436) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:388) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:160) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at clojure.main$main_opt.invoke(main.clj:315) [puppetdb.jar:na] at clojure.main$main.doInvoke(main.clj:420) [puppetdb.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:482) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:401) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:171) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.main.main(main.java:37) [puppetdb.jar:na] 2015-04-01 08:44:53,398 ERROR [c.p.p.s.migrate] Unravelled exception org.postgresql.util.PSQLException: ERROR: update or delete on table "fact_paths" violates foreign key constraint "fact_values_path_id_fk" on table "fact_values" Detail: Key (id)=(452) is still referenced from table "fact_values". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc$do_commands$fn__7301.invoke(jdbc.clj:188) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) [na:na] at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:187) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:3894) [puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.migrate$lift_fact_paths_into_facts.invoke(migrate.clj:968) ~[na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_$fn__20902$fn__20915.invoke(migrate.clj:1063) ~[na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_$fn__20902.invoke(migrate.clj:1062) [na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) [na:na] at com.puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:1059) [na:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb$fn__21109.invoke(services.clj:292) [na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) [na:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb.invoke(services.clj:290) [na:na] at com.puppetlabs.puppetdb.cli.services$reify__21157$service_fnk__18232__auto___positional$reify__21168.start(services.clj:366) [na:na] at puppetlabs.trapperkeeper.services$eval18068$fn__18082$G__18058__18085.invoke(services.clj:10) [na:na] at puppetlabs.trapperkeeper.services$eval18068$fn__18082$G__18057__18089.invoke(services.clj:10) [na:na] at puppetlabs.trapperkeeper.internal$run_lifecycle_fn_BANG_.invoke(internal.clj:154) [na:na] at puppetlabs.trapperkeeper.internal$run_lifecycle_fns.invoke(internal.clj:182) [na:na] at puppetlabs.trapperkeeper.internal$build_app_STAR_$reify__18905.start(internal.clj:449) [na:na] at puppetlabs.trapperkeeper.internal$boot_services_STAR_$fn__18917.invoke(internal.clj:473) [na:na] at puppetlabs.trapperkeeper.internal$boot_services_STAR_.invoke(internal.clj:471) [na:na] at puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke(core.clj:113) [na:na] at puppetlabs.trapperkeeper.core$run.invoke(core.clj:144) [na:na] at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:159) [na:na] at clojure.lang.RestFn.applyTo(RestFn.java:137) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at com.puppetlabs.puppetdb.cli.services$_main.doInvoke(services.clj:373) [na:na] at clojure.lang.RestFn.invoke(RestFn.java:421) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:383) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:156) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at com.puppetlabs.puppetdb.core$run_command.invoke(core.clj:87) [na:na] at com.puppetlabs.puppetdb.core$_main.doInvoke(core.clj:95) [na:na] at clojure.lang.RestFn.invoke(RestFn.java:436) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:388) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:160) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:624) [puppetdb.jar:na] at clojure.main$main_opt.invoke(main.clj:315) [puppetdb.jar:na] at clojure.main$main.doInvoke(main.clj:420) [puppetdb.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:482) [puppetdb.jar:na] at clojure.lang.Var.invoke(Var.java:401) [puppetdb.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:171) [puppetdb.jar:na] at clojure.lang.Var.applyTo(Var.java:700) [puppetdb.jar:na] at clojure.main.main(main.java:37) [puppetdb.jar:na] 2015-04-01 08:44:53,403 INFO [p.t.internal] Shutting down due to JVM shutdown hook. 2015-04-01 08:44:53,404 INFO [p.t.internal] Beginning shutdown sequence 2015-04-01 08:44:53,406 INFO [c.p.p.c.services] Shutdown request received; puppetdb exiting. 2015-04-01 08:44:53,406 INFO [p.t.s.w.jetty9-service] Shutting down web server(s). 2015-04-01 08:44:53,407 INFO [p.t.s.w.jetty9-core] Shutting down web server. 2015-04-01 08:44:53,414 INFO [o.e.j.s.ServerConnector] Stopped ServerConnector@66ace155{HTTP/1.1}{localhost:8080} 2015-04-01 08:44:53,420 INFO [o.e.j.s.ServerConnector] Stopped ServerConnector@7f17e429{SSL-HTTP/1.1}{0.0.0.0:8081} 2015-04-01 08:44:53,422 INFO [p.t.s.w.jetty9-core] Web server shutdown 2015-04-01 08:44:53,423 INFO [p.t.internal] Finished shutdown sequence ---- h3. QA Risk Analysis | Probability | Low | | Severity | Med (puppetdb crash | | Risk Level | Low | | Test Level | Unit |
    via by Pete Brown,
  • Issues with the Hive plugins: 1. The plugin uses a very specific HCatalog version. We don't know the behavior against other versions of HCatalog. 2. They don't work with avro tables because the etl apps expose avro. The apps should not expose avro. 3. Even if you change the app, they don't work if you are reading from one avro table and writing to another. The combination of HCatalogInputFormat and AvroSerDe have a bug where it reads the avro schema from a property that the input and output both set. This only happens for map-only jobs. 4. Even if you change the AvroSerDe to ignore the schema in that property, it will be able to read correctly but there is some issue writing to partitioned avro tables: {code} Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : MetaException(message:javax.jdo.JDODataStoreException: Add request failed : INSERT INTO "COLUMNS_V2" ("CD_ID","COMMENT","COLUMN_NAME","TYPE_NAME","INTEGER_IDX") VALUES (?,?,?,?,?) at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252) at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:3048) at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2992) at org.apache.hadoop.hive.metastore.ObjectStore.copyMSD(ObjectStore.java:2958) at org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:2813) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.alterTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:241) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3345) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3325) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy5.alter_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9105) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9089) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO "COLUMNS_V2" ("CD_ID","COMMENT","COLUMN_NAME","TYPE_NAME","INTEGER_IDX") VALUES ('2649',NULL,'user','string','2') was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2737) at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628) at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:409) at org.datanucleus.store.rdbms.scostore.JoinListStore.internalAdd(JoinListStore.java:304) at org.datanucleus.store.rdbms.scostore.AbstractListStore.addAll(AbstractListStore.java:136) at org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:136) at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:519) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143) at org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784) at org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2314) at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:567) at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:326) at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:193) at org.datanucleus.state.JDOStateManager.providedObjectField(JDOStateManager.java:1269) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(MStorageDescriptor.java) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(MStorageDescriptor.java) at org.datanucleus.state.JDOStateManager.provideFields(JDOStateManager.java:1346) at org.datanucleus.store.rdbms.request.UpdateRequest.execute(UpdateRequest.java:305) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateTable(RDBMSPersistenceHandler.java:417) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateObject(RDBMSPersistenceHandler.java:390) at org.datanucleus.state.JDOStateManager.flush(JDOStateManager.java:5027) at org.datanucleus.flush.FlushOrdered.execute(FlushOrdered.java:106) at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4119) at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450) at org.datanucleus.store.query.Query.prepareDatastore(Query.java:1575) at org.datanucleus.store.query.Query.executeQuery(Query.java:1760) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243) at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:3048) at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2992) at org.apache.hadoop.hive.metastore.ObjectStore.copyMSD(ObjectStore.java:2958) at org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:2813) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.alterTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:241) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3345) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3325) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy5.alter_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9105) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9089) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ) at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:969) at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:249) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: MetaException(message:javax.jdo.JDODataStoreException: Add request failed : INSERT INTO "COLUMNS_V2" ("CD_ID","COMMENT","COLUMN_NAME","TYPE_NAME","INTEGER_IDX") VALUES (?,?,?,?,?) at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252) at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:3048) at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2992) at org.apache.hadoop.hive.metastore.ObjectStore.copyMSD(ObjectStore.java:2958) at org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:2813) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.alterTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:241) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3345) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3325) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy5.alter_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9105) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9089) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO "COLUMNS_V2" ("CD_ID","COMMENT","COLUMN_NAME","TYPE_NAME","INTEGER_IDX") VALUES ('2649',NULL,'user','string','2') was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2737) at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628) at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:409) at org.datanucleus.store.rdbms.scostore.JoinListStore.internalAdd(JoinListStore.java:304) at org.datanucleus.store.rdbms.scostore.AbstractListStore.addAll(AbstractListStore.java:136) at org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:136) at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:519) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143) at org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784) at org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2314) at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:567) at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:326) at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:193) at org.datanucleus.state.JDOStateManager.providedObjectField(JDOStateManager.java:1269) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(MStorageDescriptor.java) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(MStorageDescriptor.java) at org.datanucleus.state.JDOStateManager.provideFields(JDOStateManager.java:1346) at org.datanucleus.store.rdbms.request.UpdateRequest.execute(UpdateRequest.java:305) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateTable(RDBMSPersistenceHandler.java:417) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateObject(RDBMSPersistenceHandler.java:390) at org.datanucleus.state.JDOStateManager.flush(JDOStateManager.java:5027) at org.datanucleus.flush.FlushOrdered.execute(FlushOrdered.java:106) at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4119) at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450) at org.datanucleus.store.query.Query.prepareDatastore(Query.java:1575) at org.datanucleus.store.query.Query.executeQuery(Query.java:1760) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243) at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:3048) at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2992) at org.apache.hadoop.hive.metastore.ObjectStore.copyMSD(ObjectStore.java:2958) at org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:2813) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.alterTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:241) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3345) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3325) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102) at com.sun.proxy.$Proxy5.alter_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9105) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9089) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:36822) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:36799) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result.read(ThriftHiveMetastore.java:36741) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_environment_context(ThriftHiveMetastore.java:1261) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_environment_context(ThriftHiveMetastore.java:1245) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:338) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:327) at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.updateTableSchema(FileOutputCommitterContainer.java:481) at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:874) ... 6 more {code}
    via by Albert Shau,
  • Unfortunatelly I couldn't reproduce it on fisheye.dev. *Steps to "reproduce":* # Create new review # Upload the patch # While crucible tries to anchor the patch to the repository click "cancel" # Select the repository # Click to anchor it *Expected result* The patch should be anchored to the repository *Current result* Crucible display the error dialog (bad gateway) {code} [2015-11-17 09:07:17,052] WARN org.hibernate.util.JDBCExceptionReporter SQL Error: 0, SQLState: 23505 [2015-11-17 09:07:17,052] ERROR org.hibernate.util.JDBCExceptionReporter ERROR: duplicate key value violates unique constraint "cru_fr_detail_pkey" Detail: Key (cru_revision_id, cru_name)=(461358, ANCHORED) already exists. [2015-11-17 09:07:17,158] ERROR fisheye Commit failed. Rolling back. org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:96) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:262) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:181) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1206) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:375) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137) at com.cenqua.crucible.hibernate.SessionState.commit(SessionState.java:165) at com.cenqua.crucible.hibernate.SessionState.commitAllTx(SessionState.java:118) at com.cenqua.crucible.hibernate.SessionState.commitTx(SessionState.java:109) at com.cenqua.crucible.hibernate.HibernateUtil.commitTransaction(HibernateUtil.java:334) at com.atlassian.fecru.tx.TxHelperStylePlatformTransactionManager.commit(TxHelperStylePlatformTransactionManager.java:34) at com.atlassian.fisheye.spi.impl.DefaultTxTemplate.execute(DefaultTxTemplate.java:88) at com.atlassian.crucible.actions.create.AnchorPatchAjaxAction.execute(AnchorPatchAjaxAction.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358) at com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:192) at com.cenqua.crucible.interceptors.ReviewInterceptor.intercept(ReviewInterceptor.java:52) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.cenqua.crucible.interceptors.CruciblePermissionInterceptor.intercept(CruciblePermissionInterceptor.java:53) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.cenqua.crucible.interceptors.HasCrucibleInterceptor.intercept(HasCrucibleInterceptor.java:27) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.atlassian.fisheye.xwork.SimpleConversionErrorInterceptor.intercept(SimpleConversionErrorInterceptor.java:64) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:186) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.atlassian.xwork.interceptors.XsrfTokenInterceptor.intercept(XsrfTokenInterceptor.java:100) at com.cenqua.fisheye.web.util.XsrfCheckInterceptor.intercept(XsrfCheckInterceptor.java:20) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:190) at com.opensymphony.xwork.DefaultActionProxy.execute(DefaultActionProxy.java:116) at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:274) at com.opensymphony.webwork.dispatcher.ServletDispatcher.service(ServletDispatcher.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1526) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.crucible.filters.CrucibleFilter.doFilter(CrucibleFilter.java:148) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.cenqua.fisheye.web.filters.TotalityFilter.doFilter(TotalityFilter.java:307) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:276) at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:103) at com.cenqua.fisheye.web.filters.TotalityFilter.doFilter(TotalityFilter.java:209) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256) at com.cenqua.fisheye.web.filters.CustomIncludableGzipFilter.doFilter(CustomIncludableGzipFilter.java:27) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.cenqua.fisheye.web.filters.ProductInfoFilter.doFilter(ProductInfoFilter.java:44) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.cenqua.fisheye.web.filters.UpfrontFilter.doFilter(UpfrontFilter.java:60) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at com.atlassian.fecru.profiling.ProfilingServletFilter.doFilter(ProfilingServletFilter.java:90) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1502) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into cru_fr_detail (cru_revision_id, cru_name, cru_value) values ('461358', 'ANCHORED', 'true') was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2619) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2752) at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) at sun.reflect.GeneratedMethodAccessor1268.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cenqua.crucible.hibernate.trackingstatements.TrackingStatementFactory$1.handleInvocation(TrackingStatementFactory.java:25) at com.atlassian.guava.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:79) at com.sun.proxy.$Proxy71.executeBatch(Unknown Source) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268) ... 198 more [2015-11-17 09:07:17,453] ERROR fisheye Exception "org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update" (javax.servlet.ServletException) while processing "/json/cru/CR-FE-9730/anchor-patch/" (Referer:"https://fisheye.dev.internal.atlassian.com/cru/CR-FE-9730/edit-patch/") javax.servlet.ServletException: org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:285) {code}
    via by Lukasz Kuzynski ,
  • Some of the the slaves in PDB-1124 (e.g 2) seem to hit this every time they attempt to gc the master database. {code} 2015-01-04 05:23:24,952 INFO [c.p.p.command] [b9d19827-c06b-4396-ac07-0afb6707e89f] [replace catalog] 2015-01-04 05:23:32,427 INFO [c.p.p.c.services] Starting sweep of stale reports (threshold: 14 days) 2015-01-04 05:23:32,439 INFO [c.p.p.c.services] Finished sweep of stale reports (threshold: 14 days) 2015-01-04 05:23:32,439 INFO [c.p.p.c.services] Starting database garbage collection 2015-01-04 05:23:35,208 ERROR [c.p.p.c.services] Error during garbage collection java.sql.BatchUpdateException: Batch entry 0 DELETE FROM resource_params_cache WHERE NOT EXISTS (SELECT * FROM catalog_resources cr WHERE cr.resource=resource_params_cache.resource) was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:457) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887) ~[puppetdb.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[puppetdb.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893) ~[puppetdb.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) ~[puppetdb.jar:na] at clojure.java.jdbc.internal$do_prepared_STAR_$fn__6350.invoke(internal.clj:356) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at clojure.java.jdbc.internal$do_prepared_STAR_.doInvoke(internal.clj:356) ~[na:na] at clojure.lang.RestFn.invoke(RestFn.java:423) [puppetdb.jar:na] at clojure.java.jdbc$delete_rows.invoke(jdbc.clj:297) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$delete_unassociated_params_BANG_$fn__9696.invoke(storage.clj:740) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$delete_unassociated_params_BANG_.invoke(storage.clj:739) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn__9708$fn__9709.invoke(storage.clj:767) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn__9708.invoke(storage.clj:766) ~[na:na] at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_.invoke(storage.clj:765) ~[na:na] at com.puppetlabs.puppetdb.cli.services$garbage_collect_BANG_$fn__19190.invoke(services.clj:165) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761$fn__6762$fn__6763.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761$fn__6762.invoke(jdbc.clj:290) ~[na:na] at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6761.invoke(jdbc.clj:287) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740$fn__6741$fn__6742.invoke(jdbc.clj:259) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740$fn__6741.invoke(jdbc.clj:258) ~[na:na] at com.puppetlabs.jdbc$eval6739$retry_sql_STAR___6740.invoke(jdbc.clj:250) ~[na:na] at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:286) ~[na:na] at com.puppetlabs.puppetdb.cli.services$garbage_collect_BANG_.invoke(services.clj:164) ~[na:na] at com.puppetlabs.puppetdb.cli.services$perform_db_maintenance_BANG_.doInvoke(services.clj:177) [na:na] at clojure.lang.RestFn.applyTo(RestFn.java:139) [puppetdb.jar:na] at clojure.core$apply.invoke(core.clj:626) [puppetdb.jar:na] at com.puppetlabs.puppetdb.cli.services$start_puppetdb$fn__19276.invoke(services.clj:347) [na:na] at clojure.lang.AFn.run(AFn.java:22) [puppetdb.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] {code} That is from host 2's puppetdb.log-20150111, which is in the ticket. This is also present in the master puppetdb logs for PE-7392 on 2014-12-25, but that is the only time it occurs for them.
    via by Wyatt Alt,
    • java.sql.BatchUpdateException: Batch entry 4 ALTER TABLE factsets ADD CONSTRAINT factsets_certname_fk FOREIGN KEY (certname) REFERENCES certnames(name) ON UPDATE CASCADE ON DELETE CASCADE was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746)[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887)[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)[postgresql-9.2-1003-jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893)[postgresql-9.2-1003-jdbc4.jar:na] at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)[bonecp-0.7.1.RELEASE.jar:0.7.1.RELEASE] at clojure.java.jdbc$do_commands$fn__7845.invoke(jdbc.clj:188)[na:na] at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223)[na:na] at clojure.java.jdbc$do_commands.doInvoke(jdbc.clj:187)[na:na] at clojure.lang.RestFn.invoke(RestFn.java:512)[clojure-1.6.0.jar:na] at puppetlabs.puppetdb.scf.migrate$migrate_to_report_id_and_noop_column_and_drop_latest_reports.invoke(migrate.clj:1004)[na:na]

    Users with the same issue

    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    6 more bugmates