org.apache.tomcat.jdbc.pool.PoolExhaustedException: [main] NoWait: Pool empty. Unable to fetch a connection, none available[5 in use].

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Tomcat JDBC Exception PoolExhausted
    via Stack Overflow by user1597915
    ,
  • Currently if the database connection is lost and then restored, Artifactory cann't recover from this and only need a restart. An exception then thrown when you perform any operation: {code} [art-exec-3383] [ERROR] (o.q.c.ErrorLogger :2425) - Job (artifactory.ArchiveIndexJob#5f9ba163-f078-4840-8156-cfeedaee6c1c threw an exception. org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.2.1.jar:na] at org.artifactory.schedule.ArtifactoryConcurrentExecutor$RunnableWrapper.run(ArtifactoryConcurrentExecutor.java:104) [artifactory-storage-common-4.11.1.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: org.artifactory.storage.StorageException: Failed to get Artifactory server [art1] at org.artifactory.storage.db.servers.service.ArtifactoryServersServiceImpl.getArtifactoryServer(ArtifactoryServersServiceImpl.java:58) ~[artifactory-storage-db-4.11.1.jar:na] at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE] at com.sun.proxy.$Proxy36.getArtifactoryServer(Unknown Source) ~[na:na] at org.artifactory.storage.db.servers.service.ArtifactoryServersCommonServiceImpl.getCurrentMember(ArtifactoryServersCommonServiceImpl.java:124) ~[artifactory-core-4.11.1.jar:na] at org.artifactory.addon.ha.HaAddonImpl.isPrimary(HaAddonImpl.java:139) ~[artifactory-addon-ha-4.11.1.jar:na] at org.artifactory.schedule.TaskCallback.beforeExecute(TaskCallback.java:93) ~[artifactory-storage-common-4.11.1.jar:na] at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:45) ~[artifactory-storage-common-4.11.1.jar:na] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.2.1.jar:na] ... 4 common frames omitted Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [art-exec-3383] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000]. at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:681) ~[tomcat-jdbc-8.0.32.jar:na] at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:185) ~[tomcat-jdbc-8.0.32.jar:na] at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127) ~[tomcat-jdbc-8.0.32.jar:na] at org.artifactory.storage.db.spring.ArtifactoryTomcatDataSource.getConnection(ArtifactoryTomcatDataSource.java:189) ~[artifactory-storage-db-4.11.1.jar:na] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.5.RELEASE.jar:4.1.5.RELEASE] at org.artifactory.storage.db.util.JdbcHelper.getConnection(JdbcHelper.java:64) ~[artifactory-storage-db-4.11.1.jar:na] at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:95) ~[artifactory-storage-db-4.11.1.jar:na] at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:79) ~[artifactory-storage-db-4.11.1.jar:na] at org.artifactory.storage.db.servers.dao.ArtifactoryServersDao.getArtifactoryServer(ArtifactoryServersDao.java:71) ~[artifactory-storage-db-4.11.1.jar:na] at org.artifactory.storage.db.servers.service.ArtifactoryServersServiceImpl.getArtifactoryServer(ArtifactoryServersServiceImpl.java:56) ~[artifactory-storage-db-4.11.1.jar:na] ... 15 common frames omitted {code} It seems that the connections are not being released and Artifactory cann't reconnect to the database. This is part of the Thread dump that was taken: {code} "http-nio-8081-exec-12" #55 daemon prio=5 os_prio=0 tid=0x00007f820802d800 nid=0x4cf2 waiting on condition [0x00007f81ea430000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000099f59188> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.tomcat.jdbc.pool.FairBlockingQueue.poll(FairBlockingQueue.java:153) at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:665) at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:187) at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132) at org.artifactory.storage.db.spring.ArtifactoryTomcatDataSource.getConnection(ArtifactoryTomcatDataSource.java:190) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) at org.artifactory.storage.db.util.JdbcHelper.getConnection(JdbcHelper.java:122) at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:157) at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:138) at org.artifactory.storage.db.fs.dao.NodesDao.exists(NodesDao.java:173) at org.artifactory.storage.db.fs.service.FileServiceImpl.exists(FileServiceImpl.java:80) at org.artifactory.repo.db.DbStoringRepoMixin.itemExists(DbStoringRepoMixin.java:542) at org.artifactory.repo.db.DbLocalRepo.itemExists(DbLocalRepo.java:116) at org.artifactory.repo.db.DbLocalRepo.assertNotBlockedByXray(DbLocalRepo.java:144) at org.artifactory.repo.db.DbLocalRepo.checkDownloadIsAllowed(DbLocalRepo.java:136) at org.artifactory.repo.RemoteRepoBase.checkDownloadIsAllowed(RemoteRepoBase.java:479) at org.artifactory.repo.RemoteRepoBase.getInfo(RemoteRepoBase.java:275) at org.artifactory.engine.DownloadServiceImpl.process(DownloadServiceImpl.java:190) at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy92.process(Unknown Source) at org.artifactory.webapp.servlet.RepoFilter.doDownload(RepoFilter.java:232) at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:157) at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:394) at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:270) at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:200) at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:160) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:111) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) - locked <0x0000000084d1ddf8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) {code} Reproduced on both MySQL and Postgres and affects all DBs.
    via by Alexei Vainshtein,
    • org.apache.tomcat.jdbc.pool.PoolExhaustedException: [main] NoWait: Pool empty. Unable to fetch a connection, none available[5 in use]. at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:679) at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:468) at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:143) at org.apache.tomee.jdbc.TomEEDataSourceCreator$TomEEConnectionPool.<init>(TomEEDataSourceCreator.java:221)

    Users with the same issue

    Axelo
    Axelo54 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,
    Unknown visitor1 times, last one,