Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-12728

Artifactory not freeing the connections when the connection to the database is lost

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 4.13.2
    • Fix Version/s: 5.0.0
    • Component/s: Database
    • Labels:
      None

      Description

      Currently if the database connection is lost and then restored, Artifactory cann't recover from this and only need a restart. An exception then thrown when you perform any operation:

      [art-exec-3383] [ERROR] (o.q.c.ErrorLogger   :2425) - Job (artifactory.ArchiveIndexJob#5f9ba163-f078-4840-8156-cfeedaee6c1c threw an exception.
      org.quartz.SchedulerException: Job threw an unhandled exception.
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.2.1.jar:na]
      	at org.artifactory.schedule.ArtifactoryConcurrentExecutor$RunnableWrapper.run(ArtifactoryConcurrentExecutor.java:104) [artifactory-storage-common-4.11.1.jar:na]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
      Caused by: org.artifactory.storage.StorageException: Failed to get Artifactory server [art1]
      	at org.artifactory.storage.db.servers.service.ArtifactoryServersServiceImpl.getArtifactoryServer(ArtifactoryServersServiceImpl.java:58) ~[artifactory-storage-db-4.11.1.jar:na]
      	at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source) ~[na:na]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77]
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
      	at com.sun.proxy.$Proxy36.getArtifactoryServer(Unknown Source) ~[na:na]
      	at org.artifactory.storage.db.servers.service.ArtifactoryServersCommonServiceImpl.getCurrentMember(ArtifactoryServersCommonServiceImpl.java:124) ~[artifactory-core-4.11.1.jar:na]
      	at org.artifactory.addon.ha.HaAddonImpl.isPrimary(HaAddonImpl.java:139) ~[artifactory-addon-ha-4.11.1.jar:na]
      	at org.artifactory.schedule.TaskCallback.beforeExecute(TaskCallback.java:93) ~[artifactory-storage-common-4.11.1.jar:na]
      	at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:45) ~[artifactory-storage-common-4.11.1.jar:na]
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.2.1.jar:na]
      	... 4 common frames omitted
      Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [art-exec-3383] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000].
      	at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:681) ~[tomcat-jdbc-8.0.32.jar:na]
      	at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:185) ~[tomcat-jdbc-8.0.32.jar:na]
      	at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127) ~[tomcat-jdbc-8.0.32.jar:na]
      	at org.artifactory.storage.db.spring.ArtifactoryTomcatDataSource.getConnection(ArtifactoryTomcatDataSource.java:189) ~[artifactory-storage-db-4.11.1.jar:na]
      	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.5.RELEASE.jar:4.1.5.RELEASE]
      	at org.artifactory.storage.db.util.JdbcHelper.getConnection(JdbcHelper.java:64) ~[artifactory-storage-db-4.11.1.jar:na]
      	at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:95) ~[artifactory-storage-db-4.11.1.jar:na]
      	at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:79) ~[artifactory-storage-db-4.11.1.jar:na]
      	at org.artifactory.storage.db.servers.dao.ArtifactoryServersDao.getArtifactoryServer(ArtifactoryServersDao.java:71) ~[artifactory-storage-db-4.11.1.jar:na]
      	at org.artifactory.storage.db.servers.service.ArtifactoryServersServiceImpl.getArtifactoryServer(ArtifactoryServersServiceImpl.java:56) ~[artifactory-storage-db-4.11.1.jar:na]
      	... 15 common frames omitted
      
      

      It seems that the connections are not being released and Artifactory cann't reconnect to the database.

      This is part of the Thread dump that was taken:

      "http-nio-8081-exec-12" #55 daemon prio=5 os_prio=0 tid=0x00007f820802d800 nid=0x4cf2 waiting on condition [0x00007f81ea430000]
         java.lang.Thread.State: TIMED_WAITING (parking)
          at sun.misc.Unsafe.park(Native Method)
          - parking to wait for  <0x0000000099f59188> (a java.util.concurrent.CountDownLatch$Sync)
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
          at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
          at org.apache.tomcat.jdbc.pool.FairBlockingQueue.poll(FairBlockingQueue.java:153)
          at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:665)
          at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:187)
          at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132)
          at org.artifactory.storage.db.spring.ArtifactoryTomcatDataSource.getConnection(ArtifactoryTomcatDataSource.java:190)
          at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
          at org.artifactory.storage.db.util.JdbcHelper.getConnection(JdbcHelper.java:122)
          at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:157)
          at org.artifactory.storage.db.util.JdbcHelper.executeSelect(JdbcHelper.java:138)
          at org.artifactory.storage.db.fs.dao.NodesDao.exists(NodesDao.java:173)
          at org.artifactory.storage.db.fs.service.FileServiceImpl.exists(FileServiceImpl.java:80)
          at org.artifactory.repo.db.DbStoringRepoMixin.itemExists(DbStoringRepoMixin.java:542)
          at org.artifactory.repo.db.DbLocalRepo.itemExists(DbLocalRepo.java:116)
          at org.artifactory.repo.db.DbLocalRepo.assertNotBlockedByXray(DbLocalRepo.java:144)
          at org.artifactory.repo.db.DbLocalRepo.checkDownloadIsAllowed(DbLocalRepo.java:136)
          at org.artifactory.repo.RemoteRepoBase.checkDownloadIsAllowed(RemoteRepoBase.java:479)
          at org.artifactory.repo.RemoteRepoBase.getInfo(RemoteRepoBase.java:275)
          at org.artifactory.engine.DownloadServiceImpl.process(DownloadServiceImpl.java:190)
          at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
          at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
          at com.sun.proxy.$Proxy92.process(Unknown Source)
          at org.artifactory.webapp.servlet.RepoFilter.doDownload(RepoFilter.java:232)
          at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:157)
          at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:91)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
          at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:394)
          at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:270)
          at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:200)
          at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:160)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
          at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:63)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
          at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
          at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
          at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:58)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
          at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:111)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
          at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
          at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
          at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
          at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
          at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
          - locked <0x0000000084d1ddf8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
          at java.lang.Thread.run(Thread.java:745)
      
      

      Reproduced on both MySQL and Postgres and affects all DBs.

        Attachments

          Activity

            People

            • Assignee:
              davids David Shin
              Reporter:
              alexeiv Alexei Vainshtein
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: