[RTFACT-19853] db connection pool empty Created: 08/Aug/19  Updated: 11/Aug/19  Resolved: 11/Aug/19

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Database
Affects Version/s: 6.5.17
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Andreas Pelzer Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None


 Description   

Artifactory freezes with error messages like

2019-08-08 09:56:49,167 [http-nio-8901-exec-218] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) - Failed to check for children of: 'app-release-local:': [http-nio-8901-exec-218] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:98; busy:98; idle:0; lastwait:120000].
org.artifactory.storage.fs.VfsException: Failed to check for children of: 'app-release-local:': [http-nio-8901-exec-218] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:98; busy:98; idle:0; lastwait:120000].
...
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-8901-exec-218] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:98; busy:98; idle:0; lastwait:120000].
        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:684)
...

On the Oracle database (which is used for indizes, not for files), it has been recognized that the first session was establied at 0:44 and the last session at 4:46. The issue has been recognized at 8:00.
There were 97 open connections (most of them active) that remained after shutdown of the application server. The load on the database server was extreme high due the these connections. A reboot of the database was necessary to kill all these sessions and to get the system operate normal again.

So far, the db.properties value pool.max.active has not been used thus should have had the default value 100 according to ConfiguringtheDatabase-AdvancedSettings.
The value has been increased to 500. However, we fear that there is an issue that database connections are not handled properly

This issue happened already twice and only after updating from 6.5.2 to 6.5.18, however it is unclear if the update is related to this issue.
Actually, we have version 6.5.18, however in cannot be selected in the Jira field "Affects Version/s".



 Comments   
Comment by Ariel Seftel [ 11/Aug/19 ]

It seems that you reached the maximum connections between tomcat to your database. 
You can increase the 'pool.max.active' in the 'db.properties' file located in $ARTIFACTORY_HOME/etc

By default, the value is set to 100.

In case you don't have this parameter in the db.properties file, please add it. 

pool.max.active=200

 

Generated at Tue Feb 18 07:52:52 UTC 2020 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.