Affects Version/s: 7.4.0
Fix Version/s: None
The customer noticed that when they upgraded Artifactory to version 7.X the connection to secure LDAP server is failing due to SSL exceptions. This was not an issue when they were in Artifactory version 6.X and they still have an Artifactory 6.X that is able to connect to the secure LDAPS server successfully.
The major difference between 6.X and 7.X is the java version. In 6.X JDK 8 is used and in 7.X JDK 11 is being used. As part of narrowing down the issue we switched Artifactory 6.X to use JDK 11 and saw that as soon as we switched to JDK 11 in 6.X the connection was failing with the same error again. At this point it seemed to be an issue with JDK, but we cannot be 100% sure since disabling TLSv1.2 and TLSv1.3 in JDK 11 allowed the connection to be successful to the secure LDAPS server. When using only TLSv1.1 we could see that the same Artifactory 7.X server is able to connect to the LDAPS server successfully.
We are not able to isolate the origin of the issue, as it can be either JDK 11 or there is a chance the issue could be with Artifactory. The issue is most probably not in the LDAPS server as the same server worked with JDK version 8 in 6.X Artifactory.
Artifactory version is 7.3.2
JDK version 11.0.2
As a final step we have tried upgrading JDK from 11.0.2 to 11.0.4 to see if it help in resolving this issue, but after upgrading the JDK also it did not help and we still see the same error.
Researching online we found some articles where users were running into similar issues with their applications after upgrading to JDK 11 and as a workaround they recommended disabling the SSL protocol TLSv1.3, which is supported in JDK 11. We tried this in the customer environment by passing the java options to disable the TLSv1.3 protocol in JDK 11, but we saw that the connection to LDAPS server is still failing with the same error.
extraJavaOpts: "-Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2"
We verified that the LDAPS server does support TLSv1.2 protocol, so not sure why the SSL connection was still failing with the same SSL remote host terminated exception. We know that the customer LDAPS server does not support TLSv1.3, so we already had the TLSv1.3 protocol disabled in JDK using the java options.
At this point, I searched for the SSL exceptions online and found the following Jenkins JIRA which described a very similar issue that Jenkins experienced after JDK was upgraded to version 11. Here is the Jenkins issue https://issues.jenkins-ci.org/browse/JENKINS-58603
As a workaround in Jenkins they had to disable both TLSv1.2 and TLSv1.3 in the JDK. We went ahead and tried this in the customer environment by disabling both the TLSv1.2 and TLSv1.3 and this allowed Artifactory to connect to secure LDAPS server. But we were using only TLSv1 and TLSv1.1 and since TLSv1.1 is very old and close to being deprecated, the customer did not want to be using only TLSv1.1 and they wanted to enable at least TLSv1.2.
In order to identify why the SSL handshake is being terminated when using TLSv1.2 we added the SSL debug logs parameter with the highest logging to the java options in Artifactory. After adding these options we tried connecting to the LDAPS server and saw the below exception, which again doesn’t clearly tell why the SSL handshake is being terminated. The ClientHello SSL handshake message is being generated, but it seems incomplete as the ClientHello handshake message should include the hostname and other details, which i noticed when i compared this ClientHello message to a successful SSL handshake transaction. Right after this the expected flow is to receive a ServerHello SSL handshake message, but we never see this as the SSl connection is terminated with an exception “javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake”. The full exception seen in the Artifactory "console.log" where the SSL debug logs are captured and I have posted them in a private comment due to security concern. Based on the debug logs it is not clear if the ClientHello SSL handshake message is ever reaching the LDAPS server. Also from the debug logs we could not determine if the SSL handshake is being terminated by the LDAPS server or if it is being terminated by Artifactory.
We have also verified that the CIPHER suite that the LDAPS server uses is part of the JDK cipher suites. This is the cipher suite the LDAPS server uses “Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”. You can see that this is present in the list of the cipher suites presented with the ClientHello SSL handshake message. When the issue occurs the LDAP team confirmed that they don’t see any connection initiated by Artifactory in their LDAP logs.
What is the expected behavior?
The expected behavior is for Artifactory to connect to secure LDAP server without any SSL exceptions. **
Steps to reproduce:**
There is no easy way to reproduce this issue as it is unique to certain environments when using it in combination with JDK 11. I did try to reproduce the issue, but I wasn't able to replicate the exact same behavior that we noticed in the customer environment. Here are the steps I have tried:
- Setup a LDAPS server. The easier and quickest way of doing it is by using the docker image available here https://github.com/osixia/docker-openldap
- use openssl command to list the self signed certs of the LDAPS server, so that you can import them to the JDK cacerts file. This step is important in order to avoid certificate related errors.
- Now add the LDAPS url to Artifactory and try test connection. We can see the SSL handshake termination error in the Artifactory logs. Make sure to enable SSL debug loggers to see the full SSL handshake flow.
- Unlike in the customer env, I see the issue in both Artifactory 6.X and 7.X versions with both JDK 11 and JDK 8 irrespective of the type of TLS protocols enabled. This is why I think the behavior for us is quite different when trying to reproduce the issue and is very unique to a specific environment.
I have also tried connecting to a public LDAPS server which i found online and in this case the connection was successful in both 7.X and 6.X Artifactory with JDK 11 when using TLSv1.2 protocol. In this case the public LDAPS server had valid SSL certificates signed by a CA, so we did not even have to import the SSL certs to JDK.
The workaround is to disable TLSv1.3 and TLSv1.2 using java options in the JVM and use only TLSv1 and TLSv1.1 protocols. However this is not an acceptable workaround as TLSv1.1 is quite old and might have security issues. Also there are plans to deprecate TLSv1.1.
Was that reproduced on GCP machine/mill?
No, I tried the reproduction steps in my local Artifactory