2019-10-15T05:27:14.379Z [jfac ] [INFO ] [29e9d618c8aa156b] [s.r.NodeRegistryServiceImpl:44] [http-nio-8040-exec-4] - join: request to "join" with serviceId jfrpg@01dq4d0f56sr4z79da0tkrf5pd and nodeId leela-arti-ha1 2019-10-15T05:27:14.480Z [jfac ] [INFO ] [29e9d618c8aa156b] [s.r.NodeRegistryServiceImpl:60] [http-nio-8040-exec-4] - join: success returning token with id 56e91a82-d197-41bd-b42c-abc697609dc6 nodeId leela-arti-ha1 serviceId jfrpg@01dq4d0f56sr4z79da0tkrf5pd 2019-10-15T05:27:15.037Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 20 Elapsed 9.55 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:15.341Z [jfrt ] [INFO ] [fc7fbaa97d86cfd8] [actorySchedulerFactoryBean:726] [art-init ] - Starting Quartz Scheduler now 2019-10-15T05:27:15.454Z [jfrt ] [INFO ] [fc7fbaa97d86cfd8] [ifactoryApplicationContext:266] [art-init ] - Artifactory context starting up 57 Spring Beans... 2019-10-15T05:27:15.873Z [jfrt ] [INFO ] [fc7fbaa97d86cfd8] [o.a.s.a.AccessServiceImpl:456 ] [art-init ] - Initialized new service id: jfrt@01dq4d1wc9e62g0188fk7b060s 2019-10-15T05:27:15.910Z [jfrt ] [INFO ] [fc7fbaa97d86cfd8] [oryAccessClientConfigStore:586] [art-init ] - Using Access Server URL: http://localhost:8046/access source: System Config 2019-10-15T05:27:16.227Z [jfrt ] [INFO ] [fc7fbaa97d86cfd8] [o.a.s.a.AccessServiceImpl:401 ] [art-init ] - Waiting for access server... 2019-10-15T05:27:20.049Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 30 Elapsed 14.56 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:25.060Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 40 Elapsed 19.57 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:30.075Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 50 Elapsed 24.59 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:35.089Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 60 Elapsed 29.6 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:40.102Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 70 Elapsed 34.62 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:45.113Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 80 Elapsed 39.63 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:50.124Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 90 Elapsed 44.64 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:27:55.136Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 100 Elapsed 49.65 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:00.145Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 110 Elapsed 54.66 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:05.155Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 120 Elapsed 59.67 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:10.166Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 130 Elapsed 1.08 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:15.178Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 140 Elapsed 1.16 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:20.188Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 150 Elapsed 1.25 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:25.200Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 160 Elapsed 1.33 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:30.211Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 170 Elapsed 1.41 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:35.221Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 180 Elapsed 1.5 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:40.232Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 190 Elapsed 1.58 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:45.244Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 200 Elapsed 1.66 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again panic: failed initializing Access client: ping failed to Access at: http://localhost:8046/access with error: operation timed out after 2m0s. last error: Get http://localhost:8046/access/api/v1/system/ping: dial tcp 127.0.0.1:8046: connect: connection refused goroutine 1 [running]: runtime/debug.Stack(0x10, 0xc00042a390, 0x422b88) /src/runtime/debug/stack.go:24 +0x9d jfrog.com/metadata/common/log.(*logger).Panicf(0xc0000b8550, 0x7f49a0bd2ab0, 0xc00058a300, 0x11889d0, 0x26, 0xc00042a390, 0x1, 0x1) /src/jfrog.com/metadata/common/log/logger.go:73 +0x6a jfrog.com/metadata/pkg/access.NewAccessClientWithRouter(0x7f49a0bd2ab0, 0xc00058a300, 0x7f49a0be0610, 0xc0000b8550, 0x13915a0, 0x1e8f2b0, 0xc0000b56c0, 0xe, 0xc00052c7e0, 0x1f, ...) /src/jfrog.com/metadata/pkg/access/accessclient.go:61 +0x5dc jfrog.com/metadata/application.InitAccessClientWithRouter(0x13c2dc0, 0xc00058a300, 0x13b04e0, 0xc0000aca90, 0x13915a0, 0x1e8f2b0, 0x13affa0, 0xc00000f640) /src/jfrog.com/metadata/application/access_client_init.go:22 +0x686 jfrog.com/metadata/application.CreateApp(0x13c2dc0, 0xc00058a300, 0x13a0420, 0xc0000bc400) /src/jfrog.com/metadata/application/application.go:102 +0xa4d main.main() /src/jfrog.com/metadata/metadata.go:22 +0x2b6 goroutine 1 [running]: github.com/rs/zerolog.(*Logger).Panic.func1(0xc0003f8a00, 0x4cf) /pkg/mod/github.com/rs/zerolog@v1.11.0/log.go:307 +0x4f github.com/rs/zerolog.(*Event).msg(0xc000018050, 0xc0003f8a00, 0x4cf) /pkg/mod/github.com/rs/zerolog@v1.11.0/event.go:146 +0x1c1 github.com/rs/zerolog.(*Event).Msgf(0xc000018050, 0xc0006661e0, 0x29, 0xc000529340, 0x2, 0x2) /pkg/mod/github.com/rs/zerolog@v1.11.0/event.go:121 +0x83 jfrog.com/metadata/common/log.logMessage(0x7f49a0bd2ab0, 0xc00058a300, 0xc000018050, 0xc0006661e0, 0x29, 0xc000529340, 0x2, 0x2) /src/jfrog.com/metadata/common/log/logger.go:82 +0x88 jfrog.com/metadata/common/log.(*logger).Panicf(0xc0000b8550, 0x7f49a0bd2ab0, 0xc00058a300, 0x11889d0, 0x26, 0xc000529340, 0x2, 0x2) /src/jfrog.com/metadata/common/log/logger.go:74 +0x1d2 jfrog.com/metadata/pkg/access.NewAccessClientWithRouter(0x7f49a0bd2ab0, 0xc00058a300, 0x7f49a0be0610, 0xc0000b8550, 0x13915a0, 0x1e8f2b0, 0xc0000b56c0, 0xe, 0xc00052c7e0, 0x1f, ...) /src/jfrog.com/metadata/pkg/access/accessclient.go:61 +0x5dc jfrog.com/metadata/application.InitAccessClientWithRouter(0x13c2dc0, 0xc00058a300, 0x13b04e0, 0xc0000aca90, 0x13915a0, 0x1e8f2b0, 0x13affa0, 0xc00000f640) /src/jfrog.com/metadata/application/access_client_init.go:22 +0x686 jfrog.com/metadata/application.CreateApp(0x13c2dc0, 0xc00058a300, 0x13a0420, 0xc0000bc400) /src/jfrog.com/metadata/application/application.go:102 +0xa4d main.main() /src/jfrog.com/metadata/metadata.go:22 +0x2b6 2019-10-15T05:28:46.409Z [jfrt ] [ERROR] [fc7fbaa97d86cfd8] [ctoryContextConfigListener:103] [art-init ] - Application could not be initialized: Waiting for access server to respond timed-out java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener.configure(ArtifactoryContextConfigListener.java:237) at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener$1.run(ArtifactoryContextConfigListener.java:99) Caused by: org.springframework.beans.factory.BeanInitializationException: Failed to initialize bean 'org.artifactory.security.access.AccessService'.; nested exception is java.lang.IllegalStateException: Waiting for access server to respond timed-out at org.artifactory.spring.ArtifactoryApplicationContext.refresh(ArtifactoryApplicationContext.java:276) at org.artifactory.spring.ArtifactoryApplicationContext.(ArtifactoryApplicationContext.java:154) ... 6 common frames omitted Caused by: java.lang.IllegalStateException: Waiting for access server to respond timed-out at org.artifactory.security.access.AccessServiceImpl.waitForAccessServer(AccessServiceImpl.java:416) at org.artifactory.security.access.AccessServiceImpl.waitForAccessServer(AccessServiceImpl.java:380) at org.artifactory.security.access.AccessServiceImpl.bootstrapAccessClient(AccessServiceImpl.java:1286) at org.artifactory.security.access.AccessServiceImpl.initAccessService(AccessServiceImpl.java:351) at org.artifactory.security.access.AccessServiceImpl.initAccessClientIfNeeded(AccessServiceImpl.java:340) at org.artifactory.security.access.AccessServiceImpl.init(AccessServiceImpl.java:333) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.artifactory.storage.fs.lock.aop.LockingAdvice.invoke(LockingAdvice.java:76) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy189.init(Unknown Source) at org.artifactory.spring.ArtifactoryApplicationContext.refresh(ArtifactoryApplicationContext.java:274) ... 7 common frames omitted Error: Couldn't join Access after 120 retries. at AccessClient.logAndCreateError (/opt/jfrog/artifactory/app/frontend/bin/server/node_modules/@jfrog/access-nodejs-client/dist/AccessClient.js:172:16) at AccessClient.join (/opt/jfrog/artifactory/app/frontend/bin/server/node_modules/@jfrog/access-nodejs-client/dist/AccessClient.js:58:20) 2019-10-15T05:28:50.254Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 210 Elapsed 1.75 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:55.264Z [jfac ] [WARN ] [ ] [o.j.c.ExecutionUtils:111 ] [pool-5-thread-2 ] - Retry 220 Elapsed 1.83 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again 2019-10-15T05:28:56.895Z [jfac ] [INFO ] [ ] [.AccessWithRouterRegistrar:106] [pool-5-thread-1 ] - Access Server successfully registered on Router on URL: http://localhost:8046 2019-10-15T05:28:57.103Z [jfac ] [INFO ] [ ] [.AccessServerBootstrapImpl:166] [pool-5-thread-1 ] - [ACCESS BOOTSTRAP] Updating server 'leela-arti-ha1' private key finger print to: 40b4b81d628fb496edd30d40963cef76fa15509c5661820348ba8f1013f58dfb ca private key finger print to: 020accfaef6c36f08b03f2d0eb91a47b04fb9409bd7f5c1307a53c06c0e9a3a4 2019-10-15T05:28:57.110Z [jfac ] [INFO ] [ ] [.AccessServerBootstrapImpl:144] [pool-5-thread-1 ] - Validated connections to all access servers successfully