Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-17930

A previously Derby - equipped node transitioned to a new DB and HA, will fail startup

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: 2 - Critical
    • Resolution: Unresolved
    • Affects Version/s: 6.4.0
    • Fix Version/s: None
    • Component/s: HA, startup
    • Labels:
      None

      Description

      If 'about-to-be-converted' HA node will already be attempted to be startup'ed once, the admin-access user check will fail due to the fact that it is not created in the database (and will not be created) since Artifactory will not bootstrap the creds.bootstrap file to Access. The log entry entry for this:

      2018-11-25 21:19:06,451 [localhost-startStop-1] [JFrog-Access] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:373) - [ACCESS BOOTSTRAP] Writing updated credentials to the bundling Artifactory: /Users/andreik/Downloads/2DiffHATest/artifactory-pro-6.5.0/etc/security/access/bootstrap.creds
      

      This happens only on the first startup.

      This scenario can happen to onboarding customers, or those who have accidentally startup'ed a standalone instance (before making it a node).

      Steps to reproduce:

      1. Startup an HA node with Derby
      2. Convert the node to HA (add ha-node.properties & db.properties if needed)
      3. Startup Artifactory and observe the below message (Could not find the new Access admin user. Sleeping for 2 seconds and retrying):

      2018-12-02 17:54:41,120 [art-init] [INFO ] (o.a.s.ArtifactorySchedulerFactoryBean:647) - Starting Quartz Scheduler now
      2018-12-02 17:54:41,233 [art-init] [INFO ] (o.a.s.ArtifactoryApplicationContext:243) - Artifactory context starting up 53 Spring Beans...
      2018-12-02 17:54:41,269 [art-init] [INFO ] (o.a.c.CentralConfigServiceImpl:280) - Loading bootstrap configuration (artifactory home dir is /Users/andreik/Downloads/2DiffHATest/artifactory-pro-6.5.0).
      2018-12-02 17:54:41,637 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:370) - Initialized new service id: jfrt@01cxqt2032xz941mg1cs7e010h
      2018-12-02 17:54:41,756 [art-init] [INFO ] (o.a.s.a.ArtifactoryAccessClientConfigStore:643) - Using Access Server URL: http://localhost:8040/access (bundled) source: detected
      2018-12-02 17:54:41,784 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:308) - Waiting for access server...
      2018-12-02 17:54:41,987 [localhost-startStop-1] [JFrog-Access] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:198) - Could not find the new Access admin user. Sleeping for 2 seconds and retrying
      2018-12-02 17:54:43,994 [localhost-startStop-1] [JFrog-Access] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:198) - Could not find the new Access admin user. Sleeping for 2 seconds and retrying
      

      This will persist for 90 seconds until Artifactory gives up. In the background there's a SQL query that does existence check for the access-admin.

      The same thing will not happen to a single instance - it will always try to generate a new admin user if needed.

      The suggestion is to make a similar resilient improvement for HA as well.

      =========================================================================================================

      Workaround:

      1. Bootstrap the admin-access user so it will be re-registered in the Database: https://www.jfrog.com/confluence/display/ACC/Configuring+Access#ConfiguringAccess-PreparingTheCredentialsFile
      and also delete the $ARTIFACTORY_HOME/etc/security/access/keys/access.creds file

      2. Delete the adminToken either from the config descriptor or the file system (above 6.4.0 it is on the file system under $ARTIFACTORY_HOME/etc/security/access/access.admin.token) and restart the cluster (each node holds an access HA "communication" token in it's memory after reach start).

      3. Restart the cluster to avoid propagation errors.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              andreik Andrei Komarov (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-17930 -
                  SYNCHRONIZED
                  • Last Sync Date: