Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >





Overview

The JFrog Cloud Migrator tool enables you to move from a self-hosted JFrog Platform installation to JFrog Cloud. The migration tool uses JFrog CLI and is available as an official plugin for JFrog CLI. You can run the tool as-is or customize the environment using a set of advanced options to suit the network bandwidth and the size of your Artifactory filestore. 

To learn more about migration from a self-hosted JPD to JFrog Cloud, see Migrating from Self-Hosted JFrog Platform to JFrog Cloud. Ensure that you read Before You Begin before you attempt to run the migration tool. You must also follow the steps in Migration Process.

The process of running the migration tool includes the following steps:

  1. Review the prerequisites.
  2. Configure JFrog CLI.
  3. Add the migration tool plugin to JFrog CLI.
  4. Run the migration tool.


Page Contents


Prerequisites

Download and setup JFrog CLI on the machine that hosts the self-hosted JFrog platform. You must add JFrog CLI to the path. For more information, see JFrog CLI. Use JFrog CLI version 2.6.1 or later. We recommend that you use the latest version of JFrog CLI.

Ensure that you have the administrator credentials for both source and target JFrog Platform servers. You must configure JFrog CLI with these credentials.

Permissions

The migration tool must be run from the machine that hosts the self-hosted JFrog platform. The user running the tool must have sufficient permissions to access the JFrog home folder and JFrog CLI home folder. The user must have sudo permission to run the migration tool. You must provide the Artifactory user in the source JFrog Platform with write permissions on the <work-dir> folder.

Ensure that you have the administrator credentials for both source and target JFrog Platform servers. You must configure JFrog CLI with these credentials.

Space Requirements

The source Artifactory machine must have enough space to handle the temp cache directory during the migration process . You can get the minimum required space for temp cache directory with the following formula:

Size of the largest file in the source Artifactory * Number of threads in the migration tool

You can disable the check for space in the temp cache by setting the environment variable, JFROG_MIGRATION_TOOL_DISABLE_SIZE_CHECK, to true. If you set the value as true, ensure that you allocate enough space when there are huge artifacts to process.

export JFROG_MIGRATION_TOOL_DISABLE_SIZE_CHECK=true

The migration tool spawns threads equivalent to the value that you provide in the option, concurrent-files, when you run the migration tool with the migrate-artifacts command. For example, if the largest file in Artifactory is 300 MB and the number of threads in the migration tool is 8, the minimum space requirement for the temp cache directory is 2400 MB.


Migration Tool Options

You can run the JFrog Cloud migration tool with the following options.

CommandDescription
migrate-artifacts, ma 
Start the artifact migration.
migrate-config, mc
Configure the migration tool.
status, s
Provide the current status of migration.
help, h
Shows a list of commands or help for a command.
version, v
Displays the version of the migrator tool.



Step 1: Configure JFrog CLI

  1. Download and Install JFrog CLI. For more information, see JFrog CLI#Downloadandinstallation.

  2. Add the source JPD to JFrog CLI. 

    1. Run the following command.

      jf config add
    2. Enter a unique ID for the server. 
      You need to provide this ID as the input to the migration tool.

    3. Enter the JFrog Platform URL.

    4. Select Save and Continue and press Enter.

    5. Enter the JFrog access token for the server. 

      For more information on generating an access token, see Access Tokens.

    6. Enter the client certificate details if required.

      For more information on the command options, see JFrog CLI#AddingandEditingConfiguredServers.

  3. Add the target JFrog Cloud JPD to JFrog CLI.
    Repeat the steps that you did to add the source JFrog Platform server with JFrog Cloud details.

  4. Run the following command to show the configurations.

    jf config show
  5. Note down the server ID for both source and target systems so that you can use them in the migration tool commands.

Step 2: Add the Migration Tool Plugin to JFrog CLI

  1. Configure JFrog CLI with, releases.jfrog.io, the public JFrog repository for binaries.
    For more information on configuring the JFrog CLI, see Configure JFrog CLI. You need not enter any credentials when you configure the JFrog CLI since it is a public repository.

  2. Run the following command and note down the server ID for releases.jfrog.io.

    jf config show
  3. Run the following commands set up the installation of the plugin.

    export JFROG_CLI_PLUGINS_SERVER=<Server ID of releases.jfrog.io>
    export JFROG_CLI_PLUGINS_REPO=migration-tool
  4. Run the following command to install the migration tool to JFrog CLI as a plugin.

    jf plugin install migration-tool

    You can run the migration tool commands from JFrog CLI after the plugin installation is complete.


Step 3: Run the Migration Tool

  1. Configure JFrog CLI with both the source JPD and JFrog Cloud JPD. For more information, see Configure JFrog CLI.

  2. Run the migration tool with the migrate-config command to initiate the migration of configuration.The configuration command generates an export bundle, which is consumed by the migrate artifacts command.

    Run the migrate-config command only if you wish to migrate configuration settings from the self-hosted JPD. You can skip this command if you wish the migrate only the artifacts.

    jf migration-tool migrate-config <[Source-Id]> <[Target-Id]> <migrate-token>

    [Source-Id] is the server ID of the source self-hosted Artifactory, available when you complete JFrog CLI configuration for the source JFrog Platform server. [Target-Id] is the server ID of the target Artifactory in JFrog Cloud, available when you complete JFrog CLI configuration for the target JFrog Cloud platform. You can generate migrate-token from the MyJFrog portal for the target JFrog Cloud instance.


    For more information on the command options, see Configuration Command Options.

    After you run the migrate-config command, all contents in the target JFrog Cloud instance is wiped out. All the user credentials from the source self-hosted JFrog Platform will be available in the JFrog Cloud instance. You can use the source user credentials to log in to the JFrog Cloud instance.

    Do not proceed to the next step unless the command runs successfully.

  3. Configure JFrog CLI with the target JFrog Cloud JPD with the credentials in the source self-hosted JPD if you ran the migrate-config command.

  4. Optionally, run the migration tool with the set-replication command to configure replication between the source and target JPD so that the artifacts added to the source JPD after the start of the migration process are also migrated to the target JPD.

    jf migration-tool set-replication [command options]  <[Source-Id] [Target-Id]>
  5. Run the migration tool with the migrate-artifacts command to migrate the artifacts.

    jf migration-tool migrate-artifacts  <[Source-Id] [Target-Id]>

    [Source-Id] is service ID of the source Artifactory and [Target-Id] is the service ID of the target Artifactory. You configure the server ID when you complete the JFrog CLI configuration for the source JFrog Platform server.
    You can add command options to customize migration based on your network and environment. For more information on the command options, see  Migrate Artifacts Command Options

    When you run the migration tool, by default the tool requests you to accept the end user license agreement. You can set a command option, accept-eula to true, to implicitly accept the end user license agreement while running the tool. Ensure that you read the end user license agreement and accept the agreement before you enable the flag.

    After the command is complete, a report appears with the details of the migration. 
    The following sample shows an example.

    Repo Status :
      
        local-helm       :  Total - 4 | Success - 4 | Failed - 0 | Skipped - 0 | Unavailable - 0
        local-puppet     :  Total - 4 | Success - 4 | Failed - 0 | Skipped - 0 | Unavailable - 0
        local-sbt        :  Total - 1 | Success - 1 | Failed - 0 | Skipped - 0 | Unavailable - 0
        maven-private-fs :  Total - 1 | Success - 1 | Failed - 0 | Skipped - 0 | Unavailable - 0
      
      
    Total Artifact Count : Processed - 10 | Success - 10 | Failed - 0 | Skipped - 0 | Unavailable - 0
      
    Total Repo Count     : Processed - 4 | Previously Completed - 0 | Missing Repos - 0
  6. Run the status command to check the status of the migration.

    jf migration-tool status

    You can add command options to get more detailed status. For more information on the command options, see  Status Command Options .


Advanced Options in the Migration Tool

The migration tool provides several command options to fine tune the migration process.

Configuration Command Options

Usage
jf migration-tool migrate-config [command options] <[Source-Id]> <[Target-Id]> <migrate-token>

[Source-Id] is server ID of the source Artifactory. You configure the server ID when you complete JFrog CLI configuration for the source JFrog Platform server. [Target-Id] is the server ID of the target Artifactory in JFrog Cloud, available when you complete JFrog CLI configuration for the target JFrog Cloud platform. You can generate migrate-token from the MyJFrog portal for the target JFrog Cloud instance.

The following options are available with the migrate-config command.

OptionMandatory?Description
--migrate-token
NoToken that you generate in MyJFrog in the JFrog Cloud platform.
--work-dir
No

The directory used for fetching the configuration. By default, the tool uses the JFrog CLI home directory. 

If you specify an alternate directory, JFrog CLI must be accessible from this location. 

--insecure-tls
No

Use the tool in an environment with insecure TLS.

The default value is false.

--accept-eula


No

Automatically accept the end user license agreement.

The default value is false.

--force
No

Force run all steps of configuration migration regardless of whether any of the previous steps were completed.

The default is false.

Set Replication Command Options

Usage
jf migration-tool set-replication [command options] <[Source-Id] [Target-Id]>

[Source-Id] is server ID of the source Artifactory. You configure the server ID when you complete JFrog CLI configuration for the source JFrog Platform server. [Target-Id] is the server ID of the target Artifactory in JFrog Cloud, available when you complete JFrog CLI configuration for the target JFrog Cloud platform. 

The following options are available with the migrate-config command.

OptionDescription
--work-dir

The directory used for fetching the configuration. By default, the tool uses the JFrog CLI home directory. 

If you specify an alternate directory, JFrog CLI must be accessible from this location. 

--include-repos

List of repositories whose artifacts should be migrated.

By default, artifacts from all repositories are migrated.

--exclude-repos

List of repositories whose artifacts shouldn't be migrated

By default, no repositories are excluded.

--proxy
URL of the proxy server to be set for replication
--remove

Remove the current replication configuration set by the migration tool.

The default value is false.

--check-binary-exists

Enhance the checksum-based upload to check if the binary exists in the target binary store. Using this option impacts upload performance. It is useful in cases where the binaries were migrated in other means, before starting the artifacts migration using this tool.

The default is false.

--insecure-tls

Use the tool in an environment with insecure TLS.

The default value is false.

--accept-eula


Automatically accept the end user license agreement.

The default value is false.

Migrate Artifacts Command Options

Usage
jf migration-tool migrate-artifacts [command options] <[Source-Id] [Target-Id]>

[Source-Id] is service ID of the source Artifactory and [Target-Id] is the service ID of the target Artifactory. You configure the server ID when you complete the JFrog CLI configuration for the source JFrog Platform server.

You can stop running the migrate artifacts command to pause the migration. Migration resumes when you run the migrate artifacts command again.

The migrate artifact command options are optional. The tool does not persist any values you provide for the options when you enter the command. Therefore, if you want to use options other than the default values, you must provide them each time you run the tool.

The following options are available with the migrate-artifacts command.

OptionDescription
--work-dir

Work directory to be used for downloading the artifacts.

By default, the tool uses the JFrog CLI home directory. 

If you specify an alternate directory in the migrate-config command, you must enter the same along with the migrate artifacts command.

--include-repos

List of repositories whose artifacts should be migrated.

By default, artifacts from all repositories are migrated.

--exclude-repos

List of repositories whose artifacts shouldn't be migrated

By default, no repositories are excluded.

--concurrent-files

Number of files to handle concurrently. The tool spawns threads equivalent to the file that you provide in this option.

The default value is 8. The number of threads have an impact on the performance. You can provide the value depending on the size of data and the available system resources. For more information on performance with the respect to the value of the option, see Migration Tool Performance Enhancements.

--split-count

Number of segments to split a file when downloading in case it is larger than the value defined for the --min-split option.

The default value is 3.

--min-split

Size in kilobytes above which a downloaded file is split into a number of segments, based on the --split-count option.

The default value is 5120.

--retries

Number of retries to use for the data transfer (downloads and uploads)

The default value is 3.

--fail-fast

Fail migration on the first error.

The default value is false. The tool registers the error and continues with the operation.

--retry-failed

Process only the failed artifacts of the previous run.

The default value is false. The tool processes all the artifacts.

--insecure-tls

Use the tool in an environment with insecure TLS.

The default value is false.

--skip-statistics

Skip the migration of the artifact download statistics.

The default value is false. Set this value to true to improve the speed of migration in certain situations. For more information, see Migration Tool Performance Enhancements.

--overwrite

Overwrite existing files in the target environment.

The default value is false. 

Set the value to true to improve the speed of migration in certain situations. For more information, see Migration Tool Performance Enhancements.

--accept-eula

Automatically accept the end user license agreement.

The default value is false.

--paginated

Fetches the artifacts list from repositories as paginated. Use this option when you have a large number of artifacts in the repository.

The default value is false.

--include-items-modified-after

Include the artifacts that were modified after the given date-time.

You can specify the date-time in the ISO 8601 Standard for Date and Time Formats. The complete notation is YYYY-MM-DDThh:mm:ss.sTZD. You can also use partial precision, like just the year, year and month, and year, month and day. For more information, see Date and Time Format.

--check-binary-exists

Enhance the checksum-based upload to check if the binary exists in the target binary store. Using this option impacts upload performance. It is useful in cases where the binaries were migrated in other means, before starting the artifacts migration using this tool.

The default is false.

--force

Force run all steps of artifact migration regardless of whether any of the previous steps were completed.

The default is false.

You can run the migration tool with various options depending upon your requirements. For information on performance enhancements, see Migration Tool Performance Enhancements.


Status Command Options

Usage
jf migration-tool status [command options]

The following options are available with the status command.

OptionDescription
--detailed

Provide a detailed summary of the migration. 

The default value is false.

--work-dir

Work directory to be used for downloading the artifacts.

By default, the tool uses the JFrog CLI home directory. 

If you specify an alternate directory in the Configure command, you must enter the same along with the migrate artifacts command.

The command options are optional. If you enter the command without options, you get a concise summary of the migration.

  • No labels
Copyright © 2022 JFrog Ltd.