The JFrog Metadata server is a JFrog Platform microservice that is used for storing, managing, and sharing metadata between different components of the JFrog Platform. The microservice is part of JFrog Artifactory and is bundled with JFrog Artifactory 7.x installations. The microservice is used to store and share metadata for all packages from any JFrog services, such as Artifactory and JFrog Xray, and has its own configuration settings and logs files.
When upgrading from one version of Artifactory to another, the metadata must also be migrated to the new version. The length of this process will vary depending on the number of packages being migrated - as well as the number of versions being migrated.
Note
This migration occurs automatically when upgrading from JFrog Artifactory 6.x to 7.x and requires no user action.
Fetching Metadata Using GraphQL
GraphQL is a query language that enables you to query your APIs and fetch specific data that you need. Using GraphQL to query metadata provides a simple way to fetch packages data stored in the metadata microservice. To learn more, see GraphQL.
Viewing Metadata in the JFrog Platform UI
The Metadata server stores metadata, collected or calculated, from all the different JFrog products and services.
The Metadata data-model is used to manage the package-versions metadata, which can be consumed using various filters and search options through the JFrog platform UI - "Packages View". To learn more, see Package Management.
Frequently Asked Questions on Metadata Migration
What happens to the UI during the migration?
When the migration takes place, the JFrog Platform UI displays a notification that the metadata is currently being migrated.
What happens if the migration stops in the middle?
The migration will continue from the same point the next time Artifactory is started.
Does the migration happen during or after the upgrade?
The migration occurs after the upgrade, and will be triggered on the first system initialization after the upgrade (the first time you start Artifactory v7.x).
Is there any downtime during migration?
JFrog tested the migration process using a benchmark of Artifactory of 27k packages and 1.2M versions:
Artifactory ran with the default migration configurations (5 threads migrating to MDS).
The Artifactory and a PostgreSQL database were run on different machines with the following spec - GCP VM with 8 CPU & 30 Gb RAM.
Is there any downtime during migration?
There is no downtime caused by the migration, however, search results in the UI and in the GraphQL API will be limited.
How long will the migration take and what is the rate of conversion?
Based on the benchmark above, the migration rate was 60k-67k versions per hour. The total migration using these specifications took between 18-22 hours.
Database Details
Database Connections
The Metadata service uses a database connection pool that by default contains 100 connections. Running Artifactory with the default MDS migration settings (5 threads migrating metadata), no significant raise in the active connections to the database (about 10 live connections) were detected.In the worst case, the number of connections will not pass 100 connections.
CPU
The PostgreSQL CPU max value was 60%.
The MDS CPU max value was 23%
RAM
The PostgreSQL RAM consumption was about 5%
The MDS RAM consumption was about 10%
Storage
The database size increased from 95GB to 155GB (~60% increase).
Managing the Metadata Migration Process
Customers can control the metadata migration by limiting the number of threads in Artifactory (threads<=1 means that the migration is off):
shared.extraJavaOpts -Dartifactory.metadata.event.operator.threads=5
Customers can check the migration status by querying the database as follows.
SELECT encode(migration_info_blob::bytea, 'escape') FROM migration_status where identifier = 'metadata-service-migration';