Have a question? Want to report an issue? Contact JFrog support

Skip to end of metadata
Go to start of metadata

Overview

This page provides tips to solve common problems that users have encountered.

Configuration Folder

In several cases, the solution requires accessing Xray internal data and configuration files. The location of these files can be obtained by running ./xray info.

For example:

./xray info
 
server port: 8000
docker mount folder: /home/vagrant/.jfrog/xray

Page contents

Xray is not indexing artifacts

 Xray is not receiving indexing events
CauseThere may be a problem with the RabbitMQ message broker
Resolution

Check the RabbitMQ event, index, or persist queues for messages. You can access the queues through the RabbitMQ console using:  http://localhost:15672/#/queues

If you are not able to acess the RabbitMQ UI, you can try to create an SSH tunnel using: ssh -L15672:127.0.0.1:15672 root@<machine ip>

 Disk usage has reached its limit
CauseXray has reached the configured limit for disk usage (the default is 80%) 
ResolutionIncrease the disk usage limit by increasing the value of maxDiskDataUsage in Xray's xray_config.yaml configuration file.

Xray is not able to communicate with Artifactory

 Artifactory is using a self-signed SSL certificate
CauseXray is not configured to work in insecure mode
Resolution

Enable insecure mode by setting the following parameter in Xray's xray_config.yaml configuration file.

sslInsecure: true

Xray does not not load the web UI or interact with the REST API

 Click here to expand...
CauseTypically, this could be caused due to the fact that one or more of Xray's microservices are not started
Resolution

Use the Xray sh script to check the status of the microservices and verify all of them are fully started and up, e.g.:

./xray.sh status all

Or for Docker:

./xray.sh ps

Xray's RabbitMQ's queue has a large number of messages

 Click here to expand...
Cause

A working Xray system should have its RabbitMQ's queues' count be relatively levelled (close to 0) which would signify and provide the best performance and fastest results.

Large could of messages may happen be due to either:

1. One Xray service/component is not working fast enough to handle its queue

2. Some error/unexpected behavior happened on one of the components, searching for the logs according to the corresponding RabbitMQ queue will provide an insight


Resolution

1. Temporarily increase the number of workers responsible for the appropriate queue (e.g. analysis) or allocate more resources to the machine; RAM & CPUs for faster processing, SSD for faster disk I/O.

2. Search the appropriate log file (e.g. xray-analysis.log) for any [EROR] or "Caused by" messages for the cause.

Xray is utilizing a large amount of space and potentially halts indexing

 Click here to expand...
Cause

This may happen be due to either (or a combination of):

1. The $XRAY_DATA location is not configured correctly - thus preventing the Xray disk sampler from checking the correct path for disk space availability

2. The allowed disk utilization percentage value is too high

3. Temporary files which should be removed (under $XRAY_HOME/data/data/), are consuming a large amount space

4. * Mostly applicable for large scale: Other databases (RabbitMQ/PostgreSQL/MongoDB) are occupying a large amount of space


Resolution

1. Enable debug logging on the Event component to see what is the path checked, expect to see a log entry stating: "Data Folder <location> , used: <%>, configured Threshold <%>. Remedy this by altering the  xrayDataFolder property under the xray_config.yaml file to a correct location.

2. Configure under the xray_config.yaml the maxDiskDataUsage flag to a lower value. On large scale enviroment and especially when working with large files, more space will be consumed (temporary) to allow the recursive indexing process to complete.

3. Xray version 1.9 significantly improved on handling scenarios where indexing has failed due an unexpected error (such as one related to files corruption) and will automatically remove these.

It is also possible to check the RabbitMQ's retry queues content (Get Message) to inspect messages and correlate their location under the xray_indexer.log* files. This will provide an insight for the retry action.

4. Configure the RabbitMQ database (mnesia) along with $XRAY_HOME (the two should reside under the same mount) to a large mount (typically 1-2TB)

/etc/rabbitmq/rabbitmq-env.conf (not created by default):

RABBITMQ_MNESIA_BASE=/home/rabbitmq/mnesia

Separate the Mongo and PostgreSQL (can and/or should be paired under the same mount but separate) from the RabbitMQ & $XRAY_HOME:

Postgres:

  • To check: Postgresql home data folder location is configurable using the installer

If to be moved after the installation completion it is possible to follow this guideMake sure PostgreSQL is stopped

Mongo:

  • Stop Mongo (preferably stop all Xray services using the xray.sh stop all command)
  • edit /etc/mongodb.conf, change:
    dbpath=/var/lib/mongo
    
    to a path of your choice.

  • Copy your mongo data folder to a new location - make sure the permissions are set with the mongod user (the -a flag of the cp command should take care of keeping permissions) 
    cp -ra /var/lib/mongo  /home/myuser/data/mongo

Xray does not startup after a system failure incident or disk space exhaustion

 Click here to expand...
Cause

If RabbitMQ is not started successfully (use the commands from the troubleshooting above), it could be that RabbitMQ ran into data corruption after an unexpected system state occurred and caused its messages' and/or queues' storage to loss integrity.

Resolution
  • Please note that this could result in queued messages loss - therefore some already made processing by Xray's microservices will be lost. No data related to already indexed files persisted to PostgreSQL (where the impact graph is stored and built) will be affected.
Check the following RabbitMQ files to diagnose the reason:

 /var/log/rabbitmq/startup_log

 /var/log/rabbitmq/startup_err

These log files are useful in particular when trying to diagnose RabbitMQ related issues.

  • Typically commands such as $ head or $ tail would help considerably in troubleshooting of these.
Below is a common error indicating a bad_match error (excerpted):

 

BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
       {{badmatch,
            {error,
                {{{badmatch,
                      {error,
                          {not_a_dets_file,
                              "/var/lib/rabbitmq/mnesia/rabbit@<id>/recovery.dets"}}},...

 

In this case the removal of RabbitMQ's queues' related files under the path below would help:

/var/lib/rabbitmq/mnesia/<username>@<id>/recovery.dets

Then try to restart Xray's microservices using:

$ ./xray.sh restart all


  • Sometimes, the restart of RabbitMQ alone could suffice (e.g. using $ service rabbitmq-server restart)

If this does not help, carry on to remove the queues message stores directors:


/var/lib/rabbitmq/mnesia/<username>@<id>/msg_store_transient
 
/var/lib/rabbitmq/mnesia/<username>@<id>/msg_store_persistent
 
/var/lib/rabbitmq/mnesia/<username>@<id>/queues
 

Again, restart Xray's microservices and access Xray using the web UI.

Xray is not able to send email notifications

 The email server configured for Xray uses a self-signed SSL certificate
CauseXray is not configured to work with an email server in insecure mode
Resolution

 Enable insecure mode by setting the following parameter in Xray's xray_config.yaml configuration file . .

sslInsecure: true

Xray is not providing enough information in its log files

 Click here to expand...
CauseBy default, Xray is configured for INFO level logging.
Resolution

For each log file you want to analyze, set the log level to DEBUG. The log level for each Xray service is set in a corresponding configuration file found under /var/opt/jfrog/xray/data/config (for a Linux installation). For example, the analyzer service debug level is set in analysis_log4go_config.xml, while the indexer service debug level is set in indexer_log4go_config.xml. Similarly for all the other Xray services.

Xray database migration failed and needs to be retried

 Click here to expand...
Cause

After performing an upgrade, Xray is performing a migration. When the migrations starts, it creates a lock mechanism, so that external operations on the database will be prevented, while performing the migration.

In case a migration has failed, the lock will remain, locking Xray, but it can be retried and can be successful on next run.

Resolution

Quick summary of the steps we are about to take:

1. Check why and which the migration failed. This is optional and although not mandatory but is useful for future avoidance.

2. Delete the lock so that the migration will be retried.


Here is a break down of the steps that will need to be taken:

1. Stop all Xray services:

Non-Docker

$ ./xray.sh stop all

Docker:

$ ./xray.sh stop

 

 2. Start MongoDB and PostgreSQL only:

Non-Docker:

$ service postgresql-9.5 start

$ service mongod start

Docker:

$ docker start xray_mongodb_1

$ docker start xray_postgres_1


 3.  Perform actions on MongoDB side:

a. Connect to MongoDB:

Non-Docker:

$ mongo -u xray -p password --authenticationDatabase xray

Docker:

$  docker exec -it xray_mongodb_1 bash

Inside the Docker bash shell connect to MongoDB:

$ mongo -u xray -p password --authenticationDatabase xray


b. Switch to Xray DB:

$ use xray

c  Find the current state of DB migrations, including the failed ones (important for understanding the DB schemas migration versions):

 $ db.db_migrations_running.find({}).pretty()

e. Copy the output aside for investigation:

$ db.db_migrations.find({}).sort({"version":-1}).limit(1)


example for the output (do this for the 4. analysis step below):

{ "_id" : ObjectId("5a4bd9be95e701000100c37b"), "version" : NumberLong(27) }

 * The line above will tell us the ID of the migration running


f. Remove all migrations locks with the following command:

$ db.db_migrations_running.deleteMany({})

 

At this point - you may skip to step 5.


4. For analysis and when contacting JFrog Support, the following information needs to be collected:

Database state only information:

a. MongoDB, copy the the output of two commands (were provided in step 3 above):

a. $ db.db_migrations_running.find({}).pretty()

b. $ db.db_migrations.find({}).sort({"version":-1}).limit(1)

 

b. PostgreSQL, collect the information below from PostgreSQL database:

Connect to the PostgreSQL database:

Non-Docker:

$ psql xraydb xray   

Docker:

  $ docker exec -it  xray_postgres_1 /bin/bash

$ psql xraydb xray


* If/When prompted for password, enter xray

* xraydb is the name of the db, xray is the username


    1. Run the following query (note to use ‘;’ in the end, without it, it will not execute):

$ SELECT * FROM schema_migrations;

* Copy the output aside

 

b. In order to exit, enter:

  $ \q


c. Xray Services logs from the migration timeframe:

xray_server.log

xray_persist.log

xray_event.log

 

5. Restart all Xray services. This will trigger the migration to re-run.

 

 

  • No labels