Troubleshooting

Use the Xray sh script to check the status of the microservices and verify all of them are fully started and up, e.g.:

./xray.sh status all

Or for Docker:

./xray.sh ps

A working Xray system should have its RabbitMQ's queues' count be relatively levelled (close to 0) which would signify and provide the best performance and fastest results.

Large cloud of messages may happen be due to either:

1. One Xray service/component is not working fast enough to handle its queue

2. Some error/unexpected behavior happened on one of the components, searching for the logs according to the corresponding RabbitMQ queue will provide an insight

1. Temporarily increase the number of workers responsible for the appropriate queue (e.g. analysis) or allocate more resources to the machine; RAM & CPUs for faster processing, SSD for faster disk I/O.

2. Search the appropriate log file (e.g. xray-analysis.log) for any [EROR] or "Caused by" messages for the cause.

This may happen be due to either (or a combination of):

1. The $XRAY_DATA location is not configured correctly - thus preventing the Xray disk sampler from checking the correct path for disk space availability

2. The allowed disk utilization percentage value is too high

3. Temporary files which should be removed (under $XRAY_HOME/data/data/), are consuming a large amount space

4. * Mostly applicable for large scale: Other databases (RabbitMQ/PostgreSQL/MongoDB) are occupying a large amount of space

1. Enable debug logging on the Event component to see what is the path checked, expect to see a log entry stating: "Data Folder <location> , used: <%>, configured Threshold <%>. Remedy this by altering the  xrayDataFolder property under the xray_config.yaml file to a correct location.

2. Configure under the xray_config.yaml the maxDiskDataUsage flag to a lower value. On large scale enviroment and especially when working with large files, more space will be consumed (temporary) to allow the recursive indexing process to complete.

3. Xray version 1.9 significantly improved on handling scenarios where indexing has failed due an unexpected error (such as one related to files corruption) and will automatically remove these.

It is also possible to check the RabbitMQ's retry queues content (Get Message) to inspect messages and correlate their location under the xray_indexer.log* files. This will provide an insight for the retry action.

4. Configure the RabbitMQ database (mnesia) along with $XRAY_HOME (the two should reside under the same mount) to a large mount (typically 1-2TB)

/etc/rabbitmq/rabbitmq-env.conf (not created by default):

RABBITMQ_MNESIA_BASE=/home/rabbitmq/mnesia

Separate the Mongo and PostgreSQL (can and/or should be paired under the same mount but separate) from the RabbitMQ & $XRAY_HOME:

Postgres:

• To check: Postgresql home data folder location is configurable using the installer

If to be moved after the installation completion it is possible to follow this guide - Make sure PostgreSQL is stopped

Mongo:

• Stop Mongo (preferably stop all Xray services using the xray.sh stop all command)
• edit /etc/mongodb.conf, change:

  dbpath=/var/lib/mongo
  

  to a path of your choice.
  
  
• Copy your mongo data folder to a new location - make sure the permissions are set with the mongod user (the -a flag of the cp command should take care of keeping permissions) 
  

  cp -ra /var/lib/mongo  /home/myuser/data/mongo

If RabbitMQ is not started successfully (use the commands from the troubleshooting above), it could be that RabbitMQ ran into data corruption after an unexpected system state occurred and caused its messages' and/or queues' storage to loss integrity.

For each log file you want to analyze, set the log level to DEBUG. The log level for each Xray service is set in a corresponding configuration file found under /var/opt/jfrog/xray/data/config (for a Linux installation). For example, the analyzer service debug level is set in analysis_log4go_config.xml, while the indexer service debug level is set in indexer_log4go_config.xml. Similarly for all the other Xray services.

After performing an upgrade, Xray is performing a migration. When the migrations starts, it creates a lock mechanism, so that external operations on the database will be prevented, while performing the migration.

In case a migration has failed, the lock will remain, locking Xray, but it can be retried and can be successful on next run.

Quick summary of the steps we are about to take:

1. Check why and which the migration failed. This is optional and although not mandatory but is useful for future avoidance.

2. Delete the lock so that the migration will be retried.

Here is a break down of the steps that will need to be taken:

1. Stop all Xray services:

Non-Docker

$ ./xray.sh stop all

Docker:

$ ./xray.sh stop

 2. Start MongoDB and PostgreSQL only:

Non-Docker:

$ service postgresql-9.5 start

$ service mongod start

Docker:

$ docker start xray_mongodb_1

$ docker start xray_postgres_1

 3.  Perform actions on MongoDB side:

a. Connect to MongoDB:

Non-Docker:

$ mongo -u xray -p password --authenticationDatabase xray

Docker:

$  docker exec -it xray_mongodb_1 bash

Inside the Docker bash shell connect to MongoDB:

$ mongo -u xray -p password --authenticationDatabase xray

b. Switch to Xray DB:

$ use xray

c  Find the current state of DB migrations, including the failed ones (important for understanding the DB schemas migration versions):

 $ db.db_migrations_running.find({}).pretty()

e. Copy the output aside for investigation:

$ db.db_migrations.find({}).sort({"version":-1}).limit(1)

example for the output (do this for the 4. analysis step below):

{ "_id" : ObjectId("5a4bd9be95e701000100c37b"), "version" : NumberLong(27) }

 * The line above will tell us the ID of the migration running

f. Remove all migrations locks with the following command:

$ db.db_migrations_running.deleteMany({})

At this point - you may skip to step 5.

4. For analysis and when contacting JFrog Support, the following information needs to be collected:

Database state only information:

a. MongoDB, copy the the output of two commands (were provided in step 3 above):

a. $ db.db_migrations_running.find({}).pretty()

b. $ db.db_migrations.find({}).sort({"version":-1}).limit(1)

b. PostgreSQL, collect the information below from PostgreSQL database:

Connect to the PostgreSQL database:

Non-Docker:

$ psql xraydb xray   

Docker:

  $ docker exec -it  xray_postgres_1 /bin/bash

$ psql xraydb xray

* If/When prompted for password, enter xray

* xraydb is the name of the db, xray is the username

1. 1. Run the following query (note to use ‘;’ in the end, without it, it will not execute):

$ SELECT * FROM schema_migrations;

* Copy the output aside

b. In order to exit, enter:

  $ \q

c. Xray Services logs from the migration timeframe:

xray_server.log

xray_persist.log

xray_event.log

5. Restart all Xray services. This will trigger the migration to re-run.