Using the latest JFrog products?
JFrog Platform User Guide
JFrog Xray 2.x Documentation
To get the latest version, go to the JFrog Unified Platform
- Created by Adi Atzmony, last modified by Unknown User (eliezerb) on Feb 10, 2020
Overview
This page provides tips to solve common problems that users have encountered.
Configuration Folder
In several cases, the solution requires accessing Xray internal data and configuration files. The location of these files can be obtained by running ./xray info
.
For example:
./xray info server port: 8000 docker mount folder: /home/vagrant/.jfrog/xray
Xray is not indexing artifacts
Cause | There may be a problem with the RabbitMQ message broker |
Resolution | Check the RabbitMQ event, index, or persist queues for messages. You can access the queues through the RabbitMQ console using: http://localhost:15672/#/queues If you are not able to acess the RabbitMQ UI, you can try to create an SSH tunnel using: ssh -L15672:127.0.0.1:15672 root@<machine ip> |
Cause | Xray has reached the configured limit for disk usage (the default is 80%) |
Resolution | Increase the disk usage limit by increasing the value of maxDiskDataUsage in Xray's xray_config.yaml configuration file. |
Xray is not able to communicate with Artifactory
Cause | Xray is not configured to work in insecure mode |
Resolution | Enable insecure mode by setting the following parameter in Xray's
|
Xray does not load the web UI or interact with the REST API
Cause | Typically, this could be caused due to the fact that one or more of Xray's microservices are not started |
Resolution | Use the Xray sh script to check the status of the microservices and verify all of them are fully started and up, e.g.: ./xray.sh status all Or for Docker: ./xray.sh ps |
Xray's RabbitMQ's queue has a large number of messages
Cause | A working Xray system should have its RabbitMQ's queues' count be relatively levelled (close to 0) which would signify and provide the best performance and fastest results. Large cloud of messages may happen be due to either: 1. One Xray service/component is not working fast enough to handle its queue 2. Some error/unexpected behavior happened on one of the components, searching for the logs according to the corresponding RabbitMQ queue will provide an insight |
Resolution | 1. Temporarily increase the number of workers responsible for the appropriate queue (e.g. analysis) or allocate more resources to the machine; RAM & CPUs for faster processing, SSD for faster disk I/O. 2. Search the appropriate log file (e.g. xray-analysis.log) for any [EROR] or "Caused by" messages for the cause. |
Xray is utilizing a large amount of space and potentially halts indexing
Cause | This may happen be due to either (or a combination of): 1. The $XRAY_DATA location is not configured correctly - thus preventing the Xray disk sampler from checking the correct path for disk space availability 2. The allowed disk utilization percentage value is too high 3. Temporary files which should be removed (under $XRAY_HOME/data/data/), are consuming a large amount space 4. * Mostly applicable for large scale: Other databases (RabbitMQ/PostgreSQL/MongoDB) are occupying a large amount of space |
Resolution | 1. Enable debug logging on the Event component to see what is the path checked, expect to see a log entry stating: "Data Folder <location> , used: <%>, configured Threshold <%>. Remedy this by altering the xrayDataFolder property under the xray_config.yaml file to a correct location. 2. Configure under the xray_config.yaml the maxDiskDataUsage flag to a lower value. On large scale enviroment and especially when working with large files, more space will be consumed (temporary) to allow the recursive indexing process to complete. 3. Xray version 1.9 significantly improved on handling scenarios where indexing has failed due an unexpected error (such as one related to files corruption) and will automatically remove these. It is also possible to check the RabbitMQ's retry queues content (Get Message) to inspect messages and correlate their location under the xray_indexer.log* files. This will provide an insight for the retry action. 4. Configure the RabbitMQ database (mnesia) along with $XRAY_HOME (the two should reside under the same mount) to a large mount (typically 1-2TB) /etc/rabbitmq/rabbitmq-env.conf (not created by default): RABBITMQ_MNESIA_BASE=/home/rabbitmq/mnesia Separate the Mongo and PostgreSQL (can and/or should be paired under the same mount but separate) from the RabbitMQ & $XRAY_HOME: Postgres:
If to be moved after the installation completion it is possible to follow this guide - Make sure PostgreSQL is stopped
|
Xray does not startup after a system failure incident or disk space exhaustion
Cause | If RabbitMQ is not started successfully (use the commands from the troubleshooting above), it could be that RabbitMQ ran into data corruption after an unexpected system state occurred and caused its messages' and/or queues' storage to loss integrity. |
Resolution |
Check the following RabbitMQ files to diagnose the reason: /var/log/rabbitmq/startup_log /var/log/rabbitmq/startup_err These log files are useful in particular when trying to diagnose RabbitMQ related issues.
Below is a common error indicating a bad_match error (excerpted): BOOT FAILED =========== Error description: {could_not_start,rabbit, {{badmatch, {error, {{{badmatch, {error, {not_a_dets_file, "/var/lib/rabbitmq/mnesia/rabbit@<id>/recovery.dets"}}},... In this case the removal of RabbitMQ's queues' related files under the path below would help: /var/lib/rabbitmq/mnesia/<username>@<id>/recovery.dets Then try to restart Xray's microservices using: $ ./xray.sh restart all
If this does not help, carry on to remove the queues message stores directors: /var/lib/rabbitmq/mnesia/<username>@<id>/msg_store_transient /var/lib/rabbitmq/mnesia/<username>@<id>/msg_store_persistent /var/lib/rabbitmq/mnesia/<username>@<id>/queues Again, restart Xray's microservices and access Xray using the web UI. |
Xray is not able to send email notifications
Cause | Xray is not configured to work with an email server in insecure mode |
Resolution | Enable insecure mode by setting the following parameter in Xray's
|
Xray is not providing enough information in its log files
Cause | By default, Xray is configured for INFO level logging. |
Resolution | For each log file you want to analyze, set the log level to DEBUG. The log level for each Xray service is set in a corresponding configuration file found under |
Xray database migration failed and needs to be retried
Cause | After performing an upgrade, Xray is performing a migration. When the migrations starts, it creates a lock mechanism, so that external operations on the database will be prevented, while performing the migration. In case a migration has failed, the lock will remain, locking Xray, but it can be retried and can be successful on next run. |
Resolution | Quick summary of the steps we are about to take: 1. Check why and which the migration failed. This is optional and although not mandatory but is useful for future avoidance. 2. Delete the lock so that the migration will be retried.
1. Stop all Xray services: Non-Docker $ ./xray.sh stop all Docker: $ ./xray.sh stop 2. Start MongoDB and PostgreSQL only: Non-Docker: $ service postgresql-9.5 start $ service mongod start Docker: $ docker start xray_mongodb_1 $ docker start xray_postgres_1 3. Perform actions on MongoDB side: a. Connect to MongoDB: Non-Docker: $ mongo -u xray -p password --authenticationDatabase xray Docker: $ docker exec -it xray_mongodb_1 bash Inside the Docker bash shell connect to MongoDB: $ mongo -u xray -p password --authenticationDatabase xray b. Switch to Xray DB: $ use xray c Find the current state of DB migrations, including the failed ones (important for understanding the DB schemas migration versions): $ db.db_migrations_running.find({}).pretty() e. Copy the output aside for investigation: $ db.db_migrations.find({}).sort({"version":-1}).limit(1) example for the output (do this for the 4. analysis step below): { "_id" : ObjectId("5a4bd9be95e701000100c37b"), "version" : NumberLong(27) } * The line above will tell us the ID of the migration running f. Remove all migrations locks with the following command: $ db.db_migrations_running.deleteMany({}) At this point - you may skip to step 5.
Database state only information: a. MongoDB, copy the the output of two commands (were provided in step 3 above): a. $ db.db_migrations_running.find({}).pretty() b. $ db.db_migrations.find({}).sort({"version":-1}).limit(1)
b. PostgreSQL, collect the information below from PostgreSQL database: Connect to the PostgreSQL database: Non-Docker: $ psql xraydb xray Docker: $ docker exec -it xray_postgres_1 /bin/bash $ psql xraydb xray * If/When prompted for password, enter xray * xraydb is the name of the db, xray is the username
$ SELECT * FROM schema_migrations; * Copy the output aside b. In order to exit, enter: $ \q c. Xray Services logs from the migration timeframe: xray_server.log xray_persist.log xray_event.log 5. Restart all Xray services. This will trigger the migration to re-run. |
- No labels
JFrog.com | Documentation | Featured | Have a question? Want to report an issue? Contact JFrog support