Unified Logging Layer - Fluentd
To enable JFrog to support numerous log vendors, an open source common logging platform Fluentd is used. It covers log input and field extraction for all products in the JFrog Platform and structures them as JSON. As part of the downstream data processing, the JSON output is forwarded leveraging the plugins to the desired log vendors such as Splunk, Datadog, EFK, and Prometheus. To learn more about Fluentd, see the Fluentd documentation.
Configuring Fluentd with the JFrog Platform
JFrog Platform users who want to use our log analytics will install td-agent for fluentd and configurations per Artifactory node. Fluentd supports HA configuration by shipping logs to a fluentd log aggregator which is configured in a HA setup to prevent the log aggregator service from downtime in the event of a crash of the aggregation node. Below is a link with more details on HA setup for Fluentd.
Viewing Log Data in JFrog Dashboards
The aggregated data from Fluentd is plugged into different log analytics tools to search, analyze and visualize the data. The tools will start populating the JFrog logs dashboard to present timeline and count data for key operating metrics.
Some of the metrics supported in the dashboard are
- Data Transfers (GBs) Over Time. Displays data transfers in GBs for uploads and downloads as a stacked column chart on a timeline (binned by hour or a convenient time slice) for the last 24 hours.
- Errors Over Time. Number of errors every 15 minutes, parsed from artifactory-service.log as a column chart on a timeline for the last 24 hours.
- Requests by Status Code (Every 10 Minutes). Displays requests by status code every 10 minutes in a stacked column chart on a timeline for the last 24 hours.
- Requests by Repo. Shows the number of requests by repo as a bar chart for the last 24 hours. Display top 10 repositories.
- Most Downloaded Artifacts. Shows the number of requests by artifact as a bar chart for the last 24 hours. Display top 10 downloaded artifacts.
- 5xx Status Codes. Displays 5xx status codes in a single value chart for the last 24 hours.
- Active Downloading IPs. Shows the top 10 active downloading IPs in GB in a bar chart for the last 24 hours.
- Active Uploading IPs. Provides the top 10 active uploading IPs in GB in a bar chart for the last 24 hours.
- Most Active Repos by Data Transfer. Displays the top 10 most active repos by data transfer in GB in a bar chart for the last 24 hours.
- Audit Admin Users. Displays all users in the last 24 hours who performed an action in the access-audit log, and the number of actions they took in a bar chart.
- Denied Login Attempts. Displays the number of denied login attempts in a statistics chart/table view, including details on IP address, user name, and number of attempts for the last 24 hours.
- Denied Actions by IP. Displays the number of denied actions by IP address in a bar chart, for the last six hours. Display top 10 denied IPs
- Denied Actions by Username. Displays the number of denied actions by username in a bar chart, for the last six hours. Display top 10 users
- Accepted Deploys by Username. Displays the number of accepted deploys by username in an statistics chart/table view, for the last six hours.