Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Overview

JFrog supports integration with Datadog, a SaaS-based data analytics platform, for log analytics and real-time observability. With Datadog you can now explore the log data, build visualizations and dashboards to gain valuable insights, such as system performance, storage consumption, and connection statistics associated with Artifactory and Xray, as well as, insights into the count and type of artifacts and components scanned by Xray. In addition, the Xray violations data is also available to provide you with additional insights in Datadog dashboards correlated with Artifactory logs.

JFrog Subscription Levels

SELF-HOSTED

PRO 
PRO X
ENTERPRISE
ENTERPRISE+

Page contents


Setting Up Fluentd for Datadog

To set up Fluentd for Datadog, perform the following steps:

  1. Install Fluentd using the jFrog documentation based on your installation type, and define the environment variable.
  2. Configure Fluentd with Artifactory by downloading the Artifactory Fluentd configuration file to a directory you have permissions to write in, such as $JF_PRODUCT_DATA_INTERNAL locations.

    cd $JF_PRODUCT_DATA_INTERNAL
    wget https://raw.githubusercontent.com/jfrog/log-analytics-datadog/master/fluent.conf.rt

    Override the match directive (lasts section) of the downloaded fluent.conf.rt file with the following:

    match jfrog.**>
     @type datadog
     @id datadog_agent_jfrog_artifactory
     api_key API_KEY
     include_tage_key true
     dd_source fluentd
    </match>

    Parameters:

    • API_KEY: Required. API key from Datadog.
    • dd_source: Name of the log integration in your logs in order to trigger the integration automatic setup in Datadog.
    • include_tag_key: Defaults to false and it will add fluentd tag in the JSON record if set to true
  3. Configure Fluentd with Xray by downloading the Xray Fluentd configuration file to a directory you have permissions to write, such as the $JF_PRODUCT_DATA_INTERNAL locations.

    cd $JF_PRODUCT_DATA_INTERNAL
    wget https://raw.githubusercontent.com/jfrog/log-analytics-datadog/master/fluent.conf.xray
    <source>
     @type jfrog_siem
     tage jfrog.xray.sien.vulnerabilities
     jpd_url JPD_URL
     username USER
     apikey JFROG_API_KEY
     pos_file "#{ENV['JF_PRODUCT_DATA_INTERNAM']}/log/jfrog.siem.log.pos"
    </source>

    Fill in the JPD_URL, USER, JFROG_API_KEY fields in the source directive of the downloaded fluent.conf.xray with the details given below:

    • JPD_URL: Required. The Artifactory JPD URL of the format http://<ip_address>, which is used to pull Xray Violations.
    • USER: Required. Artifactory username for authentication.
    • JFROG_API_KEY: Required. The Artifactory API Key for authentication.
       Override the match directive (last section) of the downloaded fluent.conf.xray with the details given below:

      <match jfrog.**>
       @type datadog
       @id datadog_agent_jfrog_xray
       api_key API_KEY
       include_tag_+key true
       dd_source fluentd
      </match>
  1. Enable the integration by running td-agent on Artifactory and Xray instances:

    td-agent

    The API key is configured in td-agent, which starts sending logs to Datadog. For other types of installations, see JFrog github project.


Metrics Collection

  1. Enable metrics for Artifactory and Xray by doing the following:
    1. Enable Open Metrics.
    2. Create Admin access tokens Generating Admin Tokens
  2. Configure Datadog by configuring this check for an agent running on a host.
    For containerized environments, see the Containerized section. These values override the configuration specified below.

    ARTIFACTORY_HOST_NAME_OR_IP -> IP address or DNS of Artifactory
    ARTIFACTORY_ADMIN_TOKEN     -> Admin token for Artifactory
    XRAY_ADMIN_TOKEN            -> Admin token for Xray


    1. Edit the openmetrics.d/conf.yaml file at the root of your Agent's configuration directory to start collecting your Artifactory and Xray Metrics. See the sample openmetrics.d/conf.yaml for all available configuration options.

      instances:
            -prometheus_url:http://<ARTIFACTORY_HOST_NAME_OR_IP>:80/artifactory/api/v1/metrics
             scheme: http
             headers:
               Authorization: "Bearer <ARTIFACTORY_ADMIN_TOKEN>"
             static_configs:
                -targets: ["<ARTIFACTORY_HOST_NAME_OR_IP>:80"]
             namespace: jfrog.articatory
             metrics:
                -sys*
                -jfrt*
                -app*
            -prometheus_url:http://<ARTIFACTORY_HOST_NAME_OR_IP>:80/xray/api/v1/metrics
             scheme: http
             headers:
               Authorization: "Bearer <XRAY_ADMIN_TOKEN>"
             static_configs:
                -targets: ["<ARTIFACTORY_HOST_NAME_OR_IP>:80"]
             namespace: jfrog.xray
             metrics:
                -app*
                -db*
                -go*
                -queue*
                -sys*
                -jfxr*
    2. Restart the Agent. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying the parameters specified above. To validate that the changes are applied, run the Agent's status subcommand and look for openmetrics under the Checks section.

Viewing JFrog Log Data in Datadog

Forward logs to your Datadog account from Fluentd using the Datadog Fluentd plugin. Import the dashboard from export.json that contains information displayed through JFrog’s data widgets to give you real time observability into JFrog Unified Platform. 




The JFrog Platform Logs dashboard presents timeline and count data for key operating metrics:

  • Log volumes, which can be filtered by type
  • Service errors
  • HTTP response codes
  • Accessed image
  • Accessed repositories
  • Data transfers in GB for uploads/downloads
  • Top 10 IPs for upload/download
  • Audit actions by username
  • Denied actions and logins by IP and username
  • Accepted deploys by username


For more detailed instructions on the Datadog integration refer to Jfrog Datadog Integration in Github.

  • No labels
Copyright © 2021 JFrog Ltd.