Overview

Retention Policy enables you to configure a policy to discard pipelines run data from older pipelines runs. Pipelines run data includes Pipeline Run Logs, Pipeline StateSteps Cache, and Runs State. This data, over time and depending on your Pipelines usage, can become very large and lead to increased storage costs.

Retention Policy can be set at two levels:

  • System-level
  • Pipeline-level


Setting Retention Policy at System-level

To enable retention policy at the system-level, in the system.yaml file, set the retentionPolicy’s enabled flag to true. This policy applies to all your pipelines.

By default, Retention Policy is disabled at system-level, which effectively means infinite retention.

When enabled, system-level Retention Policy uses two properties to control which pipelines run data is to be retained:

  •   Max Age Days: Specifies the number of days, after which the pipelines run data will be deleted. This can be set in granularity of days. Default value is 0 (infinite retention).
  •   Minimum Runs: Specifies the minimum number of pipeline runs to keep, regardless of their age. Default value is 10.

Example

 retentionPolicy:
    enabled: true
    maxAgeDays: 90
    minRuns: 10

In this example, system-level Retention Policy is enabled, and pipelines run data for the last 90 days is set to be retained and data older than 90 days is set to be deleted. The minimum runs property (minRuns) ensures that for each one of your pipelines, the latest 10 pipeline runs data is retained, regardless of the policy’s max allowed age (maxAgeDays). This means, if some (or all) of your 10 latest pipelines runs are older than 90 days, Pipelines will still keep their run data. If you have less than 10 pipeline runs for certain pipelines, Pipelines will keep all their pipelines run data.


Setting Retention Policy at Pipeline-level

Once Retention Policy is enabled at system-level, it can also be configured at at pipeline-level. For information about setting the values for pipeline-level Retention Policy, see Defining a Pipeline.

Just like Retention Policy at system-level, you can control the values of the number of days and minimum number of runs to keep at pipeline-level. These settings override the system-level Retention Policy, but pipeline-level policy values can only be equal to or lesser than the system-level configuration.

For example, if the system-level Retention Policy configuration is:

 retentionPolicy:
    enabled: true
    maxAgeDays: 90
    minRuns: 10

And the pipeline-level Retention Policy configuration is:

 retentionPolicy:
    maxAgeDays: 30
    minRuns: 5

Then Pipelines will discard Pipeline run data for runs that are older than 30 days. Pipelines, however, will also keep at least 5 pipeline runs data, even if they are older than 30 days (or even if they are older than system-level 90 days value).

If you set the values of pipeline-level Retention Policy to greater than your system-level values, the Pipeline Source Sync will fail.

Example

pipelines:
 - name: example_pipeline_retention_policy
   configuration:
     retentionPolicy:
       maxAgeDays: 30
       minRuns: 5
   steps:
     - name: step_1
       type: Bash          
       execution:
         onExecute:
           - echo "Hello world"