Administering Dynamic Node Pools
Dynamic node pools spin build nodes up and down on-demand from a cloud or Kubernetes service. To create a dynamic node pool, you must configure and provide an administration integration.
A dynamic node is provisioned when a step is triggered, then either cached or destroyed after a preset amount of idle time if no new step is triggered during that interval. When used on a cloud service, this can help manage costs by not incurring charges to run idle nodes.
The following actions can only be performed by administrator users to configure Pipelines dynamic node pools for operation.
Adding a Dynamic Node Pool
To create a new dynamic node pool for on-demand build nodes, go to Administration | Pipelines | Node Pools, click Add Node Pool, and then select Dynamic.
Fields that are not required may be left blank. For example, you may not wish to set the fields to limit node count, disk usage, or timeout. When finished, click Save Node Pool.
| General Settings | |
Name | An alphanumeric friendly name for the node pool. |
Set as Default Node Pool | When checked, sets the node pool as the default node pool for all pipelines. All steps that do not explicitly define which node pool to use will run on nodes of the default node pool. |
| Select an Integration | |
Integration | Select from the available Dynamic Node Integrations. |
| Additional Settings | |
Machine Type | Select from the available machine (instance) types for the service. For example: c4.large on AWS. |
Node Limits | The maximum number of nodes that can created on-demand for this node pool. |
Timeout | Number of seconds for connection timeout |
Node Idle Interval | Number of minutes a node can be idle before it is destroyed. |
Insecure Docker Registries | A comma-separated list of allowed insecure docker registries. |
| Kubernetes Settings (see Creating Dynamic Nodes on Kubernetes for configuration procedures) | |
Storage Class | The storage class available from the service to use. |
CPU | The CPU limit resource settingfor every pod. |
Labels | Optional labels for node affinity. |
Memory | The memory limit resource settingfor every pod. |
Namespace | The namespace for nodes. |
| AWS Settings (see AWS Keys Integration for configuration requirements) | |
SSH Key Pair | The SSH encrypted access key for the cloud service. Name of the SSH Key pair created on AWS. |
Security Group ID | An ID for an optional security group for all created nodes. |
Subnet Id | Optional VPC subnet ID |
| GCP Settings (see Google Cloud Integration for configuration requirements) | |
Zone | Zonewithin the hosting GCP region |
Network Tags | Optional Network tags (text attributes) for the VM instances |
Subnet Name | Optional subnet name |
Assign Public IP | When selected, nodes will be created with public IP addresses |
| Azure Settings (see Azure Keys Integration for configuration requirements) | |
Subscription ID | GUID that uniquely identifies your subscription to use Azure services |
Subnet Name | Vnet subnet name |
Vnet Name | Azure Virtual Network name (e.g., myVirtualNetwork) |
Resource Group Name | Name of the resource group (e.g., myResourceGroup) |
Security Group Name | Optional NIC network security group |
Public SSH Key | Public Secure Shell (SSH) key for Linux VMs |
Admin Username | Username for RDP (Windows nodes only, optional) |
Admin Password | Password for RDP (Windows nodes only, optional) |
| Cache Settings | |
Enable Cache | When checked, enables node caching. |
Reset Cache | Select the type of interval to reset the cache: On Day of Week or By Number of Days. Available only when node caching is enabled. |
| Assign Pipelines to this Node Pool | |
Any Pipeline Source | When selected, pipelines loaded from any pipeline source can use this node pool. Unselect this option to restrict node pool use to a set of pipeline sources. |
Node Duration
When a step is triggered, a dynamic node is spun up to execute the step. After the step is complete, the node is available to execute subsequent steps until it is terminated.
When node caching is not enabled, Pipelines will check the node at each node idle interval after the node was first spun up. If the node is idle Pipelines terminates the node. This ensures that your build infrastructure is managed in an efficient manner, to help you control computing costs.
Dynamic Node Caching
When enabled, node caching helps speed up execution of your builds by avoiding having to repeat time-consuming processes, such as installing or loading large amounts of dependency data. This feature is available only on dynamic node pools.
When node caching is enabled for a dynamic node pool, Pipelines pauses your node after executing a step, instead of terminating it after the specified idle time. When the next step executes, the paused node is restarted, retaining any Docker images or dependencies that were installed as part of the previous step. This greatly speeds up builds that pull or build Docker images as part of their workflow.
In addition, the node is available to execute the job much sooner, since restarting the node from a paused state is much faster than provisioning a new node. So if your build takes just a few minutes to execute, you save on the node provisioning time for every build, which adds up quickly.
Administering Static Node Pools
Sttatic node pools are collections of persistently available build nodes (virtual machines). Static nodes enable you to execute pipeline steps on your own infrastructure. For more information, see Managing Pipelines Static Nodes.
The following actions can only be performed by administrator users to configure Pipelines static node pools for operation.
Adding a Static Node Pool
To create a new node pool of static build nodes, go to Administration | Pipelines | Node Pools, click Add Node Pool, and then select Static.
Fields that are not required may be left blank. For example, you may not wish to set the fields to limit node count, disk usage, or timeout. When finished, click Save Node Pool.
| General Settings | |
Name | An alphanumeric friendly name for the node pool. |
Set as Default Node Pool | When checked, sets the node pool as the default node pool for all pipelines. All steps that do not explicitly define which node pool to use will run on nodes of the default node pool. |
| Additional Settings | |
Architecture | Currently limted to x86_64 |
OS | Select from the list of operating systems supported for the selected architecture. For a list of supported OS and architecture combinations, refer to Runtime Images. |
Runtime Version | The default runtime image that will be used to run pipeline steps. Choose a specific runtime to make sure the build nodes have the versions you need for languages/services/CLIs/etc. To learn more, see Managing Runtimes. |
Node Limits | Limits the maximum number of nodes that can be added to the node pool. |
Max Disk Usage % | Limits the maximum disk usage. A build node that exceeds this maximum will be marked as failed. |
Timeout | The maximum time (in minutes) after which you want your pipeline steps to timeout. This timeout is used when your step does not have any timeout specified. Default value for timeout is 60 minutes. |
Insecure Docker Registries | A comma-separated list of allowed insecure docker registries. |
| Assign Pipelines to this Node Pool | |
Any Pipeline Source | When selected, pipelines loaded from any pipeline source can use this node pool. Unselect this option to restrict node pool use to a set of pipeline sources. |
Once you have created a static node pool, you can add static nodes.
Administering All Node Pools
The following actions can only be performed by administrator users to configure Pipelines node pools (both static and dynamic) for operation.
Restricting Node Pool Use
By default, a node pool is available to all Pipelines users. However, you may restrict use of a node pool to pipelines that are loaded from a particular set of pipeline sources and the users that have been given permissions to use those pipeline sources.
You can establish those usage restrictions when you add or modify a node pool.
- Under Assign Pipeines to this Node Pool, uncheck the option for Any Pipeline Source.
- In the revealed interface, select from the list of available pipeline sources and move the row(s) (by click-and-drag or arrow button) to the list of included pipeline sources.
- When finished, click Save Node Pool.
Viewing Available Node Pools
To view the the list of node pools already added and available for use, go to Administration | Pipelines | Node Pools or Application | Pipelines | Node Pools.
Non-administrator users will only be shown node pools that they are permitted to use.
Viewing Node Pool Details
To view the details of a node pool, click the name of the node pool in the list of available node pools.
Modifying or Removing a Node Pool
To edit the node pool settings:
- View the node pool as shown above.
- Click Edit.
Settings that may not be changed have gray fields.
- When finished changing settings, click Save Node Pool.
To delete a node pool:
- View the node pool as shown above.
- Click Delete.
- Confirm you wish to delete by clicking Confirm.