Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Overview

Node Pools provide a convenient way to logically group build nodes in a JFrog Pipelines environment. This enables you to run steps simultaneously on different nodes.


Administrators may create any number of node pools. There must be at least one node pool created, designated as the default node pool, and assigned at least one build node for pipelines to execute in that Project.

A node pool must be designated to contain build nodes of one of these types:

  • Dynamic Nodes - Build nodes that are spun up and down on-demand by Pipelines from a cloud or Kubernetes service. This can help scale operations, and help manage costs by not incurring cloud service charges to run idle nodes.
  • Static Nodes – Build nodes (virtual machines) that are persistently available through a fixed IP address.
Page Contents




Administering Dynamic Node Pools

Dynamic node pools spin build nodes up and down on-demand from a cloud or Kubernetes service. To create a dynamic node pool, you must configure and provide an administration integration.

A dynamic node is provisioned when a step is triggered, then either cached or destroyed after a preset amount of idle time if no new step is triggered during that interval. When used on a cloud service, this can help manage costs by not incurring charges to run idle nodes.

The following actions can only be performed by administrator users to configure Pipelines dynamic node pools for operation.

Adding a Dynamic Node Pool

To create a new dynamic node pool for on-demand build nodes, go to Administration | Pipelines | Node Pools, click Add Node Pool, and then select Dynamic.

Fields that are not required may be left blank. For example, you may not wish to set the fields to limit node count, disk usage, or timeout. When finished, click Save Node Pool.

General Settings
Name
An alphanumeric friendly name for the node pool.
Set as Default Node Pool
When checked, sets the node pool as the default node pool for all pipelines in this Project. All steps that do not explicitly define which node pool to use will run on nodes of the default node pool.
Select an Integration/Select a Build Image
Integration
Select from the available Dynamic Node Integrations.
Build Image
Select from the available build images. For example, an Amazon Machine Image (AMI) on AWS.
Additional Settings
Machine Type
Select from the available machine (instance) types for the service. For example: c4.large on AWS.
Node Limits
The maximum number of nodes that can created on-demand for this node pool.
Disk Size

Optional. Configure dynamic node disk size.

The minimum limit is:

  • AWS (Windows):  120 GB
  • Azure (Windows): 128 GB
  • Others: 60 GB

  • This option is available for Pipelines on-premise versions only.
  •  If the disk size is not specified, the node will use the image’s disk size by default.

Timeout

Number of seconds for connection timeout. Default is 3600 seconds.

For more information, see Setting Step Timeouts.

Node Idle Interval
Number of minutes a node can be idle before it is destroyed.
Insecure Docker Registries
A comma-separated list of allowed insecure docker registries.

Kubernetes Settings (see Creating Dynamic Nodes on Kubernetes for configuration procedures)
Storage Class

The   storage class available from the service to use.

CPU

The CPU limit resource settingfor every pod.

Labels

Optional labels for node affinity.

Memory

The memory limit resource settingfor every pod.

Namespace

The  namespace for nodes.


AWS Settings (see AWS Keys Integration for configuration requirements)
SSH Key Pair

The SSH encrypted access key for the cloud service. Name of the SSH Key pair created on AWS.

Security Group ID
An ID for an optional security group for all created nodes.
Subnet Id
Optional VPC subnet ID

GCP Settings (see Google Cloud Integration for configuration requirements)
Zone

Zonewithin the hosting GCP region

Network Tags

Optional  Network tags (text attributes) for the VM instances

Subnet Name
Optional subnet name
Assign Public IP
When selected, nodes will be created with public IP addresses

Azure Settings (see Azure Keys Integration for configuration requirements)
Subscription ID
GUID that uniquely identifies your subscription to use Azure services
Subnet Name
Vnet subnet name
Vnet Name
Azure Virtual Network name (e.g., myVirtualNetwork)
Resource Group Name
Name of the resource group (e.g., myResourceGroup)
Security Group Name
Optional NIC network security group
Public SSH Key
Public Secure Shell (SSH) key for Linux VMs
Admin Username
Username for RDP (Windows nodes only, optional)
Admin Password
Password for RDP (Windows nodes only, optional)
Cache Settings
Enable Cache
When checked, enables node caching.
Reset Cache

Select the type of interval to reset the cache: On Day of Week or By Number of Days.

Available only when node caching is enabled.

Assign Pipelines to this Node Pool
Any Pipeline Source

When selected, pipelines loaded from any pipeline source can use this node pool.

Unselect this option to restrict node pool use to a set of pipeline sources.

Node Duration

When a step is triggered, a dynamic node is spun up to execute the step. After the step is complete, the node is available to execute subsequent steps until it is terminated.

When node caching is not enabled, Pipelines will check the node at each node idle interval after the node was first spun up. If the node is idle Pipelines terminates the node. This ensures that your build infrastructure is managed in an efficient manner, to help you control computing costs.

Dynamic Node Caching

When enabled, node caching helps speed up execution of your builds by avoiding having to repeat time-consuming processes, such as installing or loading large amounts of dependency data. This feature is available only on dynamic node pools.

When node caching is enabled for a dynamic node pool, Pipelines pauses your node after executing a step, instead of terminating it after the specified idle time. When the next step executes, the paused node is restarted, retaining any Docker images or dependencies that were installed as part of the previous step. This greatly speeds up builds that pull or build Docker images as part of their workflow.

In addition, the node is available to execute the job much sooner, since restarting the node from a paused state is much faster than provisioning a new node. So if your build takes just a few minutes to execute, you save on the node provisioning time for every build, which adds up quickly.


Administering Static Node Pools

Sttatic node pools are collections of persistently available build nodes (virtual machines). Static nodes enable you to execute pipeline steps on your own infrastructure. For more information, see Managing Pipelines Static Nodes.

The following actions can only be performed by administrator users to configure Pipelines static node pools for operation.

For information about troubleshooting node initialization issues, see Troubleshooting - Node Initialization


Adding a Static Node Pool

To create a new node pool of static build nodes, go to Administration | Pipelines | Node Pools, click Add Node Pool, and then select Static.

Fields that are not required may be left blank. For example, you may not wish to set the fields to limit node count, disk usage, or timeout. When finished, click Save Node Pool.

General Settings
Name
An alphanumeric friendly name for the node pool.
Set as Default Node Pool
When checked, sets the node pool as the default node pool for all pipelines. All steps that do not explicitly define which node pool to use will run on nodes of the default node pool.
Additional Settings
Architecture 

Currently limited to x86_64

OS 

Select from the list of operating systems supported for the selected architecture. For a list of supported OS and architecture combinations, refer to Runtime Images.

Runtime Version

The default runtime image that will be used to run pipeline steps. Choose a specific runtime to make sure the build nodes have the versions you need for languages/services/CLIs/etc. To learn more, see Managing Runtimes.

Node Limits
Limits the maximum number of nodes that can be added to the node pool.
Max Disk Usage %
Limits the maximum disk usage. A build node that exceeds this maximum will be marked as failed. 
Timeout 

The maximum time (in seconds) after which you want your pipeline steps to timeout. This timeout is used when your step does not have any timeout specified. Default value for timeout is 3600 seconds.

For more information, see Setting Step Timeouts.

Insecure Docker Registries
A comma-separated list of allowed insecure docker registries.
Assign Pipelines to this Node Pool
Any Pipeline Source

When selected, pipelines loaded from any pipeline source can use this node pool.

Unselect this option to restrict node pool use to a set of pipeline sources.


Once you have created a static node pool, you can add static nodes.


Administering All Node Pools

The following actions can only be performed by administrator users to configure Pipelines node pools (both static and dynamic) for operation.

Restricting Node Pool Use

If a node pool is in a Project, it may be assigned to one or more environments, and pipelines in pipeline sources that share at least one environment with the node pool may use the node pool.  By default, node pools are in the DEV environment.

Node pools that are not in a Project are available to pipelines in pipeline sources that are not in a Project. In addition, further restrictions may be specified to restrict use of particular pipeline sources and users that have been given permissions to use those pipeline sources.

You can establish those usage restrictions when you add or modify a node pool.

  1. Under Assign Pipeines to this Node Pool, uncheck the option for Any Pipeline Source.
  2. In the revealed interface, select from the list of available pipeline sources and move the row(s) (by click-and-drag or arrow button) to the list of included pipeline sources.


  3. When finished, click Save Node Pool.


Viewing Available Node Pools

To view the the list of node pools already added and available for use, go to Administration | Pipelines | Node Pools or Application | Pipelines | Node Pools.

Non-administrator users will only be shown node pools that they are permitted to use.


Max Limit Exceeded

If the number of nodes in a node pool exceeds its maximum limit, it is listed with an exclamation mark in the Max Limit column.

Node Provision Status

In the Node Pools list view, the Provision Status column provides a color representation of the provision status for each node. Every color represents one of the stages in the lifecycle of a node.

ColorProvision StatusDescription

Not provisionedThe node has not been requested to be provisioned.

WaitingWhen a node is provisioned, it first goes into waiting state. This means that the cloud provider is initializing the node and/or has not reported the status back to Pipelines.

Processing
  • The cloud provider reported the node's status back to Pipelines and Pipelines is setting up the agent.
    or 
  • The node is running a build.

SuccessThe node successfully initialized and is waiting for a Pipelines build.

Failure

The node failed. This is usually caused by an out of memory issue. However, there could be several other reasons, which are reflected in the node console.

StoppedWhen a node is deleted, its provisioning is first stopped and then it is deleted. 

Cached

The node is stopped and not running, but the actual instance is still available for instant use.

Caching must be enabled for a node to be cached.

Viewing Node Pool Details

To view the details of a node pool, click the name of the node pool in the list of available node pools.


Modifying or Removing a Node Pool

To edit the node pool settings:

  1. View the node pool as shown above. 
  2. Click Edit.
    Settings that may not be changed have gray fields.


  3. When finished changing settings, click Save Node Pool.


To delete a node pool:

  1. View the node pool as shown above. 
  2. Click Delete.
  3. Confirm you wish to delete by clicking Confirm.
  • No labels
Copyright © 2021 JFrog Ltd.