Karpenter – Tasks

V1.15: Using Nitro Enclaves

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to configure Karpenter to provision nodes with AWS Nitro Enclaves enabled.

Nitro Enclaves provide an isolated compute environment to protect and process highly sensitive data such as personally identifiable information (PII), healthcare, financial, and intellectual property data.

What are Nitro Enclaves?

AWS Nitro Enclaves are isolated compute environments built on the AWS Nitro System that provide:

Isolated compute environment: Enclaves run in a separate memory space from the parent instance with no persistent storage, interactive access, or external networking
Cryptographic attestation: You can verify the enclave’s identity and integrity before trusting it with sensitive data
Reduced attack surface: No SSH access, no persistent storage, and limited communication to only the parent instance
CPU and memory isolation: Dedicated vCPUs and memory allocated from the parent instance

Common use cases include:

Processing sensitive financial data and transactions
Running confidential computing workloads
Implementing secure key management and cryptographic operations
Supporting compliance efforts for workloads with data-isolation requirements, such as PCI DSS or HIPAA-regulated workloads

For more information, see the AWS Nitro Enclaves documentation.

Karpenter enables Nitro Enclaves in EC2 launch templates, excludes instance types whose EC2 NitroEnclavesSupport value is not supported, and models NodeOverlay capacity. You must configure the node, install the Kubernetes device plugin, and keep the NodeOverlay capacity consistent with the resources advertised by the device plugin and kubelet.

Warning

Karpenter does not inspect your AMI, generate Nitro Enclaves bootstrap configuration, or subtract enclave CPU from the instance’s standard allocatable cpu. Hugepage resources declared in a NodeOverlay are subtracted from standard allocatable memory in Karpenter’s scheduling simulation. The actual allocation happens on the node; a NodeOverlay does not configure the node or modify the capacity reported by kubelet.

Warning

All pods and containers on the parent node can communicate with enclaves attached to that node. A taint controls scheduling; it is not a security boundary. Use a dedicated NodePool with a NoSchedule taint to reduce which workloads can run on these nodes, and add the matching toleration only to the device-plugin DaemonSet and enclave workloads.

Prerequisites

Before continuing:

Enable the alpha NodeOverlay feature gate.
Prepare an AMI or user data that configures Nitro Enclaves on the node, including the allocator and the 1 GiB hugepages used by this example.
Install the AWS Nitro Enclaves Kubernetes device plugin on nodes labeled aws-nitro-enclaves-k8s-dp: enabled. Configure the device-plugin DaemonSet as shown below.
Decide how many enclave slots and CPUs and how much memory each node will advertise.

Enable NodeOverlays

When installing Karpenter with Helm, enable NodeOverlays through the chart settings:

settings:
 featureGates:
 nodeOverlay: true

Enable Nitro Enclaves on the EC2NodeClass

Set spec.enclaveOptions.enabled to true:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: enclave-enabled
spec:
 enclaveOptions:
 enabled: true

 amiFamily: AL2023
 amiSelectorTerms:
 - alias: al2023@latest

 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

Switching enabled between true and false changes the generated launch template and participates in EC2NodeClass drift. When enabled is true, Karpenter also filters the EC2 instance-type candidates to those whose NitroEnclavesSupport value is supported.

The example uses an EKS-optimized AMI only to keep the EC2NodeClass concise. You must still provide the node-level Nitro Enclaves configuration, either by baking it into an AMI or supplying appropriate user data. It uses @latest for brevity; follow the AMI pinning guidance for production.

Omitting enclaveOptions or setting enabled to false disables Nitro Enclaves. When enclaveOptions is specified, enabled is required.

Warning

Nitro Enclaves are not supported in AWS Local Zones, AWS Wavelength Zones, or AWS Outposts. Configure subnetSelectorTerms to resolve only subnets in standard Availability Zones. Karpenter filters instance types based on EC2’s NitroEnclavesSupport value, but does not filter these unsupported locations. EC2 can reject launch attempts that target them. If only unsupported locations match, the workload remains pending.

Label the NodePool

Use the default device-plugin label on the nodes that should run the plugin:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: enclave-pool
spec:
 template:
 metadata:
 labels:
 aws-nitro-enclaves-k8s-dp: enabled
 spec:
 taints:
 - key: aws-nitro-enclaves-k8s-dp
 value: enabled
 effect: NoSchedule
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: enclave-enabled

You do not need to maintain a static list of enclave-capable instance families. The EC2NodeClass compatibility filter removes unsupported instance types. You can add normal NodePool requirements when you need tighter cost, architecture, or size constraints.

Configure the Device Plugin

The upstream device-plugin DaemonSet selects nodes with the aws-nitro-enclaves-k8s-dp: enabled label. Enable enclave CPU advertisement and add a toleration for the NodePool taint:

kubectl patch daemonset aws-nitro-enclaves-k8s-daemonset \
 --namespace kube-system \
 --type strategic \
 --patch '{
 "spec": {
 "template": {
 "spec": {
 "tolerations": [{
 "key": "aws-nitro-enclaves-k8s-dp",
 "operator": "Equal",
 "value": "enabled",
 "effect": "NoSchedule"
 }],
 "containers": [{
 "name": "aws-nitro-enclaves-k8s-dp",
 "env": [{
 "name": "ENCLAVE_CPU_ADVERTISEMENT",
 "value": "true"
 }]
 }]
 }
 }
 }
 }'

Without the toleration, the DaemonSet cannot run on the tainted nodes. Without CPU advertisement, the device plugin does not publish the aws.ec2.nitro/nitro_enclaves_cpus resource used by the example.

Model Enclave Resources with a NodeOverlay

Create a NodeOverlay that matches the device-plugin label. Include numeric CPU and memory requirements so the overlay is only considered for instances larger than the enclave reservation.

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
 name: nitro-enclaves
spec:
 requirements:
 - key: aws-nitro-enclaves-k8s-dp
 operator: In
 values: ["enabled"]
 - key: karpenter.k8s.aws/instance-cpu
 operator: Gt
 values: ["2"]
 - key: karpenter.k8s.aws/instance-memory
 operator: Gt
 values: ["4096"]
 capacity:
 aws.ec2.nitro/nitro_enclaves: "4"
 aws.ec2.nitro/nitro_enclaves_cpus: "2"
 hugepages-1Gi: "4Gi"

The instance-memory requirement is expressed in MiB. Adjust the capacity values to match the device plugin, allocator, and hugepage configuration on your nodes.

The overlay must describe what the device plugin and kubelet will actually advertise:

aws.ec2.nitro/nitro_enclaves represents the number of enclaves
aws.ec2.nitro/nitro_enclaves_cpus represents enclave vCPUs
hugepages-1Gi represents memory reserved as 1 GiB hugepages

Schedule an Enclave Workload

Request the same resources from the workload:

apiVersion: v1
kind: Pod
metadata:
 name: enclave-workload
spec:
 tolerations:
 - key: aws-nitro-enclaves-k8s-dp
 operator: Equal
 value: enabled
 effect: NoSchedule
 containers:
 - name: workload
 image: public.ecr.aws/your-repository/your-image:latest
 volumeMounts:
 - name: hugepages
 mountPath: /dev/hugepages
 resources:
 requests:
 cpu: 250m
 aws.ec2.nitro/nitro_enclaves: "1"
 aws.ec2.nitro/nitro_enclaves_cpus: "2"
 hugepages-1Gi: "4Gi"
 limits:
 aws.ec2.nitro/nitro_enclaves: "1"
 aws.ec2.nitro/nitro_enclaves_cpus: "2"
 hugepages-1Gi: "4Gi"
 volumes:
 - name: hugepages
 emptyDir:
 medium: HugePages-1Gi

Replace the placeholder image with an enclave application that starts an enclave. The workload does not need to select a NodePool or instance family. Karpenter matches these resource requests against the capacity declared by the NodeOverlay.

Validate the Configuration

Confirm that the NodeOverlay is ready and passed validation:

kubectl get nodeoverlay nitro-enclaves \
 -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

Verify that the output includes Ready=True and ValidationSucceeded=True.

Confirm that EC2 launched the instance with enclaves enabled:

INSTANCE_ID="i-0123456789abcdef0"
aws ec2 describe-instances \
 --instance-ids "${INSTANCE_ID}" \
 --query 'Reservations[].Instances[].EnclaveOptions'

Verify that Enabled is true.

On the node, use the Nitro CLI to inspect running enclaves:

nitro-cli describe-enclaves

Confirm that the real node capacity matches the overlay:

NODE_NAME="ip-10-0-1-2.us-west-2.compute.internal"
kubectl get node "${NODE_NAME}" -o json | jq '.status.capacity | {
 enclaves: ."aws.ec2.nitro/nitro_enclaves",
 enclaveCPUs: ."aws.ec2.nitro/nitro_enclaves_cpus",
 hugepages: ."hugepages-1Gi"
}'

If these values are absent or differ from the NodeOverlay, fix the node image, allocator, hugepage configuration, or device-plugin deployment. Changing only the overlay can make Karpenter provision a node for a pod that the Kubernetes scheduler cannot place.

Confirm that the enclave workload schedules successfully:

kubectl get pod enclave-workload -o wide

Capacity Considerations

The enclave allocator takes CPU and memory from the parent instance. The hugepages-1Gi capacity in this example reduces the allocatable memory Karpenter computes, but aws.ec2.nitro/nitro_enclaves_cpus does not reduce the allocatable CPU Karpenter computes.

After launch, compare standard cpu in the NodeClaim’s .status.allocatable with the Node’s .status.allocatable. Keep ordinary workloads off the enclave NodePool. The sum of standard cpu requests for every pod allowed onto the NodePool must fit the Node’s actual allocatable CPU. Separately, the sum of aws.ec2.nitro/nitro_enclaves_cpus requests must fit the capacity declared by the NodeOverlay. Requesting enclave CPUs does not reserve standard CPU, and a NoSchedule taint does not change Karpenter’s CPU calculation.

Additional Resources

Follow-up

If you have questions or issues with Nitro Enclaves in Karpenter, feel free to:

Open an issue on GitHub
Ask in the Karpenter Slack channel
Check the Troubleshooting Guide

V1.15: Managing AMIs

Mon, 01 Jan 0001 00:00:00 +0000

Important

Karpenter heavily recommends against opting-in to use an amiSelectorTerm with @latest unless you are doing this in a pre-production environment or are willing to accept the risk that a faulty AMI may cause downtime in your production clusters. In general, if using a publicly released version of a well-known AMI type (like AL2, AL2023, or Bottlerocket), we recommend that you pin to a version of that AMI and deploy newer versions of that AMI type in a staged approach when newer patch versions are available.

amiSelectorTerms:
 - alias: al2023@v20240807

More details are described in Controlling AMI Replacement below.

Understanding how Karpenter assigns AMIs to nodes can help ensure that your workloads will run successfully on those nodes and continue to run if the nodes are upgraded to newer AMIs. Below we describe how Karpenter assigns AMIs to nodes when they are first deployed and how newer AMIs are assigned later when nodes are spun up to replace old ones. Later, it describes the options you have to assert control over how AMIs are used by Karpenter for your clusters.

Features for managing AMIs described here should be considered as part of the larger upgrade policies that you have for your clusters. See How do I upgrade an EKS Cluster with Karpenter for details on this process.

How Karpenter assigns AMIs to nodes

Here is how Karpenter assigns AMIs nodes:

When you create an EC2NodeClass, you are required to specify amiSelectorTerms. amiSelectorTerms allow you to select on AMIs that can be spun-up by this EC2NodeClass based on tags, id, name, or an alias. Multiple AMIs may be specified, and Karpenter will choose the newest compatible AMI when spinning up new nodes.
Some amiSelectorTerm types are static and always resolve to the same AMI (e.g. id). However, some are dynamic and may resolve to different AMIs over time. Examples of dynamic types include alias, tags, and name (when using a wildcard). For example, if you specify an amiSelectorTerm with an alias set to @latest (e.g. al2023@latest, al2@latest, or bottlerocket@latest), Karpenter will use the latest release for that AMI type when spinning up a new node.
When a node is replaced, Karpenter checks to see if a newer AMI is available based on your amiSelectorTerms. If a newer AMI is available, Karpenter will automatically use the new AMI to spin up the new node. In particular, if you are using a dynamic amiSelectorTerm type, you may get a new AMI deployed to your environment without having properly tested it.

Whenever a node is replaced, the replacement node will be launched using the newest AMI based on your amiSelectorTerms. Nodes may be replaced due to manual deletion, or any of Karpenter’s automated methods:

Expiration: Automatically initiates replacement at a certain time after the node is created.
Consolidation: If Karpenter detects that a cheaper node can be used to run the same workloads, Karpenter may replace the current node automatically.
Drift: If a node’s state no longer matches the desired state dictated by the NodePool or EC2NodeClass, it will be replaced, including if the node’s AMI no longer matches the latest AMI selected by the amiSelectorTerms.
Interruption: Nodes are sometimes involuntarily disrupted by things like Spot interruption, health changes, and instance events, requiring new nodes to be deployed.

See Automated Methods for details on how Karpenter uses these automated actions to replace nodes.

The most relevant automated disruption method is Drift, since it is initiated when a new AMI is selected-on by your amiSelectorTerms. This could be due to a manual update (e.g. a new id term was added), or due to a new AMI being resolved by a dynamic term.

If you’re using an alias with the latest pin (e.g. al2023@latest), Karpenter periodically checks for new AMI releases. Since AMI releases are outside your control, this could result in new AMIs being deployed before they have been properly tested in a lower environment. This is why we strongly recommend using version pins in production environments when using an alias (e.g. al2023@v20240807).

Important

If you are new to Karpenter, you should know that the behavior described here is different than you get with Managed Node Groups (MNG). MNG will always use the assigned AMI when it creates a new node and will never automatically upgrade to a new AMI when a new node is required. See Updating a Managed Node Group to see how you would manually update MNG to use new AMIs.

Controlling AMI Replacement

Karpenter’s automated node replacement functionality in tandem with the EC2NodeClass gives you a lot of flexibility to control the desired state of nodes on your cluster. For example, you can opt-in to AMI auto-upgrades using alias set to @latest; however, this has to be weighed heavily against the risk of newer versions of an AMI breaking existing applications on your cluster. Alternatively, you can choose to pin your AMIs in your production clusters to avoid the risk of breaking changes; however, this has to be weighed against the management cost of testing new AMIs in pre-production and keeping up with the latest AMI versions.

Karpenter offers you various controls to ensure you don’t take on too much risk as you rollout new versions of AMIs to your production clusters. Below shows how you can use these controls:

Pinning AMIs: If workloads require a particluar AMI, this control ensures that it is the only AMI used by Karpenter. This can be used in combination with Testing AMIs where you lock down the AMI in production, but allow the newest AMIs in a test cluster while you test your workloads before upgrading production.
Testing AMIs: The safest way for ensuring that a new AMI doesn’t break your workloads is to test it before putting it into production. This takes the most effort on your part, but most effectively models how your workloads will run in production, allowing you to catch issues ahead of time. Note that you can sometimes get different results from your test environment when you roll a new AMI into production, since issues like scale and other factors can elevate problems you might not see in test. Combining this with other controls like Using Disruption Budgets can allow you to catch problems before they impact your whole cluster.
Using Disruption Budgets: This option can be used as a way of mitigating the scope of impact if a new AMI causes problems with your workloads. With Disruption budgets you can slow the pace of upgrades to nodes with new AMIs or make sure that upgrades only happen during selected dates and times (using schedule). This doesn’t prevent a bad AMI from being deployed, but it allows you to control when nodes are upgraded, and gives you more time to respond to rollout issues.

Pinning AMIs

When you configure the EC2NodeClass, you are required to configure which AMIs you want Karpenter to select on using the amiSelectorTerms field. When pinning to a specific id, name, tags or an alias that contains a fixed version, Karpenter will only select on a single AMI and won’t automatically upgrade your nodes to a new version of an AMI. This prevents a new and potentially untested AMI from replacing existing nodes when those nodes are terminated. ).

Note

Pinning an AMI to an alias type with a fixed version will pin the AMI so long as your K8s control plane version doesn’t change. Unlike id and name types, specifying a version alias in your amiSelectorTerms will cause Karpenter to consider the K8s control plane version of your cluster when choosing the AMI. If you upgrade your Kubernetes cluster while using this alias type, Karpenter will automatically drift your nodes to a new AMI that still matches the AMI version but also matches your new K8s control plane version.

These examples show three different ways to identify the same AMI:

# Using alias
# Pinning to this fixed version alias will pull this version of the AMI,
# matching the K8s control plane version of your cluster
amiSelectorTerms:
- alias: al2023@v20240219

# Using name
# This will only ever select the AMI that contains this exact name
amiSelectorTerms:
- name: al2023-ami-2023.3.20240219.0-kernel-6.1-x86_64

# Using id
# This will only ever select this specific AMI id
amiSelectorTerms:
- id: ami-052c9ea013e6e3567

# Using tags
# You can use a CI/CD system to test newer versions of an AMI
# and automatically tag them as you validate that they are safe to upgrade to
amiSelectorTerms:
- tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: prod

See the spec.amiSelectorTerms section of the NodeClasses page for details. Keep in mind, that this could prevent you from getting critical security patches when new AMIs are available, but it does give you control over exactly which AMI is running.

Testing AMIs

Instead of avoiding AMI upgrades, you can set up test clusters where you can try out new AMI releases before they are put into production. For example, you could have:

Test clusters: On lower environment clusters, you can run the latest AMIs e.g. al2023@latest, al2@latest, bottlerocket@latest, for your workloads in a safe environment. This ensures that you get the latest patches for AMIs where downtime to applications isn’t as critical and allows you to validate patches to AMIs before they are deployed to production.
Production clusters: After you’ve confirmed that the AMI works in your lower environments, you can pin the latest AMIs to be deployed in your production clusters to roll out the AMI. Refer to Pinning AMIs for how to choose a particular AMI by alias, name or id. Remember that it is still best practice to gradually roll new AMIs into your cluster, even if they have been tested. So consider implementing that for your production clusters as described in Using Disruption Budgets.

Using Disruption Budgets

To reduce the risk of entire workloads being immediately degraded when a new AMI is deployed, you can enable Karpenter’s Node Disruption Budgets as well as ensure that you have Pod Disruption Budgets configured for applications on your cluster. Below provides more details on how to configure each.

Node Disruption Budgets

Disruption Budgets limit when and to what extent nodes can be disrupted. You can prevent disruption based on nodes (a percentage or number of nodes that can be disrupted at a time) and schedule (excluding certain times from disrupting nodes). You can set Disruption Budgets in a NodePool spec. Here is an example:

disruption:
 budgets:
 - nodes: 15%
 - nodes: "3"
 - nodes: "0"
 schedule: "0 9 * * sat,sun"
 duration: 24h
 - nodes: "0"
 schedule: "0 17 * * mon-fri"
 duration: 16h
 reasons:
 - Drifted

Settings for budgets in the above example include the following:

Percentage of nodes: From the first nodes setting, only 15% of the NodePool’s nodes can be disrupted at a time.
Number of nodes: The second nodes setting limits the number of nodes that can be disrupted at a time to 3.
Schedule: The third nodes setting uses schedule to say that zero disruptions (0) are allowed starting at 9am on Saturday and Sunday and continues for 24 (fully blocking disruptions all day). The format of the schedule follows the crontab format for identifying dates and times. See the crontab page for information on the supported values for these fields.
Reasons: The fourth nodes setting uses reasons which implies that this budget only applies to the Drifted disruption condition. This setting uses schedule to say that zero disruptions (0) are allowed starting at 5pm on Monday, Tuesday, Wednesday, Thursday, and Friday and continues for 16h (effectively blocking rolling nodes due to drift outside of working hours).

As with all disruption settings, keep in mind that avoiding updated AMIs for your nodes can result in not getting fixes for known security risks and bugs. You need to balance that with your desire to not risk breaking the workloads on your cluster.

Pod Disruption Budgets

Pod Disruption Budgets allow you to describe how much disruption an application can tolerate before it begins to become unhealthy. This is critical to configure for Karpenter, since Karpenter uses this information to determine if it can continue to replace nodes. Specifically, if replacing a node would cause a Pod Disruption Budget to be breached (for graceful forms of disruption e.g. Drift or Consolidation), Karpenter will not replace the node.

In a scenario where a faulty AMI is rolling out and begins causing downtime to your applications, configuring Pod Disruption Budgets is critical since this will tell Karpenter that it must stop replacing nodes until your applications become healthy again. This prevents Karpenter from deploying the faulty AMI throughout your cluster, reduces the imact the AMI has on your production applications, and gives you manually intervene in the cluster to remediate the issue.

Follow-up

The Karpenter project continues to add features to give you greater control over AMI upgrades on your clusters. If you have opinions about features you would like to see to manage AMIs with Karpenter, feel free to enter a Karpenter New Issue.

V1.15: Monitoring Amazon EC2 API Usage

Mon, 01 Jan 0001 00:00:00 +0000

AWS throttling can impact Karpenter’s ability to manage your cluster. Karpenter calls the Amazon EC2 API to discover infrastructure and to launch and terminate nodes, and Amazon EC2 enforces per-account, per-Region request-rate limits. When your request rate exceeds a limit, Amazon EC2 rejects the excess requests with the RequestLimitExceeded error (HTTP 503).

The volume of these calls is not fixed. It scales with the number of EC2NodeClasses and clusters you run, how often your cluster scales up and down, and the Karpenter version you run, so a large enough fleet can generate enough requests to be throttled. Because this volume can scale, you should monitor it. This task describes the Amazon EC2 APIs Karpenter calls, how to observe your call volume and throttling, how to compare that volume against your account’s request-rate limits, and other best practices.

What Amazon EC2 APIs Karpenter calls

API	Category	When Karpenter calls it
`CreateFleet`	Launch (hot path)	Launching nodes to satisfy pending pods
`CreateLaunchTemplate`	Launch (hot path)	Preparing launch configuration for new nodes
`RunInstances`	Launch (hot path)	Launching nodes
`CreateTags`	Launch (hot path)	Tagging instances, fleets, and launch templates as they are created
`TerminateInstances`	Terminate	Removing nodes during consolidation, drift, or expiration
`RebootInstances`	Reboot	Rebooting unhealthy nodes in place during Node Auto Repair with the `RebootForRepair` AWS feature gate
`DeleteLaunchTemplate`	Cleanup	Removing launch templates Karpenter manages
`DescribeSubnets`	Discovery / refresh	Resolving `subnetSelectorTerms` for each `EC2NodeClass`
`DescribeSecurityGroups`	Discovery / refresh	Resolving `securityGroupSelectorTerms` for each `EC2NodeClass`
`DescribeImages`	Discovery / refresh	Resolving `amiSelectorTerms` for each `EC2NodeClass`
`DescribeCapacityReservations`	Discovery / refresh	Resolving `capacityReservationSelectorTerms` (On-Demand Capacity Reservations)
`DescribeInstanceTypes`, `DescribeInstanceTypeOfferings`	Discovery / refresh	Resolving available instance types and their offerings
`DescribeSpotPriceHistory`	Discovery / refresh	Determining spot pricing for instance type selection
`DescribePlacementGroups`	Discovery	Resolving placement groups referenced by an `EC2NodeClass`
`DescribeInstances`, `DescribeInstanceStatus`	Discovery	Reconciling the state of launched instances
`DescribeLaunchTemplates`	Discovery	Reconciling the launch templates Karpenter manages

How to observe call volume and throttling

Karpenter exposes Prometheus metrics (by default at :8080/metrics, configurable via METRICS_PORT; see the Metrics reference). The AWS SDK request metrics are the most direct measure of Karpenter’s Amazon EC2 call volume and throttling. They are labeled by service (for example, EC2), action (the API operation, for example DescribeSubnets or CreateFleet), and code (the HTTP status code — 200 for success, and 503 for the RequestLimitExceeded throttling response):

aws_sdk_go_request_total — total AWS SDK requests, by service, action, and code.
aws_sdk_go_request_attempt_total — total request attempts (a single request may make multiple attempts when retried).
aws_sdk_go_request_retry_count — number of retry attempts per request. Sustained retries are an early indicator of throttling, because the AWS SDK retries throttled requests before they surface as an error.

For example, to graph Karpenter’s Amazon EC2 request rate by operation:

sum by (action) (rate(aws_sdk_go_request_total{service="EC2"}[5m]))

To graph the throttled fraction of Karpenter’s Amazon EC2 requests:

sum(rate(aws_sdk_go_request_total{service="EC2", code="503"}[5m]))
/ sum(rate(aws_sdk_go_request_total{service="EC2"}[5m]))

How to compare against your account’s Amazon EC2 API request-rate limits

Amazon EC2 API request-rate limits are enforced per account, per Region, and are independent for different groups of actions (for example, the non-mutating Describe* actions are limited separately from mutating actions). Review your account’s applied limits with Service Quotas for Amazon EC2 in each Region where you run Karpenter, compare them against the request rate you observe from the metrics above, and request an increase if your steady-state request rate is close to, or exceeds, your limits. Because these limits are per account and per Region, the combined request volume of every cluster in an account competes for the same limits.

Best practices

Use a multi-account architecture where clusters are isolated by account

Because request-rate limits are per account and per Region, concentrating a large number of clusters in a single account concentrates all of their Amazon EC2 request volume against that one account’s limits. Use a multi-account architecture where clusters are isolated by account to spread the volume across multiple accounts’ limits. See the multi-account guidance in the AWS Well-Architected Framework.

Increasing refresh intervals as a workaround

Warning

Increasing these refresh intervals is a workaround, not a recommended configuration — avoid relying on it. It reduces Describe* call volume only by trading away freshness: Karpenter takes longer to observe changes such as a subnet’s available IP capacity or a new AMI, and may act on stale data as a result. Prefer the best practices above and an appropriately sized request-rate limit, and reach for this only as a temporary measure while you address the underlying call volume.

Karpenter refreshes each EC2NodeClass’s cached subnet, security group, and AMI data from Amazon EC2 on an interval. These intervals are configurable (see the Settings reference):

SUBNET_REFRESH_INTERVAL — how often subnet data is refreshed (bounds DescribeSubnets). Defaults to 1m.
SECURITY_GROUP_REFRESH_INTERVAL — how often security group data is refreshed (bounds DescribeSecurityGroups). Defaults to 1m.
AMI_REFRESH_INTERVAL — how often AMI data is refreshed (bounds DescribeImages). Defaults to 1m.

Increasing an interval reduces that call’s steady-state rate proportionally — for example, changing an interval from 1m to 5m reduces that call’s rate by roughly 5x.

V1.15: Utilizing GPUs and EFAs with Dynamic Resource Allocation

Mon, 01 Jan 0001 00:00:00 +0000

Feature State: Alpha

Karpenter can provision nodes for pods that request NVIDIA GPUs and Elastic Fabric Adapters (EFAs) through Dynamic Resource Allocation (DRA). To enable this support, set settings.enableDRA in the Helm chart.

Note

DRA requires Kubernetes 1.34 or later. GPU time-slicing requires Kubernetes 1.36 or later.

Overview

With device plugins, a pod asks for devices as an integer count of an extended resource, such as nvidia.com/gpu: 1. With DRA, a pod references a ResourceClaim, which describes the devices it needs. A claim can select devices by their attributes using CEL, require several devices to share an attribute (for example, a GPU and an EFA on the same PCIe root), and request a portion of a device’s capacity. A DRA driver runs on each node and publishes the node’s devices and their attributes as ResourceSlices. The scheduler allocates each claim from the devices in those ResourceSlices.

The drivers only publish ResourceSlices once a node is running, but Karpenter has to choose an instance type before the node exists. To close that gap, Karpenter includes metadata for each supported instance type that describes the devices the NVIDIA and EFA DRA drivers publish on that instance type. When a pod with ResourceClaims is pending, Karpenter evaluates its claims against this metadata, along with the ResourceSlices of existing nodes. If no existing node can satisfy the claims, Karpenter launches the cheapest instance type that can. This works from zero: no GPU node has to exist first, and NodePools and EC2NodeClasses need no DRA-specific configuration.

Note

Static NodePools launch nodes without simulating pods, and the DRA drivers and kube-scheduler handle allocation on those nodes. DRA does not require explicit Karpenter support for static NodePools. This page covers dynamic provisioning.

Capabilities

DRA makes requests possible that extended resources can’t express. For example:

Select GPUs by their attributes, such as model, architecture, CUDA compute capability, or memory. See Requesting a specific GPU.
Align devices on the same PCIe root. For example, request a GPU and an EFA that share a PCIe root for GPUDirect RDMA, or two GPUs that share a PCIe root. See Requesting a GPU and EFA that share a PCIe root.
Share a GPU between pods with consumable capacity, where each claim takes a share of a GPU. See Requesting a timesliced GPU.

Installing the Drivers

Install each driver by following its install guide, then set the Helm values below.

Warning

A DRA driver and a device plugin must never manage the same device on the same node. If you also run the NVIDIA or EFA device plugins in your cluster, see Running DRA drivers alongside device plugins.

NVIDIA

Install the NVIDIA DRA driver by following the NVIDIA DRA driver install guide or the EKS guide. Neither value below is the chart’s default, so set both in the driver’s Helm chart:

Helm value	Default	Value	Why
`gpuResourcesEnabledOverride`	`false`	`true`	Required for GPU allocation.
`resources.computeDomains.enabled`	`true`	`false`	Optional. Disables ComputeDomains, which Karpenter doesn’t support provisioning for.

To share GPUs between pods, you also set the driver’s consumable shares values. See Requesting a timesliced GPU.

The chart creates the gpu.nvidia.com DeviceClass. The kubelet plugin tolerates the nvidia.com/gpu taint by default. Its default affinity only schedules it on nodes with a GPU presence label, such as nvidia.com/gpu.present=true. The EKS-optimized AL2023 NVIDIA AMI sets that label on GPU nodes, and Karpenter selects it automatically for GPU instance types with the al2023@latest alias. If your nodes use another AMI, check that it sets the label. If it doesn’t, add the label to your DRA NodePool. Otherwise the driver won’t run on the nodes Karpenter launches, and DRA pods stay pending:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: gpu-dra
spec:
 template:
 metadata:
 labels:
 nvidia.com/gpu.present: "true"

Note

Karpenter doesn’t support dynamic provisioning for ComputeDomains. It has no metadata for the compute-domain.nvidia.com driver, so it won’t launch nodes for claims on ComputeDomain DeviceClasses.

EFA

Install the EFA DRA driver (DRANET) by following Install the EFA DRA driver. Set this value in the aws-dranet Helm chart:

Helm value	Value	Why
`tolerations`	`[{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]`	The DaemonSet doesn’t tolerate the `nvidia.com/gpu` taint by default, so it won’t run on tainted GPU nodes without this.

The chart creates the efa.networking.k8s.aws DeviceClass. It selects devices whose dra.net/pciDevice attribute is Elastic Fabric Adapter (EFA). DRANET publishes every network interface on a node, but Karpenter’s metadata includes only EFA devices, so request EFAs through this DeviceClass.

Enabling Karpenter Support

Set settings.enableDRA: true in Karpenter’s Helm values to enable Karpenter’s support for every DRA driver it models, currently the NVIDIA GPU and EFA drivers. For how to change the values of an existing installation, see the upgrade guide.

This enables the DRANVIDIAGPU and DRAEFA feature gates and sets settings.ignoreDRARequests to false. The chart always grants Karpenter read access to DeviceClasses, ResourceClaims, and ResourceSlices.

settings.enableDRA assumes both drivers run on your DRA nodes. If a driver isn’t running, Karpenter launches nodes for its claims that can never serve them. If you only run one of the drivers, enable its gate individually instead.

Note

When you upgrade an existing release with helm upgrade --reuse-values, Helm also reuses the default values of the chart version you installed before. Earlier chart versions default settings.ignoreDRARequests to true, so setting settings.enableDRA with --reuse-values fails with the error above, even if you never set settings.ignoreDRARequests yourself. Use --reset-then-reuse-values (Helm 3.14 or later) instead. If you previously set settings.awsFeatureGates.draNVIDIAGPU or settings.awsFeatureGates.draEFA, unset them in the same upgrade, for example with --set settings.awsFeatureGates.draNVIDIAGPU=null.

Example: Requesting a specific GPU via DRA

This ResourceClaimTemplate requests one GPU that’s Hopper or newer (CUDA compute capability 9.0 or later) with at least 100Gi of memory:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: large-gpu
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 deviceClassName: gpu.nvidia.com
 count: 1
 selectors:
 - cel:
 expression: |
 device.attributes["gpu.nvidia.com"].cudaComputeCapability.compareTo(semver("9.0.0")) >= 0 &&
 device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("100Gi")) >= 0
---
apiVersion: v1
kind: Pod
metadata:
 name: inference
spec:
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 resourceClaims:
 - name: gpu
 resourceClaimTemplateName: large-gpu
 containers:
 - name: model
 image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
 command: ["/bin/sh", "-c", "nvidia-smi && sleep infinity"]
 resources:
 claims:
 - name: gpu

Karpenter only considers instance types with a GPU that matches the selector: here, H200 (p5e, p5en) and Blackwell (p6-b200, p6-b300). It launches the cheapest one the NodePool allows. To request several GPUs, raise count. Karpenter checks that the instance type has enough matching GPUs.

You can only select on attributes that Karpenter knows before launch. See Supported Attributes by Driver.

Enable DRA support with settings.enableDRA (or both individual gates), and install both drivers. This ResourceClaimTemplate requests one GPU and one EFA, and requires them to share a PCIe root:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: gpu-efa-aligned
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 deviceClassName: gpu.nvidia.com
 count: 1
 - name: efa
 exactly:
 deviceClassName: efa.networking.k8s.aws
 count: 1
 constraints:
 - requests: ["gpu", "efa"]
 matchAttribute: resource.kubernetes.io/pcieRoot

Reference the claim from a pod as in the previous example. Karpenter only launches instance types where a GPU and an EFA share a PCIe root. For example, each of the eight GPUs in a p5.48xlarge shares a PCIe root with four EFAs.

Warning

Configuring spec.networkInterfaces on an EC2NodeClass used for EFA DRA workloads is currently unsupported and results in undefined behavior. Leave it unset.

Example: Requesting a timesliced GPU

Note

GPU sharing through DRA requires Kubernetes 1.36 or later, where consumable capacity is enabled by default.

With the NVIDIA DRA driver’s consumable shares, several claims can share one GPU, and the driver time-slices between them. First, enable consumable shares by setting these values in the NVIDIA DRA driver’s Helm chart. This example splits each GPU into four shares:

Helm value	Value
`featureGates.ConsumableShares`	`true`
`consumableShares`	`4`

For the other modes and how the driver applies them, see the consumable capacity guide.

Karpenter can’t read the driver’s configuration, so set the same mode with the karpenter.k8s.aws/nvidia-consumable-capacity annotation on the EC2NodeClass:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: gpu-dra
 annotations:
 karpenter.k8s.aws/nvidia-consumable-capacity: "4"
spec:
 # ...

The annotation accepts the same values as the driver’s consumableShares setting:

Value	Karpenter models each GPU as
absent or `disabled`	Not shared. Each GPU serves one claim.
positive integer `N`	Shared, with `N` shares. A claim takes one share by default, and its memory request defaults to 0.
`memory`	Shared by memory. A claim with no memory request takes the GPU’s full memory.
`unlimited`	Shared without limit. A claim’s memory request defaults to 0.

Then give each pod its own claim from a template. A plain GPU request takes one share:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
 name: gpu-share
spec:
 spec:
 devices:
 requests:
 - name: gpu
 exactly:
 deviceClassName: gpu.nvidia.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: shared-inference
spec:
 replicas: 8
 selector:
 matchLabels:
 app: shared-inference
 template:
 metadata:
 labels:
 app: shared-inference
 spec:
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 resourceClaims:
 - name: gpu
 resourceClaimTemplateName: gpu-share
 containers:
 - name: model
 image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
 command: ["/bin/sh", "-c", "nvidia-smi && sleep infinity"]
 resources:
 claims:
 - name: gpu

Karpenter packs up to four of these claims onto each GPU, so the eight replicas need only two GPUs.

In memory mode, request memory instead, for example memory: 10Gi. Shares and memory only affect scheduling. They don’t limit how much GPU memory a process actually uses.

Appendix

Reference configuration

EC2NodeClass and NodePool

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: gpu-dra
 # Only for GPU sharing. Must match the driver's consumableShares setting.
 # annotations:
 # karpenter.k8s.aws/nvidia-consumable-capacity: "4"
spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
 amiSelectorTerms:
 - alias: al2023@latest # Resolves to the NVIDIA variant for GPU instance types
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 securityGroupSelectorTerms:
 # For EFA, include a security group that allows all traffic to and from itself
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 # Leave networkInterfaces unset. Karpenter attaches every EFA when a pod is allocated dra.net devices.
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: gpu-dra
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: gpu-dra
 requirements:
 - key: karpenter.k8s.aws/instance-family
 operator: In
 values: ["g6", "g6e", "p5", "p5en", "p6-b200"]
 taints:
 - key: nvidia.com/gpu
 effect: NoSchedule

Pods

A pod that uses DRA devices needs:

A spec.resourceClaims entry that references a ResourceClaimTemplate (one claim per pod) or a ResourceClaim (shared by every pod that references it).
A resources.claims entry on each container that uses the devices.
A toleration for the nvidia.com/gpu taint.

Karpenter treats a pod as a DRA pod only if it has spec.resourceClaims or a container lists resources.claims. A pod that requests the nvidia.com/gpu extended resource isn’t treated as a DRA pod, even when a DeviceClass serves that resource through DRA.

Running DRA drivers alongside device plugins

A DRA driver and a device plugin must never manage the same device on the same node. If both run, they can each hand out the same device, oversubscribing it without any error. If your cluster also runs the NVIDIA device plugin or the EFA device plugin, keep each node on one mechanism by giving DRA and device plugin NodePools separate labels.

Karpenter’s DRA metadata applies to every NodePool that allows a supported instance type, including NodePools whose nodes run the device plugins. If a DRA pod can schedule to both, Karpenter can launch it on a device plugin node, where no DRA driver publishes devices, and the pod stays pending. Constrain DRA pods to your DRA NodePools as well.

Label each NodePool with the mechanism its nodes use, for example example.com/device-manager: dra or example.com/device-manager: device-plugin:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: gpu-dra
spec:
 template:
 metadata:
 labels:
 example.com/device-manager: dra

Set a matching nodeSelector in each chart’s Helm values:

Chart	Helm value	Value
NVIDIA DRA driver	`kubeletPlugin.nodeSelector`	`example.com/device-manager: dra`
EFA DRA driver (DRANET)	`nodeSelector`	`example.com/device-manager: dra`
NVIDIA device plugin	`nodeSelector`	`example.com/device-manager: device-plugin`
EFA device plugin	`nodeSelector`	`example.com/device-manager: device-plugin`

Add a nodeSelector on example.com/device-manager: dra to pods that use DRA devices:
```
spec:
 nodeSelector:
 example.com/device-manager: dra
```

Supported Instance Types

Karpenter includes DRA metadata for the instance types below. An instance type with no EFA devices listed doesn’t support EFA. Support for a new instance type requires a new Karpenter release.

Instance type	GPU	GPUs	EFA devices
`g4dn.xlarge`	Tesla T4	1	-
`g4dn.2xlarge`	Tesla T4	1	-
`g4dn.4xlarge`	Tesla T4	1	-
`g4dn.8xlarge`	Tesla T4	1	1
`g4dn.12xlarge`	Tesla T4	4	1
`g4dn.16xlarge`	Tesla T4	1	1
`g4dn.metal`	Tesla T4	8	1
`g5.xlarge`	NVIDIA A10G	1	-
`g5.2xlarge`	NVIDIA A10G	1	-
`g5.4xlarge`	NVIDIA A10G	1	-
`g5.8xlarge`	NVIDIA A10G	1	1
`g5.12xlarge`	NVIDIA A10G	4	1
`g5.16xlarge`	NVIDIA A10G	1	1
`g5.24xlarge`	NVIDIA A10G	4	1
`g5.48xlarge`	NVIDIA A10G	8	1
`g5g.xlarge`	NVIDIA T4G	1	-
`g5g.2xlarge`	NVIDIA T4G	1	-
`g5g.4xlarge`	NVIDIA T4G	1	-
`g5g.8xlarge`	NVIDIA T4G	1	-
`g5g.16xlarge`	NVIDIA T4G	2	-
`g5g.metal`	NVIDIA T4G	2	-
`g6.xlarge`	NVIDIA L4	1	-
`g6.2xlarge`	NVIDIA L4	1	-
`g6.4xlarge`	NVIDIA L4	1	-
`g6.8xlarge`	NVIDIA L4	1	1
`g6.12xlarge`	NVIDIA L4	4	1
`g6.16xlarge`	NVIDIA L4	1	1
`g6.24xlarge`	NVIDIA L4	4	1
`g6.48xlarge`	NVIDIA L4	8	1
`g6e.xlarge`	NVIDIA L40S	1	-
`g6e.2xlarge`	NVIDIA L40S	1	-
`g6e.4xlarge`	NVIDIA L40S	1	-
`g6e.8xlarge`	NVIDIA L40S	1	1
`g6e.12xlarge`	NVIDIA L40S	4	1
`g6e.16xlarge`	NVIDIA L40S	1	1
`g6e.24xlarge`	NVIDIA L40S	4	2
`g6e.48xlarge`	NVIDIA L40S	8	4
`g6f.large`	NVIDIA L4-3Q	1	-
`g6f.xlarge`	NVIDIA L4-3Q	1	-
`g6f.2xlarge`	NVIDIA L4-6Q	1	-
`g6f.4xlarge`	NVIDIA L4-12Q	1	-
`gr6.4xlarge`	NVIDIA L4	1	-
`gr6.8xlarge`	NVIDIA L4	1	1
`gr6f.4xlarge`	NVIDIA L4-12Q	1	-
`p3dn.24xlarge`	Tesla V100-SXM2-32GB	8	1
`p4d.24xlarge`	NVIDIA A100-SXM4-40GB	8	4
`p4de.24xlarge`	NVIDIA A100-SXM4-80GB	8	4
`p5.4xlarge`	NVIDIA H100 80GB HBM3	1	1
`p5.48xlarge`	NVIDIA H100 80GB HBM3	8	32
`p5e.48xlarge`	NVIDIA H200	8	32
`p5en.48xlarge`	NVIDIA H200	8	16
`p6-b200.48xlarge`	NVIDIA B200	8	8
`p6-b300.48xlarge`	NVIDIA B300 SXM6 AC	8	16

Supported Attributes by Driver

Karpenter can only evaluate a claim against attributes it knows before the node launches. Some attributes can be used in two ways:

Selector: a CEL expression in a claim or DeviceClass that compares the attribute to a value, for example device.attributes["gpu.nvidia.com"].architecture == "Hopper".
matchAttribute: a constraint that requires every device allocated for a set of requests to have the same value for the attribute, without naming the value.

Other attributes can only be used with matchAttribute. They describe how devices relate to each other on a node. Their values are resolved at runtime, or are specific to a platform, so Karpenter can tell whether devices will match but not what the value will be.

NVIDIA (`gpu.nvidia.com`)

Attribute	Type	Example value	Selector	matchAttribute
`productName`	string	`NVIDIA H100 80GB HBM3`	✓	✓
`architecture`	string	`Hopper`	✓	✓
`brand`	string	`Nvidia`	✓	✓
`cudaComputeCapability`	version	`9.0.0`	✓	✓
`type`	string	`gpu`	✓	✓
`driverVersion`	version	`580.82.7`		✓
`cudaDriverVersion`	version	`13.0.0`		✓
`resource.kubernetes.io/pcieRoot`	string	`pci0000:10`		✓

Capacity	Example value	Selector	Capacity request
`memory`	`81152Mi`	✓	✓ with consumable capacity
`shares`	`4`	✓	✓ with an integer consumable capacity mode

EFA (`dra.net`)

Attribute	Type	Example value	Selector	matchAttribute
`pciDevice`	string	`Elastic Fabric Adapter (EFA)`	✓	✓
`pciVendor`	string	`Amazon.com, Inc.`	✓	✓
`pciSubsystem`	string	`efa1`	✓	✓
`rdma`	bool	`true`	✓	✓
`numaNode`	int	`0`	✓	✓
`resource.kubernetes.io/pcieRoot`	string	`pci0000:10`		✓

V1.15: Utilizing On-Demand Capacity Reservations and Capacity Blocks

Mon, 01 Jan 0001 00:00:00 +0000

Feature State: Beta

Karpenter introduced native support for EC2 On-Demand Capacity Reservations (ODCRs) in v1.3, enabling users to select upon and prioritize specific capacity reservations. In v1.6, this support was expanded to include EC2 Capacity Blocks for ML. In v1.10, this was further extended to support Interruptible Capacity Reservations. To enable native ODCR support, ensure the ReservedCapacity feature gate is enabled.

Note

If you were previously utilizing open ODCRs using Karpenter, review the migration section of this task before enabling this feature.

Selecting Capacity Reservations

To configure native ODCR support, you will need to make updates to both your EC2NodeClass and NodePool. First, you should configure capacityReservationSelectorTerms on your EC2NodeClass. Similar to amiSelectorTerms, you can specify a number of terms which are ANDed together to select ODCRs in your AWS account. The following example demonstrates how to select all capacity reservations tagged with application: foobar in addition to cr-56fac701cc1951b03:

capacityReservationSelectorTerms:
- tags:
 application: foobar
- id: cr-56fac701cc1951b03

Note

Capacity blocks are modeled as on-demand capacity reservations in EC2. To select capacity blocks, specify them in your capacityReservationSelectorTerms in the same way you would for a default ODCR.

For more information on configuring capacityReservationSelectorTerms, see the NodeClass docs.

Additionally, you will need to update your NodePool to be compatible with ODCRs. Karpenter doesn’t model ODCRs as standard on-demand capacity, and instead uses a dedicated capacity type: reserved. For a NodePool to utilize ODCRs, it must be compatible with karpenter.sh/capacity-type: reserved. The following example demonstrates how to configure a NodePool to prioritize ODCRs and fallback to on-demand capacity:

requirements:
- key: karpenter.sh/capacity-type
 operator: In
 values: ['reserved', 'on-demand']

Additionaly, Karpenter supports the following scheduling labels:

Label	Example	Description
`karpenter.k8s.aws/capacity-reservation-id`	`cr-56fac701cc1951b03`	The capacity reservation’s ID
`karpenter.k8s.aws/capacity-reservation-type`	`default` or `capacity-block`	The type of capacity reservation
`karpenter.k8s.aws/capacity-reservation-interruptible`	`true` or `false`	Whether the capacity reservation is interruptible

These labels will only be present on reserved nodes. They are supported as NodePool requirements and as pod scheduling constaints (e.g. node affinity).

Warning

Karpenter does not support open matching for ODCRs. This means that all ODCRs you wish to utilize, even those with open instance eligibility, must be included in your NodeClass’ spec.capacityReservationSelectorTerms.

Prioritization Behavior

NodePools are not limited to a single compatible capacity-type – they can be compatible with any combination of the available capacity-types (on-demand, spot, and reserved). Consider the following NodePool requirements:

requirements:
- key: karpenter.sh/capacity-type
 operator: In
 values: ['reserved', 'on-demand', 'spot']

In this example, the NodePool is compatible with all capacity types. Karpenter will prioritize ODCRs, but if none are available or none are compatible with the pending workloads it will fallback to spot or on-demand. Similarly, Karpenter will prioritize reserved capacity during consolidation. Since ODCRs are pre-paid, Karpenter will model them as free and consolidate spot / on-demand nodes when possible.

Disrupting Nodes in Full Reservations

Karpenter pre-spins a replacement before it disrupts a node. When a node’s reservation is full and its pods can’t run anywhere else, such as a lower-weight on-demand NodePool, Karpenter can’t pre-spin a replacement, and Drift and Node Auto Repair are blocked for that node. To replace these nodes in place, enable the TerminateFirstDrift and TerminateFirstRepair feature gates. Karpenter will then terminate the node first and launch its replacement into the freed reservation slot. See Terminate-First Disruption for details.

Warning

Don’t enable terminate-first disruption unless this Karpenter installation is the only thing that launches into its capacity reservations. Anything else that launches into the reservation, such as another Karpenter installation, an Auto Scaling group, or a matching instance consuming an open reservation, can claim the freed slot first, leaving the terminated node’s pods pending indefinitely.

Expiration

An instance launched into an ODCR is not necessarily in that ODCR indefinitely. The ODCR could expire, be cancelled, or the instance could be manually removed from the ODCR. If any of these occur, and Karpenter detects that the instance no longer belongs to an ODCR, it will update the karpenter.sh/capacity-type label to on-demand.

Capacity Blocks

Unlike default ODCRs, Capacity Blocks must have an end time. Additionally, instances launched into a capacity block will be terminated by EC2 ahead of the end time, rather than becoming standard on-demand capacity.

From the AWS docs:

You can use all the instances you reserved until 30 minutes (for instance types) or 60 minutes (for UltraServer type) before the end time of the Capacity Block. With 30 minutes (for instance types) or 60 minutes (for UltraServer types) left in your Capacity Block reservation, we begin terminating any instances that are running in the Capacity Block. We use this time to clean up your instances before delivering the Capacity Block to the next customer.

Karpenter will preemptively begin draining nodes launched for capacity blocks 10 minutes before EC2 begins termination, ensuring your workloads can gracefully terminate before reclaimation.

Interruptible Capacity Reservations

Unlike default ODCRs, capacity launched from interruptible ODCRs can be interrupted when capacity is reclaimed back the source ODCR.

From the AWS docs, when capacity is reclaimed:

Running instances receive a 2-minute interruption warning through EventBridge events. After the notice period, running instances in the reclaimed capacity enter a shutting down state and get terminated.

Karpenter will begin draining nodes launched for IODCRs when the 2-minute interruption warning is recieved.

Migrating From Previous Versions

Although it was not natively supported, it was possible to utilize ODCRs on previous versions of Karpenter. If a NodeClaim’s requirements happened to be compatible with an open ODCR in the target AWS account, it may have launched an instance into that open ODCR. This could be ensured by constraining a NodePool such that it was only compatible with the desired open ODCR, and limits could be used to enable fallback to a different NodePool once the ODCR was exhausted. This behavior is no longer supported when native on-demand capacity support is enabled.

If you were relying on this behavior, you should configure your EC2NodeClasses to select the desired ODCRs before enabling the feature gate. You should also ensure any NodePools which you wish to use with ODCRs are compatible with karpenter.sh/capacity-type: reserved. Performing these steps before enabling the feature gate will ensure that Karpenter can immediately continue utilizing your reservations, rather than falling back to on-demand.

Karpenter – Tasks

V1.15: Using Nitro Enclaves

What are Nitro Enclaves?

Warning

Warning

Prerequisites

Enable NodeOverlays

Enable Nitro Enclaves on the EC2NodeClass

Warning

Label the NodePool

Configure the Device Plugin

Model Enclave Resources with a NodeOverlay

Schedule an Enclave Workload

Validate the Configuration

Capacity Considerations

Additional Resources

Follow-up

V1.15: Managing AMIs

Important

How Karpenter assigns AMIs to nodes

Important

Controlling AMI Replacement

Pinning AMIs

Note

Testing AMIs

Using Disruption Budgets

Node Disruption Budgets

Pod Disruption Budgets

Follow-up

V1.15: Monitoring Amazon EC2 API Usage

What Amazon EC2 APIs Karpenter calls

How to observe call volume and throttling

See also

How to compare against your account’s Amazon EC2 API request-rate limits

Best practices

Use a multi-account architecture where clusters are isolated by account

Increasing refresh intervals as a workaround

Warning

V1.15: Utilizing GPUs and EFAs with Dynamic Resource Allocation

Note

Overview

Note

Capabilities

Installing the Drivers

Warning

NVIDIA

Note

EFA

Enabling Karpenter Support

Note

Example: Requesting a specific GPU via DRA

Example: Requesting a GPU and EFA that share a PCIe Root

Warning

Example: Requesting a timesliced GPU

Note

Appendix

Reference configuration

EC2NodeClass and NodePool

Pods

Running DRA drivers alongside device plugins

Supported Instance Types

Supported Attributes by Driver

NVIDIA (gpu.nvidia.com)

EFA (dra.net)

V1.15: Utilizing On-Demand Capacity Reservations and Capacity Blocks

Note

Selecting Capacity Reservations

Note

Warning

Prioritization Behavior

Disrupting Nodes in Full Reservations

Warning

Expiration

Capacity Blocks

Interruptible Capacity Reservations

Migrating From Previous Versions

NVIDIA (`gpu.nvidia.com`)

EFA (`dra.net`)