Amazon Web Services (AWS)

Control Provisioning with Labels

The Provisioner CRD supports defining node properties like instance type and zone.For certain well-known labels (documented below), Karpenter will provision nodes accordingly. For example, in response to a label of topology.kubernetes.io/zone=us-east-1c, Karpenter will provision nodes in that availability zone.

Instance Types

Karpenter supports specifying AWS instance type.

The default value includes all instance types with the exclusion of metal (non-virtualized), non-HVM, and GPU instances.

If necessary, Karpenter supports defining a limited list of default instance types.

If more than one type is listed, Karpenter will determine the instance type to minimize the number of new nodes.

View the full list of instance types with aws ec2 describe-instance-types.

Example

Set Default with provisioner.yaml

spec:
  instanceTypes:
    - m5.large

Override with workload manifest (e.g., pod)

spec:
  template:
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: m5.large

Availability Zones

topology.kubernetes.io/zone=us-east-1c

  • key: topology.kubernetes.io/zone
  • value example: us-east-1c
  • value list: aws ec2 describe-availability-zones --region <region-name>

Karpenter can be configured to create nodes in a particular zone. Note that the Availability Zone us-east-1a for your AWS account might not have the same location as us-east-1a for another AWS account.

Learn more about Availability Zone IDs.

Capacity Type

  • key: node.k8s.aws/capacity-type
  • values
    • on-demand (default)
    • spot

Karpenter supports specifying capacity type and defaults to on-demand.

Specify this value on the provisioner to enable spot instances. Spot instances may be preempted, and should not be used for critical workloads.

Example

Set Default with provisioner.yaml

spec:
  labels:
    node.k8s.aws/capacity-type: spot

Override with workload manifest (e.g., pod)

spec:
  template:
    spec:
      nodeSelector:
        node.k8s.aws/capacity-type: spot

Architecture

  • key: kubernetes.io/arch
  • values
    • amd64 (default)
    • arm64

Karpenter supports amd64 nodes, and arm64 nodes.

Example

Set Default with provisioner.yaml

spec:
  labels:
    kubernetes.io/arch: arm64

Override with workload manifest (e.g., pod)

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/arch: amd64

Operating System

  • key: kubernetes.io/os
  • values
    • linux (default)

At this time, Karpenter only supports Linux OS nodes.

Accelerators, GPU

Accelerator (e.g., GPU) values include

  • nvidia.com/gpu
  • amd.com/gpu
  • aws.amazon.com/neuron

Karpenter supports accelerators, such as GPUs.

To enable instances with accelerators, use the instance type well known label selector.

Additionally, include a resource requirement in the workload manifest. Thus, accelerator dependent pod will be scheduled onto the appropriate node.

accelerator resource in workload manifest (e.g., pod)

spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            nvidia.com/gpu: "1"
Last modified July 30, 2021 : Add AWS Cloud Provider Docs (#557) (2519db5)