Node Templates

Node Templates enable configuration of AWS specific settings. Each provisioner must reference an AWSNodeTemplate using spec.providerRef. Multiple provisioners may point to the same AWSNodeTemplate.

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector: { ... }        # required, discovers tagged subnets to attach to instances
  securityGroupSelector: { ... } # required, discovers tagged security groups to attach to instances
  instanceProfile: "..."         # optional, overrides the node's identity from global settings
  amiFamily: "..."               # optional, resolves a default ami and userdata
  amiSelector: { ... }           # optional, discovers tagged amis to override the amiFamily's default
  userData: "..."                # optional, overrides autogenerated userdata with a merge semantic
  tags: { ... }                  # optional, propagates tags to underlying EC2 resources
  metadataOptions: { ... }       # optional, configures IMDS for the instance
  blockDeviceMappings: [ ... ]   # optional, configures storage devices for the instance

Refer to Provisioner API for settings applicable to all providers. See below for other AWS provider-specific parameters.

SubnetSelector (required)

The AWSNodeTemplate discovers subnets using AWS tags. Subnets may be specified by any AWS tag, including Name. Selecting tag values using wildcards (*) is supported. Subnet IDs may be specified by using the key aws-ids and then passing the IDs as a comma-separated string value. When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used.

Examples

Select all subnets with a specified tag:

  subnetSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select subnets by name:

  subnetSelector:
    Name: my-subnet

Select subnets by an arbitrary AWS tag key/value pair:

  subnetSelector:
    MySubnetTag: value

Select subnets using wildcards:

  subnetSelector:
    Name: "*Public*"

Specify subnets explicitly by ID:

  subnetSelector:
    aws-ids: "subnet-09fa4a0a8f233a921,subnet-0471ca205b8a129ae"

SecurityGroupSelector (required)

The security group of an instance is comparable to a set of firewall rules. EKS creates at least two security groups by default, review the documentation for more info. Security groups may be specified by any AWS tag, including “Name”. Selecting tags using wildcards (*) is supported.

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=karpenter.sh/discovery/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need a more specific securityGroupSelector.

Examples

Select all security groups with a specified tag:

spec:
  securityGroupSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select security groups by name, or another tag (all criteria must match):

 securityGroupSelector:
   Name: my-security-group
   MySecurityTag: '' # matches all resources with the tag

Select security groups by name using a wildcard:

 securityGroupSelector:
   Name: "*Public*"

Specify security groups explicitly by ID:

 securityGroupSelector:
   aws-ids: "sg-063d7acfb4b06c82c,sg-06e0cf9c198874591"

InstanceProfile

An InstanceProfile is a way to pass a single IAM role to EC2 instance launched the provisioner. A default profile is configured in global settings, but may be overriden here. The AWSNodeTemplate will not create an InstanceProfile automatically. The InstanceProfile must refer to a Role that has permission to connect to the cluster.

spec:
  instanceProfile: MyInstanceProfile

Amazon Machine Image (AMI) Family

The AMI used when provisioning nodes can be controlled by the amiFamily field. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM). When an amiFamily of Custom is chosen, then an amiSelector must be specified that informs Karpenter on which custom AMIs are to be used.

Currently, Karpenter supports amiFamily values AL2, Bottlerocket, Ubuntu and Custom. GPUs are only supported with AL2 and Bottlerocket.

spec:
  amiFamily: Bottlerocket

AMISelector

AMISelector is used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through AWS tags, similar to subnetSelector. This field is optional, and Karpenter will use the latest EKS-optimized AMIs if an amiSelector is not specified.

EC2 AMIs may be specified by any AWS tag, including Name. Selecting tag values using wildcards (*) is supported.

EC2 AMI IDs may be specified by using the key aws-ids and then passing the IDs as a comma-separated string value.

• When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
• If multiple AMIs are found that can be used, Karpenter will randomly choose any one.
• If no AMIs are found that can be used, then no nodes will be provisioned.

For additional data on how UserData is configured for Custom AMIs, and how more requirements can be specified for custom AMIs, follow this documentation.

Examples

Select all AMIs with a specified tag:

  amiSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select AMIs by name:

  amiSelector:
    Name: my-ami

Select AMIs by an arbitrary AWS tag key/value pair:

  amiSelector:
    MyAMITag: value

Specify AMIs explicitly by ID:

  amiSelector:
    aws-ids: "ami-123,ami-456"

UserData

You can control the UserData that needs to be applied to your worker nodes via this field. Review the Operating Systems documentation to learn more.

Tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of AWS tags are listed below.

Name: karpenter.sh/provisioner-name/<provisioner-name>
karpenter.sh/provisioner-name: <provisioner-name>
kubernetes.io/cluster/<cluster-name>: owned

Additional tags can be added in the AWSNodeTemplate tags section which are merged with and can override the default tag values.

spec:
  tags:
    InternalAccountingTag: 1234
    dev.corp.net/app: Calculator
    dev.corp.net/team: MyTeam

Metadata Options

Control the exposure of Instance Metadata Service on EC2 Instances launched by this provisioner using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this provisioner, the following default settings will be used.

spec:
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 2
    httpTokens: required

Block Device Mappings

The blockDeviceMappings field in an AWSNodeTemplate can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMI Family specified. For example, the Bottlerocket AMI Family defaults with two block device mappings, one for Bottlerocket’s control volume and the other for container resources such as images and logs.

Learn more about block device mappings.

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 10000
        encrypted: true
        kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
        deleteOnTermination: true
        throughput: 125
        snapshotID: snap-0123456789

AWS Specific Labels

The AWS cloud provider adds several labels to nodes that describe the node resources to make filtering instance types easier. These work at either the provisioner level as requirements or the pod level as node selectors or node affinities. The complete list, including the instance types they are applied to, is available in the Instance Types documentation. A sampling of these include:

• karpenter.k8s.aws/instance-cpu
• karpenter.k8s.aws/instance-memory
• karpenter.k8s.aws/instance-gpu-name

The karpenter.k8s.aws/instance-cpu and karpenter.k8s.aws/instance-memory values are numeric which also allows constructing requirements for them using the Gt and Lt operators.

The standard rules for Gt and Lt apply:

1. There can be only one value in the requirement
2. The value must be an integer

These requirements can be useful to select nodes of a particular “shape”. For example the following filters out all instance types with more than 8 CPUs or more than 16 GiB of memory:

  - key: karpenter.k8s.aws/instance-cpu
    operator: Lt
    values:
    - "9"
  - key: karpenter.k8s.aws/instance-memory
    operator: Lt
    values:
    - "16385"

A requirement that specifies a specific value for karpenter.k8s.aws/instance-gpu-name can be used to select for all instance types that have a particular GPU type.

  - key: karpenter.k8s.aws/instance-gpu-name
    operator: In
    values:
      - "v100"

Other Resources

Accelerators, GPU

Accelerator (e.g., GPU) values include

• nvidia.com/gpu
• amd.com/gpu
• aws.amazon.com/neuron
• habana.ai/gaudi

Karpenter supports accelerators, such as GPUs.

Additionally, include a resource requirement in the workload manifest. This will cause the GPU dependent pod to be scheduled onto the appropriate node.

Here is an example of an accelerator resource in a workload manifest (e.g., pod):

spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            nvidia.com/gpu: "1"

Pod ENI (Security Groups for Pods)

Pod ENI is a feature of the AWS VPC CNI Plugin which allows an Elastic Network Interface (ENI) to be allocated directly to a Pod. When enabled, the vpc.amazonaws.com/pod-eni extended resource is added to supported nodes. The Pod ENI feature can be used independently, but is most often used in conjunction with Security Groups for Pods. Follow the below instructions to enable support for Pod ENI and/or Security Groups for Pods in Karpenter.

Now that Pod ENI support is enabled in the AWS VPC CNI Plugin, you can enable Pod ENI support in Karpenter by setting the settings.aws.enablePodENI Helm chart value to true.

Here is an example of a pod-eni resource defined in a deployment manifest:

spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            vpc.amazonaws.com/pod-eni: "1"