Node Templates

Configure AWS specific settings

Node Templates enable configuration of AWS specific settings. Each provisioner must reference an AWSNodeTemplate using spec.providerRef. Multiple provisioners may point to the same AWSNodeTemplate.

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector: { ... }        # required, discovers tagged subnets to attach to instances
  securityGroupSelector: { ... } # required, discovers tagged security groups to attach to instances
  instanceProfile: "..."         # optional, overrides the node's identity from global settings
  amiFamily: "..."               # optional, resolves a default ami and userdata
  amiSelector: { ... }           # optional, discovers tagged amis to override the amiFamily's default
  userData: "..."                # optional, overrides autogenerated userdata with a merge semantic
  tags: { ... }                  # optional, propagates tags to underlying EC2 resources
  metadataOptions: { ... }       # optional, configures IMDS for the instance
  blockDeviceMappings: [ ... ]   # optional, configures storage devices for the instance
  detailedMonitoring: "..."      # optional, configures detailed monitoring for the instance
status:
  subnets: { ... }               # resolved subnets
  securityGroups: { ... }        # resolved security groups

Refer to the Provisioner docs for settings applicable to all providers. See below for other AWS provider-specific parameters.

spec.subnetSelector

The AWSNodeTemplate discovers subnets using AWS tags. Subnets may be specified by any AWS tag, including Name. Selecting tag values using wildcards (*) is supported. Subnet IDs may be specified by using the key aws-ids and then passing the IDs as a comma-separated string value. When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used.

Examples

Select all with a specified tag key:

spec:
  subnetSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match)::

spec:
  subnetSelector:
    Name: my-subnet
    MyTag: '' # matches all resources with the tag

Select using comma separated tag values:

spec:
  subnetSelector:
    Name: "my-subnet-1,my-subnet-2"

Select using wildcards:

spec:
  subnetSelector:
    Name: "*Public*"

Select by ID:

spec:
  subnetSelector:
    aws-ids: "subnet-09fa4a0a8f233a921,subnet-0471ca205b8a129ae"

spec.securityGroupSelector

The security group of an instance is comparable to a set of firewall rules. EKS creates at least two security groups by default, review the documentation for more info. Security groups may be specified by any AWS tag, including “Name”. Selecting tags using wildcards (*) is supported.

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=karpenter.sh/discovery/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need a more specific securityGroupSelector.

Examples

Select all with a specified tag key:

spec:
  securityGroupSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
  securityGroupSelector:
    Name: my-security-group
    MyTag: '' # matches all resources with the tag

Select by comma-separated tag values:

spec:
  securityGroupSelector:
     Name: "my-security-group-1,my-security-group-2"

Select by name using a wildcard:

spec:
  securityGroupSelector:
    Name: "*Public*"

Select by ID:

spec:
 securityGroupSelector:
   aws-ids: "sg-063d7acfb4b06c82c,sg-06e0cf9c198874591"

spec.instanceProfile

An InstanceProfile is a way to pass a single IAM role to EC2 instance launched the provisioner. A default profile is configured in global settings, but may be overridden here. The AWSNodeTemplate will not create an InstanceProfile automatically. The InstanceProfile must refer to a Role that has permission to connect to the cluster.

spec:
  instanceProfile: MyInstanceProfile

spec.amiFamily

The AMI used when provisioning nodes can be controlled by the amiFamily field. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM). When an amiFamily of Custom is chosen, then an amiSelector must be specified that informs Karpenter on which custom AMIs are to be used.

Currently, Karpenter supports amiFamily values AL2, Bottlerocket, Ubuntu and Custom. GPUs are only supported with AL2 and Bottlerocket.

spec:
  amiFamily: Bottlerocket

spec.amiSelector

AMISelector is used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through AWS tags, similar to subnetSelector. This field is optional, and Karpenter will use the latest EKS-optimized AMIs if an amiSelector is not specified.

EC2 AMIs may be specified by any AWS tag, including Name. Selecting tag values using wildcards (*) is supported.

EC2 AMI IDs may be specified by using the key aws-ids and then passing the IDs as a comma-separated string value.

AMI Selection

If an amiSelector matches more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

  • When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
  • If multiple AMIs are found that can be used, Karpenter will choose the latest one.
  • If no AMIs are found that can be used, then no nodes will be provisioned.

If you need to express other constraints for an AMI beyond architecture, you can express these constraints as tags on the AMI. For example, if you want to limit an EC2 AMI to only be used with instanceTypes that have an nvidia GPU, you can specify an EC2 tag with a key of karpenter.k8s.aws/instance-gpu-manufacturer and value nvidia on that AMI.

All labels defined in the scheduling documentation can be used as requirements for an EC2 AMI.

> aws ec2 describe-images --image-id ami-123 --query Images[0].Tags
[
    {
        "Key": "karpenter.sh/discovery",
        "Value": "my-cluster"
    },
    {
        "Key": "Name",
        "Value": "amazon-eks-node-1.21-customized-v0"
    },
    {
        "Key": "karpenter.k8s.aws/instance-gpu-manufacturer",
        "Value": "nvidia"
    }
]

Examples

Select all AMIs with a specified tag:

  amiSelector:
    karpenter.sh/discovery/MyClusterName: '*'

Select AMIs by name:

  amiSelector:
    Name: my-ami

Select AMIs by an arbitrary AWS tag key/value pair:

  amiSelector:
    MyAMITag: value

Specify AMIs explicitly by ID:

  amiSelector:
    aws-ids: "ami-123,ami-456"

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of AWS tags are listed below.

Name: karpenter.sh/provisioner-name/<provisioner-name>
karpenter.sh/provisioner-name: <provisioner-name>
kubernetes.io/cluster/<cluster-name>: owned

Additional tags can be added in the AWSNodeTemplate tags section which are merged with global tags in aws.tags (located in karpenter-global-settings ConfigMap) and can override the default tag values.

spec:
  tags:
    InternalAccountingTag: 1234
    dev.corp.net/app: Calculator
    dev.corp.net/team: MyTeam

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this provisioner using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this provisioner, the following default settings will be used.

spec:
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 2
    httpTokens: required

spec.blockDeviceMappings

The blockDeviceMappings field in an AWSNodeTemplate can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMI Family specified. For example, the Bottlerocket AMI Family defaults with two block device mappings, one for Bottlerocket’s control volume and the other for container resources such as images and logs.

Learn more about block device mappings.

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 10000
        encrypted: true
        kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
        deleteOnTermination: true
        throughput: 125
        snapshotID: snap-0123456789

spec.userData

You can control the UserData that is applied to your worker nodes via this field.

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: bottlerocket-example
spec:
  amiFamily: Bottlerocket
  instanceProfile: MyInstanceProfile
  subnetSelector:
    karpenter.sh/discovery: my-cluster
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster
  userData:  |
    [settings.kubernetes]
    kube-api-qps = 30
    [settings.kubernetes.eviction-hard]
    "memory.available" = "20%"    
  amiSelector:
    karpenter.sh/discovery: my-cluster

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your key file):

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: al2-example
spec:
  amiFamily: AL2
  instanceProfile: MyInstanceProfile
  subnetSelector:
    karpenter.sh/discovery: my-cluster
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster
  userData: |
    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="BOUNDARY"

    --BOUNDARY
    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/bin/bash
    mkdir -p ~ec2-user/.ssh/
    touch ~ec2-user/.ssh/authorized_keys
    cat >> ~ec2-user/.ssh/authorized_keys <<EOF
    {{ insertFile "../my-authorized_keys" | indent 4  }}
    EOF
    chmod -R go-w ~ec2-user/.ssh/authorized_keys
    chown -R ec2-user ~ec2-user/.ssh
    --BOUNDARY--    

For more examples on configuring these fields for different AMI families, see the examples here.

Merge Semantics

Karpenter will evaluate and merge the UserData that you specify in the AWSNodeTemplate resources depending upon the AMIFamily that you have chosen.

Bottlerocket

  • Your UserData must be valid TOML.
  • Karpenter will automatically merge settings to ensure successful bootstrap including cluster-name, api-server and cluster-certificate. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData.
    • All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you’ve specified settings.kubernetes.cluster-name, it will be overridden.
    • If MaxPods is specified via the binary arg to Karpenter, the value will override anything specified in the UserData.
    • If ClusterDNS is specified via spec.kubeletConfiguration, then that value will override anything specified in the UserData.
  • Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Consider the following example to understand how your custom UserData settings will be merged in.

Your UserData -

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter': 'will-be-overridden'

Final merged UserData -

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/provisioner-name' = 'provisioner'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

AL2 and Ubuntu

  • Your UserData must be in the MIME multi part archive format.
  • Karpenter will merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
    • Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Your UserData -

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"

--BOUNDARY--

The final merged UserData that will be applied to your worker nodes -

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--container-runtime containerd \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test  --max-pods=110'
--//--

You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet:

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: kubelet-config-example
spec:
  subnetSelector:
    karpenter.sh/discovery: my-cluster
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster
  userData: |
    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="BOUNDARY"

    --BOUNDARY
    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/bin/bash
    echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

    --BOUNDARY--

spec.detailedMonitoring

Enabling detailed monitoring on the node template controls the EC2 detailed monitoring feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches.

spec:
  detailedMonitoring: true

status.subnets

status.subnets contains the id and zone of the subnets utilized during node launch. The subnets are sorted by the available IP address count in decreasing order.

Examples

status:
  subnets:
  - id: subnet-0a462d98193ff9fac
    zone: us-east-2b
  - id: subnet-0322dfafd76a609b6
    zone: us-east-2c
  - id: subnet-0727ef01daf4ac9fe
    zone: us-east-2b
  - id: subnet-00c99aeafe2a70304
    zone: us-east-2a
  - id: subnet-023b232fd5eb0028e
    zone: us-east-2c
  - id: subnet-03941e7ad6afeaa72
    zone: us-east-2a

status.securityGroups

status.securityGroups contains the id of the security groups utilized during node launch.

Examples

  status:
    securityGroups:
    - id: sg-041513b454818610b
    - id: sg-0286715698b894bca
Last modified March 29, 2023 : fix 404s (#3655) (7131be23)