# CloudFormation

> A description of the Getting Started CloudFormation file and permissions

The [Getting Started with Karpenter](/v1.14/getting-started/getting-started-with-karpenter/index.md) guide uses CloudFormation to bootstrap the cluster to enable Karpenter to create and manage nodes, as well as to allow Karpenter to respond to interruption events.
This document describes the `cloudformation.yaml` file used in that guide.
These descriptions should allow you to understand:

* What Karpenter is authorized to do with your EKS cluster and AWS resources when using the `cloudformation.yaml` file
* What permissions you need to set up if you are adding Karpenter to an existing cluster

## Overview

To download a particular version of `cloudformation.yaml`, set the version and use `curl` to pull the file to your local system:

```bash
export KARPENTER_VERSION="1.14.0"
curl https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > cloudformation.yaml
```

Following some header information, the rest of the `cloudformation.yaml` file describes the resources that CloudFormation deploys.
The sections of that file can be grouped together under the following general headings:

* [**Node Authorization**](#node-authorization): Creates a NodeInstanceProfile, attaches a NodeRole to it, and connects it to an IAM Identity Mapping used to authorize nodes to the cluster. This defines the permissions each node managed by Karpenter has to access EC2 and other AWS resources. This doesn't actually create the IAM Identity Mapping. That part is orchestrated by `eksctl` in the Getting Started guide.
* [**Controller Authorization**](#controller-authorization):  Creates 6 Karpenter controller policies that are attached to the service account's IAM role.
Again, the actual service account creation (`karpenter`), that is combined with these policies, is orchestrated by `eksctl` in the Getting Started guide.
* [**Interruption Handling**](#interruption-handling): Allows the Karpenter controller to see and respond to interruptions that occur with the nodes that Karpenter is managing. See the [Interruption](/v1.14/concepts/disruption/index.md#interruption) section of the Disruption page for details.

A lot of the object naming that is done by `cloudformation.yaml` is based on the following:

* Cluster name: With a username of `bob` the Getting Started Guide would name your cluster `bob-karpenter-demo`
That name would then be appended to any name below where `${ClusterName}` is included.

* Partition: Any time an ARN is used, it includes the [partition name](https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/partitions.html) to identify where the object is found. In most cases, that partition name is `aws`. However, it could also be `aws-cn` (for China Regions) or `aws-us-gov` (for AWS GovCloud US Regions).

## Node Authorization

The following sections of the `cloudformation.yaml` file set up IAM permissions for Kubernetes nodes created by Karpenter.
In particular, this involves setting up a node role that can be attached and passed to instance profiles that Karpenter generates at runtime:

* KarpenterNodeRole

### KarpenterNodeRole

This section of the template defines the IAM role attached to generated instance profiles.
Given a cluster name of `bob-karpenter-demo`, this role would end up being named `"KarpenterNodeRole-bob-karpenter-demo`.

```yaml
KarpenterNodeRole:
  Type: "AWS::IAM::Role"
  Properties:
    RoleName: !Sub "KarpenterNodeRole-${ClusterName}"
    Path: /
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              !Sub "ec2.${AWS::URLSuffix}"
          Action:
            - "sts:AssumeRole"
    ManagedPolicyArns:
      - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy"
      - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
      - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
      - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
```

The role created here includes several AWS managed policies, which are designed to provide permissions for specific uses needed by the nodes to work with EC2 and other AWS resources. These include:

* [AmazonEKS_CNI_Policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKS_CNI_Policy.html): Provides the permissions that the Amazon VPC CNI Plugin needs to configure EKS worker nodes.
* [AmazonEKSWorkerNodePolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKSWorkerNodePolicy.html): Lets Amazon EKS worker nodes connect to EKS Clusters.
* [AmazonEC2ContainerRegistryPullOnly](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEC2ContainerRegistryPullOnly.html): Allows pulling images from repositories in the Amazon EC2 Container Registry.
* [AmazonSSMManagedInstanceCore](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMManagedInstanceCore.html): Adds AWS Systems Manager service core functions for Amazon EC2.

If you were to use a node role from an existing cluster, you could skip this provisioning step and pass this node role to any EC2NodeClasses that you create. Additionally, you would ensure that the [IAM Integration Policy](#karpentercontrolleriamintegrationpolicy) has `iam:PassRole` permission to the role attached to the generated instance profiles.

## Controller Authorization

This section sets the AWS permissions for the Karpenter Controller. When used in the Getting Started guide, `eksctl` uses these permissions to create a service account (karpenter) whose IAM role has all 6 controller policies attached.

The controller permissions are split across 6 managed IAM policies:

* [KarpenterControllerNodeLifecyclePolicy](#karpentercontrollernodelifecyclepolicy) - EC2 instance and launch template lifecycle management
* [KarpenterControllerIAMIntegrationPolicy](#karpentercontrolleriamintegrationpolicy) - IAM instance profile management
* [KarpenterControllerEKSIntegrationPolicy](#karpentercontrollereksintegrationpolicy) - EKS cluster discovery
* [KarpenterControllerInterruptionPolicy](#karpentercontrollerinterruptionpolicy) - SQS interruption queue access
* [KarpenterControllerZonalShiftPolicy](#karpentercontrollerzonalshiftpolicy) - Zonal Shift status access
* [KarpenterControllerResourceDiscoveryPolicy](#karpentercontrollerresourcediscoverypolicy) - Read-only resource discovery

Someone wanting to add Karpenter to an existing cluster, instead of using `cloudformation.yaml`, would need to create these IAM policies directly and assign them to the role leveraged by the service account using IRSA or EKS Pod Identity.

### KarpenterControllerNodeLifecyclePolicy

The `NodeLifecyclePolicy` manages EC2 instance and launch template lifecycle operations. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerNodeLifecyclePolicy-bob-karpenter-demo`

```yaml
NodeLifecyclePolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerNodeLifecyclePolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

#### AllowScopedEC2InstanceAccessActions

The AllowScopedEC2InstanceAccessActions statement ID (Sid) identifies a set of EC2 resources that are allowed to be accessed with
[RunInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html) and [CreateFleet](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html) actions.
For `RunInstances` and `CreateFleet` actions, the Karpenter controller can read (but not create) `image`, `snapshot`, `security-group`, `subnet`, `capacity-reservation`, and `placement group` EC2 resources, scoped for the particular AWS partition and region.

```json
{
  "Sid": "AllowScopedEC2InstanceAccessActions",
  "Effect": "Allow",
  "Resource": [
    "arn:${AWS::Partition}:ec2:${AWS::Region}::image/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}::snapshot/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:security-group/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:subnet/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:placement-group/*"
  ],
  "Action": [
    "ec2:RunInstances",
    "ec2:CreateFleet"
  ]
}
```

#### AllowScopedEC2LaunchTemplateAccessActions

The AllowScopedEC2InstanceAccessActions statement ID (Sid) identifies launch templates that are allowed to be accessed with
[RunInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html) and [CreateFleet](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html) actions.
For `RunInstances` and `CreateFleet` actions, the Karpenter controller can read (but not create) `launch-template` EC2 resources that have the `kubernetes.io/cluster/${ClusterName}` tag be set to `owned` and a `karpenter.sh/nodepool` tag, scoped for the particular AWS partition and region. This ensures that an instance launch can't access launch templates that weren't provisioned by Karpenter.

```json
{
  "Sid": "AllowScopedEC2LaunchTemplateAccessActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
  "Action": [
    "ec2:RunInstances",
    "ec2:CreateFleet"
  ],
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
    },
    "StringLike": {
      "aws:ResourceTag/karpenter.sh/nodepool": "*"
    }
  }
}
```

#### AllowScopedEC2InstanceActionsWithTags

The AllowScopedEC2InstanceActionsWithTags Sid allows the
[RunInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html), [CreateFleet](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html), and [CreateLaunchTemplate](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateLaunchTemplate.html)
actions requested by the Karpenter controller to create all `fleet`, `instance`, `volume`, `network-interface`, `launch-template` or `spot-instances-request` EC2 resources (for the partition and region). It also requires that the `kubernetes.io/cluster/${ClusterName}` tag be set to `owned`, `aws:RequestTag/eks:eks-cluster-name` be set to `"${ClusterName}`, and a `karpenter.sh/nodepool` tag be set to any value. This ensures that Karpenter is only allowed to create instances for a single EKS cluster.

```json
{
  "Sid": "AllowScopedEC2InstanceActionsWithTags",
  "Effect": "Allow",
  "Resource": [
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:fleet/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:volume/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:network-interface/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
  ],
  "Action": [
    "ec2:RunInstances",
    "ec2:CreateFleet",
    "ec2:CreateLaunchTemplate"
  ],
  "Condition": {
    "StringEquals": {
      "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
    },
    "StringLike": {
      "aws:RequestTag/karpenter.sh/nodepool": "*"
    }
  }
}
```

#### AllowScopedResourceCreationTagging

The AllowScopedResourceCreationTagging Sid allows EC2 [CreateTags](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateTags.html)
actions on `fleet`, `instance`, `volume`, `network-interface`, `launch-template` and `spot-instances-request` resources, While making `RunInstance`, `CreateFleet`, or `CreateLaunchTemplate` calls. Additionally, this ensures that resources can't be tagged arbitrarily by Karpenter after they are created.
Conditions that must be met include that `aws:RequestTag/kubernetes.io/cluster/${ClusterName}` be set to `owned` and `aws:RequestTag/eks:eks-cluster-name` be set to `${ClusterName}`.

```json
{
  "Sid": "AllowScopedResourceCreationTagging",
  "Effect": "Allow",
  "Resource": [
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:fleet/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:volume/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:network-interface/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
  ],
  "Action": "ec2:CreateTags",
  "Condition": {
    "StringEquals": {
      "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
      "ec2:CreateAction": [
        "RunInstances",
        "CreateFleet",
        "CreateLaunchTemplate"
      ]
    },
    "StringLike": {
      "aws:RequestTag/karpenter.sh/nodepool": "*"
    }
  }
}
```

#### AllowScopedResourceTagging

The AllowScopedResourceTagging Sid allows EC2 [CreateTags](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateTags.html) actions on all instances created by Karpenter after their creation. It enforces that Karpenter is only able to update the tags on cluster instances it is operating on through the `kubernetes.io/cluster/${ClusterName}`" and `karpenter.sh/nodepool` tags.
Likewise, `RequestTag/eks:eks-cluster-name` must be set to `${ClusterName}`, if it exists, and `TagKeys` must equal `eks:eks-cluster-name`, `karpenter.sh/nodeclaim`, and `Name`, for all values.
```json
{
  "Sid": "AllowScopedResourceTagging",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
  "Action": "ec2:CreateTags",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
    },
    "StringLike": {
      "aws:ResourceTag/karpenter.sh/nodepool": "*"
    },
    "StringEqualsIfExists": {
      "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
    },
    "ForAllValues:StringEquals": {
      "aws:TagKeys": [
        "eks:eks-cluster-name",
        "karpenter.sh/nodeclaim",
        "Name"
      ]
    }
  }
}
```

#### AllowScopedDeletion

The AllowScopedDeletion Sid allows [TerminateInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_TerminateInstances.html) and [DeleteLaunchTemplate](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteLaunchTemplate.html) actions to delete instance and launch-template resources, provided that `karpenter.sh/nodepool` and `kubernetes.io/cluster/${ClusterName}` tags are set. These tags must be present on all resources that Karpenter is going to delete. This ensures that Karpenter can only delete instances and launch templates that are associated with it.

```json
{
  "Sid": "AllowScopedDeletion",
  "Effect": "Allow",
  "Resource": [
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
    "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*"
  ],
  "Action": [
    "ec2:TerminateInstances",
    "ec2:DeleteLaunchTemplate"
  ],
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
    },
    "StringLike": {
      "aws:ResourceTag/karpenter.sh/nodepool": "*"
    }
  }
}
```

### KarpenterControllerIAMIntegrationPolicy

The `IAMIntegrationPolicy` manages IAM instance profile operations. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerIAMIntegrationPolicy-bob-karpenter-demo`

```yaml
IAMIntegrationPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerIAMIntegrationPolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

#### AllowPassingInstanceRole

The AllowPassingInstanceRole Sid gives the Karpenter controller permission to pass (`iam:PassRole`) the node role (`KarpenterNodeRole-${ClusterName}`) to generated instance profiles.
This gives EC2 permission explicit permission to use the `KarpenterNodeRole-${ClusterName}` when assigning permissions to generated instance profiles while launching nodes.

```json
{
  "Sid": "AllowPassingInstanceRole",
  "Effect": "Allow",
  "Resource": "${KarpenterNodeRole.Arn}",
  "Action": "iam:PassRole",
  "Condition": {
    "StringEquals": {
      "iam:PassedToService": [
        "ec2.amazonaws.com",
        "ec2.amazonaws.com.cn"
      ]
    }
  }
}
```

#### AllowScopedInstanceProfileCreationActions

The AllowScopedInstanceProfileCreationActions Sid gives the Karpenter controller permission to create a new instance profile with [`iam:CreateInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateInstanceProfile.html),
provided that the request is made to a cluster with `RequestTag` `kubernetes.io/cluster/${ClusterName}` set to `owned`, the `eks:eks-cluster-name` set to `${ClusterName}`, and `topology.kubernetes.io/region` set to the current region.
Also, `karpenter.k8s.aws/ec2nodeclass` must be set to some value. This ensures that Karpenter can generate instance profiles on your behalf based on roles specified in your `EC2NodeClasses` that you use to configure Karpenter.

```json
{
  "Sid": "AllowScopedInstanceProfileCreationActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
  "Action": [
    "iam:CreateInstanceProfile"
  ],
  "Condition": {
    "StringEquals": {
      "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
      "aws:RequestTag/topology.kubernetes.io/region": "${AWS::Region}"
    },
    "StringLike": {
      "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
    }
  }
}
```

#### AllowScopedInstanceProfileTagActions

The AllowScopedInstanceProfileTagActions Sid gives the Karpenter controller permission to tag an instance profile with [`iam:TagInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_TagInstanceProfile.html), provided that `ResourceTag` attributes `kubernetes.io/cluster/${ClusterName}` is set to `owned` and `topology.kubernetes.io/region` is set to the current region and `RequestTag` attributes `kubernetes.io/cluster/${ClusterName}` is set to `owned`, `eks:eks-cluster-name` is set to `${ClusterName}`, and `topology.kubernetes.io/region` is set to the current region.
Also, `ResourceTag/karpenter.k8s.aws/ec2nodeclass` and `RequestTag/karpenter.k8s.aws/ec2nodeclass` must be set to some value. This ensures that Karpenter is only able to act on instance profiles that it provisions for this cluster.

```json
{
  "Sid": "AllowScopedInstanceProfileTagActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
  "Action": [
    "iam:TagInstanceProfile"
  ],
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:ResourceTag/topology.kubernetes.io/region": "${AWS::Region}",
      "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
      "aws:RequestTag/topology.kubernetes.io/region": "${AWS::Region}"
    },
    "StringLike": {
      "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
      "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
    }
  }
}
```

#### AllowScopedInstanceProfileActions

The AllowScopedInstanceProfileActions Sid gives the Karpenter controller permission to perform [`iam:AddRoleToInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_AddRoleToInstanceProfile.html), [`iam:RemoveRoleFromInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_RemoveRoleFromInstanceProfile.html), and [`iam:DeleteInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_DeleteInstanceProfile.html) actions,
provided that the request is made to a cluster with `kubernetes.io/cluster/${ClusterName}` set to owned and is made in the current region.
Also, `karpenter.k8s.aws/ec2nodeclass` must be set to some value. This permission is further enforced by the `iam:PassRole` permission. If Karpenter attempts to add a role to an instance profile that it doesn't have `iam:PassRole` permission on, that call will fail. Therefore, if you configure Karpenter to use a new role through the `EC2NodeClass`, ensure that you also specify that role within your `iam:PassRole` permission.

```json
{
  "Sid": "AllowScopedInstanceProfileActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
  "Action": [
    "iam:AddRoleToInstanceProfile",
    "iam:RemoveRoleFromInstanceProfile",
    "iam:DeleteInstanceProfile"
  ],
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned",
      "aws:ResourceTag/topology.kubernetes.io/region": "${AWS::Region}"
    },
    "StringLike": {
      "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
    }
  }
}
```

### KarpenterControllerEKSIntegrationPolicy

The `EKSIntegrationPolicy` manages EKS cluster discovery. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerEKSIntegrationPolicy-bob-karpenter-demo`

```yaml
EKSIntegrationPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerEKSIntegrationPolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

#### AllowAPIServerEndpointDiscovery

You can optionally allow the Karpenter controller to discover the Kubernetes cluster's external API endpoint to enable EC2 nodes to successfully join the EKS cluster.

> **Note**: If you are not using an EKS control plane, you will have to specify this endpoint explicitly. See the description of the `aws.clusterEndpoint` setting in the [ConfigMap](.settings/#configmap) documentation for details.

The AllowAPIServerEndpointDiscovery Sid allows the Karpenter controller to get that information (`eks:DescribeCluster`) for the cluster (`cluster/${ClusterName}`).

```json
{
  "Sid": "AllowAPIServerEndpointDiscovery",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}",
  "Action": "eks:DescribeCluster"
}
```

### KarpenterControllerInterruptionPolicy

The `InterruptionPolicy` manages access to the SQS interruption queue. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerInterruptionPolicy-bob-karpenter-demo`

```yaml
InterruptionPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerInterruptionPolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

#### AllowInterruptionQueueActions

Karpenter supports interruption queues, that you can create as described in the [Interruption](/v1.14/concepts/disruption/index.md#interruption) section of the Disruption page.
This section of the cloudformation.yaml template can give Karpenter permission to access those queues by specifying the resource ARN.
For the interruption queue you created (`${KarpenterInterruptionQueue.Arn}`), the AllowInterruptionQueueActions Sid lets the Karpenter controller have permission to delete messages ([DeleteMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.html)), get queue URL ([GetQueueUrl](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueUrl.html)), and receive messages ([ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html)).

```json
{
  "Sid": "AllowInterruptionQueueActions",
  "Effect": "Allow",
  "Resource": "${KarpenterInterruptionQueue.Arn}",
  "Action": [
    "sqs:DeleteMessage",
    "sqs:GetQueueUrl",
    "sqs:ReceiveMessage"
  ]
}
```
### KarpenterControllerZonalShiftPolicy

The `ZonalShiftPolicy` manages access to Zonal Shift data for your cluster. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerZonalShiftPolicy-bob-karpenter-demo`

```yaml
ZonalShiftPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerZonalShiftPolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

### AllowZonalShiftStatusReadOnly

Karpenter supports AWS Application Recovery Controller Zonal Shift, that you can utilize as described in the [Zonal Shift](/v1.14/concepts/scheduling/index.md#zonal-shift) section of the Scheduling page.
This section of the cloudformation.yaml template can give Karpenter permission to query the Zonal Shift service about the status of the cluster by specifying the cluster ARN. 
For the cluster that will be created (`arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}`), the AllowZonalShiftActions Sid lets the Karpenter controller have permission to get the Zonal Shift status of the cluster ([GetManagedResource](https://docs.aws.amazon.com/arc-zonal-shift/latest/api/API_GetManagedResource.html)).

```json
{
  "Sid": "AllowZonalShiftStatusReadOnly",
  "Effect": "Allow",
  "Resource": "*",
  "Action": [
    "arc-zonal-shift:GetManagedResource"
  ],
  "Condition": {
    "StringEquals": {
      "arc-zonal-shift:ResourceIdentifier": "arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}"
    }
  }
}
```

### KarpenterControllerResourceDiscoveryPolicy

The `ResourceDiscoveryPolicy` manages read-only resource discovery operations. Given a cluster name of `bob-karpenter-demo`, this policy would be named: `KarpenterControllerResourceDiscoveryPolicy-bob-karpenter-demo`

```yaml
ResourceDiscoveryPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    ManagedPolicyName: !Sub "KarpenterControllerResourceDiscoveryPolicy-${ClusterName}"
    Path: /
    PolicyDocument: !Sub |
      {
        "Version": "2012-10-17",
        "Statement": [
```

#### AllowRegionalReadActions

The AllowRegionalReadActions Sid allows [DescribeCapacityReservations](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeCapacityReservations.html), [DescribeImages](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeImages.html), [DescribeInstances](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html), [DescribeInstanceStatus](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceStatus.html), [DescribeInstanceTypeOfferings](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypeOfferings.html), [DescribeInstanceTypes](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypes.html), [DescribeLaunchTemplates](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeLaunchTemplates.html), [DescribePlacementGroups](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribePlacementGroups.html), [DescribeSecurityGroups](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html), [DescribeSpotPriceHistory](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSpotPriceHistory.html), and [DescribeSubnets](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSubnets.html) actions for the current AWS region.
This allows the Karpenter controller to do any of those read-only actions across all related resources for that AWS region.

```json
{
  "Sid": "AllowRegionalReadActions",
  "Effect": "Allow",
  "Resource": "*",
  "Action": [
    "ec2:DescribeCapacityReservations",
    "ec2:DescribeImages",
    "ec2:DescribeInstances",
    "ec2:DescribeInstanceStatus",
    "ec2:DescribeInstanceTypeOfferings",
    "ec2:DescribeInstanceTypes",
    "ec2:DescribeLaunchTemplates",
    "ec2:DescribePlacementGroups",
    "ec2:DescribeSecurityGroups",
    "ec2:DescribeSpotPriceHistory",
    "ec2:DescribeSubnets"
  ],
  "Condition": {
    "StringEquals": {
      "aws:RequestedRegion": "${AWS::Region}"
    }
  }
}
```

#### AllowSSMReadActions

The AllowSSMReadActions Sid allows the Karpenter controller to get SSM parameters (`ssm:GetParameter`) from the current region for SSM parameters generated by AWS services.

**NOTE**: If potentially sensitive information is stored in SSM parameters, you could consider restricting access to these messages further.
```json
{
  "Sid": "AllowSSMReadActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:ssm:${AWS::Region}::parameter/aws/service/*",
  "Action": "ssm:GetParameter"
}
```

#### AllowPricingReadActions

Because pricing information does not exist in every region at the moment, the AllowPricingReadActions Sid allows the Karpenter controller to get product pricing information (`pricing:GetProducts`) for all related resources across all regions.

```json
{
  "Sid": "AllowPricingReadActions",
  "Effect": "Allow",
  "Resource": "*",
  "Action": "pricing:GetProducts"
}
```

#### AllowUnscopedInstanceProfileListAction

The AllowUnscopedInstanceProfileListAction Sid gives the Karpenter controller permission to perform [`iam:ListInstanceProfiles`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_ListInstanceProfiles.html) action to list instance profiles.

```json
{
  "Sid": "AllowUnscopedInstanceProfileListAction",
  "Effect": "Allow",
  "Resource": "*",
  "Action": "iam:ListInstanceProfiles"
}
```

#### AllowInstanceProfileReadActions

The AllowInstanceProfileReadActions Sid gives the Karpenter controller permission to perform [`iam:GetInstanceProfile`](https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetInstanceProfile.html) actions to retrieve information about a specified instance profile, including understanding if an instance profile has been provisioned for an `EC2NodeClass` or needs to be re-provisioned.

```json
{
  "Sid": "AllowInstanceProfileReadActions",
  "Effect": "Allow",
  "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
  "Action": "iam:GetInstanceProfile"
}
```

## Interruption Handling

Settings in this section allow the Karpenter controller to stand-up an interruption queue to receive notification messages from other AWS services about the health and status of instances. For example, this interruption queue allows Karpenter to be aware of spot instance interruptions that are sent 2 minutes before spot instances are reclaimed by EC2. Adding this queue allows Karpenter to be proactive in migrating workloads to new nodes.
See the [Interruption](/v1.14/concepts/disruption/index.md#interruption) section of the Disruption page for details.

Defining the `KarpenterInterruptionQueuePolicy` allows Karpenter to see and respond to the following:

* AWS health events
* Spot interruptions
* Spot rebalance recommendations
* Instance state changes

The resources defined in this section include:

* KarpenterInterruptionQueue
* KarpenterInterruptionQueuePolicy
* ScheduledChangeRule
* SpotInterruptionRule
* RebalanceRule
* InstanceStateChangeRule

### KarpenterInterruptionQueue

The [AWS::SQS::Queue](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sqs-queue.html) resource is used to create an Amazon SQS standard queue.
Properties of that resource set the `QueueName` to the name of your cluster, the time for which SQS retains each message (`MessageRetentionPeriod`) to 300 seconds, and enabling serverside-side encryption using SQS owned encryption keys (`SqsManagedSseEnabled`) to `true`.
See [SetQueueAttributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SetQueueAttributes.html) for descriptions of some of these attributes.

```yaml
KarpenterInterruptionQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: !Sub "${ClusterName}"
    MessageRetentionPeriod: 300
    SqsManagedSseEnabled: true
```

### KarpenterInterruptionQueuePolicy

The Karpenter interruption queue policy is created to allow AWS services that we want to receive instance notifications from to push notification messages to the queue.
The [AWS::SQS::QueuePolicy](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sqs-queuepolicy.html) resource here applies `EC2InterruptionPolicy` to the `KarpenterInterruptionQueue`. The policy allows [sqs:SendMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) actions to `events.amazonaws.com` and `sqs.amazonaws.com` services. It also allows the `GetAtt` function to get attributes from `KarpenterInterruptionQueue.Arn`.
Additionally, it only allows access to the queue using encrypted connections over HTTPS (TLS) to adhere to [Amazon SQS Security Best Practices](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-security-best-practices.html#enforce-encryption-data-in-transit).

```yaml
KarpenterInterruptionQueuePolicy:
  Type: AWS::SQS::QueuePolicy
  Properties:
    Queues:
      - !Ref KarpenterInterruptionQueue
    PolicyDocument:
      Id: EC2InterruptionPolicy
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - events.amazonaws.com
              - sqs.amazonaws.com
          Action: sqs:SendMessage
          Resource: !GetAtt KarpenterInterruptionQueue.Arn
        - Sid: DenyHTTP
          Effect: Deny
          Action: sqs:*
          Resource: !GetAtt KarpenterInterruptionQueue.Arn
          Condition:
            Bool:
              aws:SecureTransport: false
          Principal: "*"
```

### Rules

This section allows Karpenter to gather [AWS Health Events](https://docs.aws.amazon.com/health/latest/ug/cloudwatch-events-health.html#about-public-events) and direct them to a queue where they can be consumed by Karpenter.
These rules include:

* ScheduledChangeRule: The [AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) creates a rule where the [EventPattern](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html) is set to send events from the `aws.health` source to `KarpenterInterruptionQueue`.

  ```yaml
  ScheduledChangeRule:
    Type: 'AWS::Events::Rule'
    Properties:
     EventPattern:
       source:
         - aws.health
       detail-type:
         - AWS Health Event
     Targets:
       - Id: KarpenterInterruptionQueueTarget
         Arn: !GetAtt KarpenterInterruptionQueue.Arn
  ```

* SpotInterruptionRule: An EC2 Spot Instance Interruption warning tells you that AWS is about to reclaim a Spot instance you are using. This rule allows Karpenter to gather [EC2 Spot Instance Interruption Warning](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) events and direct them to a queue where they can be consumed by Karpenter. In particular, the [AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) here creates a rule where the [EventPattern](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html) is set to send events from the `aws.ec2` source to `KarpenterInterruptionQueue`.

  ```yaml
  SpotInterruptionRule:
    Type: 'AWS::Events::Rule'
    Properties:
      EventPattern:
        source:
          - aws.ec2
        detail-type:
          - EC2 Spot Instance Interruption Warning
      Targets:
        - Id: KarpenterInterruptionQueueTarget
          Arn: !GetAtt KarpenterInterruptionQueue.Arn
  ```

* RebalanceRule: An EC2 Instance Rebalance Recommendation signal tells you that a Spot instance is at a heightened risk of being interrupted, allowing Karpenter to get new instances or simply rebalance workloads.  This rule allows Karpenter to gather [EC2 Instance Rebalance Recommendation](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/rebalance-recommendations.html) signals and direct them to a queue where they can be consumed by Karpenter. In particular, the [AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) here creates a rule where the [EventPattern](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html) is set to send events from the `aws.ec2` source to `KarpenterInterruptionQueue`.

  ```yaml
  RebalanceRule:
   Type: 'AWS::Events::Rule'
   Properties:
     EventPattern:
       source:
         - aws.ec2
       detail-type:
         - EC2 Instance Rebalance Recommendation
     Targets:
       - Id: KarpenterInterruptionQueueTarget
         Arn: !GetAtt KarpenterInterruptionQueue.Arn
  ```

* InstanceStateChangeRule: An EC2 Instance State-change Notification signal tells you that the state of an instance has changed to one of the following states: pending, running, stopping, stopped, shutting-down, or terminated. This rule allows Karpenter to gather [EC2 Instance State-change](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instance-state-changes.html) signals and direct them to a queue where they can be consumed by Karpenter. In particular, the [AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) here creates a rule where the [EventPattern](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html) is set to send events from the `aws.ec2` source to `KarpenterInterruptionQueue`.

  ```yaml
  InstanceStateChangeRule:
   Type: 'AWS::Events::Rule'
   Properties:
     EventPattern:
       source:
         - aws.ec2
       detail-type:
         - EC2 Instance State-change Notification
     Targets:
       - Id: KarpenterInterruptionQueueTarget
         Arn: !GetAtt KarpenterInterruptionQueue.Arn
  ```

* CapacityReservationInterruptionRule: An EC2 Capacity Reservation Interruption warning informs users that the capacity owner within an AWS Organization is reclaiming reserved capacity, triggering a notification for node reclamation. The [AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) here creates a rule to send events from the `aws.ec2` source to `KarpenterInterruptionQueue`.

  ```yaml
  CapacityReservationInterruptionRule:
   Type: 'AWS::Events::Rule'
   Properties:
     EventPattern:
       source:
         - aws.ec2
       detail-type:
         - EC2 Capacity Reservation Instance Interruption Warning
     Targets:
       - Id: KarpenterInterruptionQueueTarget
         Arn: !GetAtt KarpenterInterruptionQueue.Arn
  ```