Development Guide

Dependencies

The following tools are required for contributing to the Karpenter project.

Developing

Setup / Teardown

Based on how you are running your Kubernetes cluster, follow the Environment specific setup to configure your environment before you continue. You can choose to either run the Karpenter controller locally on your machine, pointing to the Kubernetes cluster specified in your ~/.kube/config or inside the Kubernetes cluster specified in your ~/.kube/config deployed with Helm.

Locally

Once you have your environment set up, run the following commands to run the Karpenter Go binary against the Kubernetes cluster specified in your ~/.kube/config

make run

Inside a Kubernetes Cluster

Once you have your environment set up, to install Karpenter in the Kubernetes cluster specified in your ~/.kube/config run the following commands.

make apply # Install Karpenter
make delete # Uninstall Karpenter

Developer Loop

• Make sure dependencies are installed
  • Run make codegen to make sure yaml manifests are generated (requires a working set of AWS credentials, see Specifying Credentials)
  • Run make toolchain to install cli tools for building and testing the project
• You will need a personal development image repository (e.g. ECR)
  • Make sure you have valid credentials to your development repository.
  • $KO_DOCKER_REPO must point to your development repository
  • Your cluster must have permissions to read from the repository

Build and Deploy

Note: these commands do not rely on each other and may be executed independently

make apply # quickly deploy changes to your cluster
make presubmit # run codegen, lint, and tests

If you are only interested in building the Karpenter images and not deploying the updated release to your cluster immediately with Helm, you can run the following:

make image # build and push the karpenter images

*Note: that this will produce a build with the version of https://github.com/kubernetes-sigs/karpenter in your local filesystem.

You can test out changes made in https://github.com/kubernetes-sigs/karpenter by replacing the dependency of https://github.com/aws/karpenter-provider-aws/. For local changes, replace $PATH_TO_KUBERNETES_SIGS_KARPENTER with the relative or absolute path and run:

go mod edit -replace sigs.k8s.io/karpenter=$PATH_TO_KUBERNETES_SIGS_KARPENTER

Then you can build your image using the previous steps.

Publishing Images Only

If you only need to build and publish an image to a container registry, run the following:

make image # build and push the karpenter images

You can test out changes made in https://github.com/kubernetes-sigs/karpenter by replacing the dependency of https://github.com/aws/karpenter-provider-aws/. For local changes, replace $PATH_TO_KUBERNETES_SIGS_KARPENTER with the relative or absolute path and run:

go mod edit -replace sigs.k8s.io/karpenter=$PATH_TO_KUBERNETES_SIGS_KARPENTER

Then you can build your image using the previous steps.

*Note: you need to commit the go.mod changes before running make image for the changes to be picked up.

Testing

make test       # E2E correctness tests

Change Log Level

By default, make apply will set the log level to debug. You can change the log level by setting the log level in your Helm values.

--set logLevel=debug

Debugging Metrics

OSX:

open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n kube-system 8080

Linux:

gio open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080

Tailing Logs

While you can tail Karpenter’s logs with kubectl, there’s a number of tools out there that enhance the experience. We recommend Stern:

stern -n karpenter -l app.kubernetes.io/name=karpenter

Environment specific setup

AWS

For local development on Karpenter you will need a Docker repo which can manage your images for Karpenter components. You can use the following command to provision an ECR repository. We recommend using a single “dev” repository for development across multiple projects, and to use specific image hashes instead of image tags.

aws ecr create-repository \
    --repository-name dev \
    --image-scanning-configuration scanOnPush=true \
    --region "${AWS_DEFAULT_REGION}"

Once you have your ECR repository provisioned, configure your Docker daemon to authenticate with your newly created repository.

export KO_DOCKER_REPO="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/dev"
aws ecr get-login-password --region "${AWS_DEFAULT_REGION}" | docker login --username AWS --password-stdin "${KO_DOCKER_REPO}"

Finally, to deploy the correct IAM permissions, including the instance profile for provisioned nodes, run

make setup

Profiling memory

Karpenter exposes a pprof endpoint on its metrics port.

Learn about profiling with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

Prerequisites

brew install graphviz
go install github.com/google/pprof@latest

Get a profile

# Connect to the metrics endpoint
kubectl port-forward service/karpenter -n karpenter 8080
open http://localhost:8080/debug/pprof/
# Visualize the memory
go tool pprof -http 0.0.0.0:9000 localhost:8080/debug/pprof/heap