Ahmed AbouZaid!: Helm

Software supply chain security has been one of the hot topics because of the continues attacks and exploits shown in the latest few years. So having a proper security practices like signing artifacts in today's CI/CD pipeline is not a luxury! Which became a standard DevSecOps practice.

However, for a long time it needed a lot of work to implement those practices, hence, projects like Open Source Security Foundation (OpenSSF) which created security framework like Supply-chain Levels for Software Artifacts (SLSA) and Sigstore which created tools like Cosign to standardize and reduce that fatigue.

Today I want to shed some lights on Helm chart signing with Cosign (part of Sigstore project). This post will focus on GitHub Actions and 2 ways for Helm chart signing based on the type of the Helm registry (simple or OCI-based).

1. Intro
2. Identity setup
3.1. Keyless signing for simple Helm repository
3.2. Keyless signing for OCI-based Helm repository
4. Recap

1. Intro

Before starting, what is actually Cosign? I will start with quoting from the project website:

Cosign is a command line utility that can sign and verify software artifact, such as container images and blobs.

Cosign aims to make signatures invisible infrastructure, and one of its main features what's known as "Keyless signing". Keyless signing means rather than using keys like GPG/PGP, it uses associates identities via OpenID Connect (i.e. it auth against providers like Microsoft, Google, and GitHub to issues short-lived certificates binding an ephemeral key).

Cosign could be used as a CLI also integrated part of other build/release tools like GoReleaser.

2. Identity setup

As mentioned, no need for signing keys here, Cosign will use the identity context like GitHub. Let's take GitHub Actions as a base here, but it is worth mentioning that Cosign works with many identity providers.

From the GitHub Actions workflow point of view, you just need to have job permission id-token: write, which allows the job to use OICD and generate JWT (which works as a key).

name: Sign Helm chart artifact
[...]
jobs:
  sign:
    name: Sign
    permissions:
      id-token: write
    [...]

You can find a full example in the repo: https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example

3.1. Keyless signing for simple Helm repository

A simple Helm repository is just a file called index.yaml which reference to the actual chart files URLs. A popular way for that is using Chart Releaser to host the Helm charts using GitHub Pages and Releases.

The simplest way to sign Helm chart using Cosign is to sign the artifact then upload the signature to GitHub release page.

First, sign the chart file:

# For explicitly, it's also possible to add '--oidc-provider=github-actions',
# but no need for that, cosign will discover the context if the GH job permission is correct.
cosign sign-blob my-app-1.0.0.tgz --bundle my-app-1.0.0.tgz.cosign.bundle

That will create a bundle file my-app-1.0.0.tgz.cosign.bundle which contains signing metadata like the signature and certificate (also it's possible to have separate sig and pem files) which should be uploaded to GitHub release page.

Now anyone can download the Helm chart file and Cosign bundle file to verify its integrity:

cosign verify-blob my-app-1.0.0.tgz \
  --bundle my-app-1.0.0.tgz.cosign.bundle \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"

If the file is valid the command will show Verified OK, otherwise it will show something like Error: none of the expected identities matched what was in the certificate ....

On the other hand, that's still some work! Maybe if Chart Releaser has a native support for Cosign. Also, the Helm plugin helm-sigstore could help a bit but you need to download the Helm plugin first.

3.2. Keyless signing for OCI-based Helm repository

Using container image to store config like Helm chart or Terraform module is one of the brilliant ideas.

One of the biggest changes in Helm 3 was ability to use container registries with OCI support to store and share chart packages. It started as an experimental feature, but by Helm v3.8.0, OCI support is enabled by default.

So instead that simple index.yaml an OCI-based registry could be used as a Helm repository and chart storage too. Any hosted registries that support OCI will work for that like Docker Hub, Amazon ECR, Azure Container Registry, Google Artifact Registry, etc.

In that setup, Cosign works a bit differently where it signs the chart as an OCI image (the same way it signs the Docker images) and store the signature in the OCI repository. But in that case it only makes sense to sign the digest not the tags since the digest is immutable.

There are two steps to make that, first push the chart to an OCI-based Helm repository which generates the chart digest, then push the signed digest.

# After login to the registry using "helm registry login ...".
helm push my-app-1.0.0.tgz oci://ttl.sh/charts &> push-metadata.txt

CHART_DIGEST=$(awk '/Digest: /{print $2}' push-metadata.txt)
cosign sign -y "ttl.sh/charts/my-app@${CHART_DIGEST}"

Finally, to verify that:

cosign verify "ttl.sh/charts/my-app@${CHART_DIGEST}" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"

4. Recap

Nowadays, it's critical to have standard DevSecOps practices in the whole SDLC, and validating integrity is one of the most essential practices in securing the software supply chain.

Any publicly distributed artifact should be signed to ensure the origin of it, and Helm charts are no different. Cosign keyless signing made it much easier to apply such practices without the hassles of managing singing keys.