Software supply chain security has been one of the hot topics because
of the continues attacks and exploits shown in the latest few years. So
having a proper security practices like signing artifacts in today's
CI/CD pipeline is not a luxury! Which became a standard DevSecOps
practice.
However, for a long time it needed a lot of work to implement those
practices, hence, projects like Open
Source Security Foundation (OpenSSF) which created security
framework like Supply-chain Levels for
Software Artifacts (SLSA) and Sigstore which created tools like
Cosign to
standardize and reduce that fatigue.
Today I want to shed some lights on Helm chart signing with Cosign
(part of Sigstore project). This post will focus on GitHub Actions and 2
ways for Helm chart signing based on the type of the Helm registry
(simple or OCI-based).
1. Intro
Before starting, what is actually Cosign? I will start with
quoting from the project website:
Cosign is a command line utility that can sign and verify software
artifact, such as container images and blobs.
Cosign aims to make signatures invisible
infrastructure, and one of its main features what's known as
"Keyless signing". Keyless signing means rather than using keys like
GPG/PGP, it uses associates identities via OpenID Connect (i.e. it auth
against providers like Microsoft, Google, and GitHub to issues
short-lived certificates binding an ephemeral key).
Cosign could be used as a CLI also integrated part of other
build/release tools like GoReleaser.
2. Identity setup
As mentioned, no need for signing keys here, Cosign will use the
identity context like GitHub. Let's take GitHub Actions as a base here,
but it is worth mentioning that Cosign works with many identity
providers.
From the GitHub Actions workflow point of view, you just need to have
job permission id-token: write, which allows the job to use
OICD and generate JWT (which works as a key).
name: Sign Helm chart artifact
[...]
jobs:
sign:
name: Sign
permissions:
id-token: write
[...]
You can find a full example in the repo: https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example
3.1. Keyless
signing for simple Helm repository
A simple Helm repository is just a file called
index.yaml which reference to the actual chart files URLs.
A popular way for that is using Chart Releaser to host
the Helm charts using GitHub Pages and Releases.
The simplest way to sign Helm chart using Cosign is to sign the
artifact then upload the signature to GitHub release page.
First, sign the chart file:
# For explicitly, it's also possible to add '--oidc-provider=github-actions',
# but no need for that, cosign will discover the context if the GH job permission is correct.
cosign sign-blob my-app-1.0.0.tgz --bundle my-app-1.0.0.tgz.cosign.bundle
That will create a bundle file
my-app-1.0.0.tgz.cosign.bundle which contains signing
metadata like the signature and certificate (also it's possible to have
separate sig and pem files) which should be
uploaded to GitHub release page.
Now anyone can download the Helm chart file and Cosign bundle file to
verify its integrity:
cosign verify-blob my-app-1.0.0.tgz \
--bundle my-app-1.0.0.tgz.cosign.bundle \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"
If the file is valid the command will show Verified OK,
otherwise it will show something like Error: none of the expected
identities matched what was in the certificate ....
On the other hand, that's still some work! Maybe if Chart Releaser has a
native support for Cosign. Also, the Helm plugin helm-sigstore could
help a bit but you need to download the Helm plugin first.
3.2. Keyless
signing for OCI-based Helm repository
Using container image to store config like Helm chart or Terraform
module is one of the brilliant ideas.
One of the biggest changes in Helm 3 was ability to use container
registries with OCI support to store and share chart packages. It
started as an experimental feature, but by Helm v3.8.0, OCI support is
enabled by default.
So instead that simple index.yaml an OCI-based registry
could be used as a Helm repository and chart storage too. Any hosted
registries that support OCI will work for that like Docker Hub, Amazon
ECR, Azure Container Registry, Google Artifact Registry, etc.
In that setup, Cosign works a bit differently where it signs the
chart as an OCI image (the same way it signs the Docker images) and
store the signature in the OCI repository. But in that case it only
makes sense to sign the digest not the tags since the digest is
immutable.
There are two steps to make that, first push the chart to an
OCI-based Helm repository which generates the chart digest, then push
the signed digest.
# After login to the registry using "helm registry login ...".
helm push my-app-1.0.0.tgz oci://ttl.sh/charts &> push-metadata.txt
CHART_DIGEST=$(awk '/Digest: /{print $2}' push-metadata.txt)
cosign sign -y "ttl.sh/charts/my-app@${CHART_DIGEST}"
Finally, to verify that:
cosign verify "ttl.sh/charts/my-app@${CHART_DIGEST}" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"
4. Recap
Nowadays, it's critical to have standard DevSecOps practices in the
whole SDLC, and validating integrity is one of the most essential
practices in securing the software supply chain.
Any publicly distributed artifact should be signed to ensure the
origin of it, and Helm charts are no different. Cosign keyless signing
made it much easier to apply such practices without the hassles of
managing singing keys.
