A couple of months ago, I wrote some notes about KRM Functions in Kustomize. Today in this blog post, I show why I love and hate the current state of KRM functions 😂️
ToC
Overview
I admired the idea of Containerized KRM function that's because you don't need to deal with managing and downloading the Kustomize plugins. It was super annoying to manage plugins across multiple operating systems (e.g., for different team members).
As mentioned before, the KRM manifest is passed to the container directly as STDIN, but if you want to pass extra files, you need to mount the files into the KRM function container, which is fine. However, there are some limitations, like you can only mount files relative to the base instead of overlay!
Let's take a look at different cases working with containerized KRM functions.
Prerequisites
This post covers the above intermediate topic in Kustomize, so it assumes that you understand how Kustomize works and the structure of Kustomization File.
Examples
Level 1: Pass no data
In this case, you don't actually need to pass any "extra" data to the containerized KRM function. All data required by the function are in the manifest files.
Here is a dummy example since it's not the main topic here (I just copied it from the Kustomize docs):
apiVersion: someteam.example.com/v1
kind: ChartInflator
metadata:
name: notImportantHere
annotations:
config.kubernetes.io/function: |
container:
image: example.docker.com/my-functions/chart-inflator:0.1.6
spec:
chartName: minecraft
Note: If you want to use Helm charts within Kustomize files, Kustomize support that out of the box now via HelmChartInflationGenerator, which is a built-in plugin.
Level 2: Pass simple data types
Now let's look at a real example where you need to pass extra files to the containerized KRM function.
Here is an example of the Kubeval plugin:
apiVersion: examples.config.kubernetes.io/v1beta1
kind: Kubeval
metadata:
name: my-validator
annotations:
config.kubernetes.io/function: |
container:
image: validator-kubeval:latest
mounts:
- type: bind
src: ./schemas
dst: /my_schemas
spec:
strict: true
schemaLocation: "file:///my_schemas"
In this case, it's straightforward, it will mount the local dir ./schemas on container dir /my_schemas, and the plugin inside the container will use it.
However, there is a limitation here, the mount paths must be under the current kustomization directory, and it's not permitted to use a src from the parent directory, which is usually used when you have some common values shared across multiple Kustomizations.
So if you try to use something like this src: ../schemas, you will see an error like this:
Error: plugin Kubeval.v1beta1.examples.config.kubernetes.io/validate.[noNs] with mount path '../my_schemas' is not permitted; mount paths must be under the current kustomization directory
Level 3: Pass complex data types
If you want to pass more extra data it will be a bit complex. To be more accurate, the problem here is not more or complex data, but the plugin itself should work a bit differently if more data types are needed for the plugin to work correctly.
Let's take SopsSecretGenerator plugin as an example, which generates Kubernetes Secrets from sops-encrypted files. I like this plugin because you don't need to download SOPS binary; the plugin includes SOPS libs as part of it.
The SopsSecretGenerator plugin needs different files to work, but mainly it needs a GPG key and sops-encrypted files. The plugin itself supports KRM (I've added it in the PR #32). However, its container doesn't support running containerized KRM functions because there is no mechanism to import the GPG key into the container keyring.
Of course, that should be part of the upstream, but given that KRM support is still super young and not many plugins support KRM in the first place. But here, an example shows how much work is needed to make a plugin works as a containerized KRM function.
As I mentioned, the current container image of SopsSecretGenerator plugin doesn't support importing the GPG key, so we need to build a new image that's able to import the GPG key.
# syntax=docker/dockerfile:1
ARG SSG_UPSTREAM_VERSION=v1.6
FROM goabout/kustomize-sopssecretgenerator:$SSG_UPSTREAM_VERSION as ssg
FROM alpine
RUN apk add --no-cache gnupg
COPY --from=ssg /SopsSecretGenerator /SopsSecretGenerator
ENV GPG_KEYS_DIR=/mnt/gpg/keys
COPY <<-EOT /docker-entrypoint.sh
# Create a writable new home dir because the KRM container works as "nobody" user with home "/".
mkdir -p /tmp/user
export HOME=/tmp/user
gpg --quiet --import $GPG_KEYS_DIR/*
/SopsSecretGenerator $@
EOT
ENTRYPOINT ["sh", "/docker-entrypoint.sh"]
Now let's build it using Docker BuildKit:
DOCKER_BUILDKIT=1 docker build -t sopssecretgenerator-custom .
As mentioned before, you cannot use bind type to mount files outside the kustomization directory. So the only option here is to copy them to a named volume and reference the volume name. To do so, and since Docker doesn't provide a way to copy data to volume directly, I spin a temporary container and use it to copy the GPG key to it.
docker run --rm \
-v <PATH_TO_LOCAL_GPG_KYES_DIR>:/mnt \
-v my-gpg-key-volume:/data busybox \
cp -r /mnt /data
And here is the final manifest with the local volume for the GPG file and bind of current sops-encrypted secrets:
apiVersion: goabout.com/v1beta1
kind: SopsSecretGenerator
metadata:
name: my-secret
annotations:
config.kubernetes.io/function: |
container:
image: sopssecretgenerator-custom
mounts:
- type: volume
src: my-gpg-key-volume
dst: /mnt/gpg
- type: bind
src: ./sops-encrypted-secrets
dst: /mnt/sops-encrypted-secrets
envs:
- /mnt/sops-encrypted-secrets/my-secret.env
Conclusion
As you see, containerized KRM functions provide a super flexible way to extend Kustomize. However, particular limitations still make it hard to use in some cases. So, in addition to containerized KRM functions, it is inevitable also to use Exec KRM functions
Given that KRM functions in Kustomize are still alpha, I believe many things will improve over time, and I can't wait to see it stable!
---
That's it! Enjoy :-)
