12/12/2022

Pass extra data to the Containerized KRM function - Kustomize

A couple of months ago, I wrote some notes about KRM Functions in Kustomize. Today in this blog post, I show why I love and hate the current state of KRM functions 😂️

ToC

Overview

I admired the idea of Containerized KRM function that's because you don't need to deal with managing and downloading the Kustomize plugins. It was super annoying to manage plugins across multiple operating systems (e.g., for different team members).

As mentioned before, the KRM manifest is passed to the container directly as STDIN, but if you want to pass extra files, you need to mount the files into the KRM function container, which is fine. However, there are some limitations, like you can only mount files relative to the base instead of overlay!

Let's take a look at different cases working with containerized KRM functions.

Prerequisites

This post covers the above intermediate topic in Kustomize, so it assumes that you understand how Kustomize works and the structure of Kustomization File.

Examples

Level 1: Pass no data

In this case, you don't actually need to pass any "extra" data to the containerized KRM function. All data required by the function are in the manifest files.

Here is a dummy example since it's not the main topic here (I just copied it from the Kustomize docs):

apiVersion: someteam.example.com/v1
kind: ChartInflator
metadata:
  name: notImportantHere
  annotations:
    config.kubernetes.io/function: |
      container:
        image: example.docker.com/my-functions/chart-inflator:0.1.6
spec:
  chartName: minecraft

Note: If you want to use Helm charts within Kustomize files, Kustomize support that out of the box now via HelmChartInflationGenerator, which is a built-in plugin.

Level 2: Pass simple data types

Now let's look at a real example where you need to pass extra files to the containerized KRM function.

Here is an example of the Kubeval plugin:

apiVersion: examples.config.kubernetes.io/v1beta1
kind: Kubeval
metadata:
  name: my-validator
  annotations:
    config.kubernetes.io/function: |
      container:
        image: validator-kubeval:latest
        mounts:
        - type: bind
          src: ./schemas
          dst: /my_schemas
spec:
  strict: true
  schemaLocation: "file:///my_schemas"

In this case, it's straightforward, it will mount the local dir ./schemas on container dir /my_schemas, and the plugin inside the container will use it.

However, there is a limitation here, the mount paths must be under the current kustomization directory, and it's not permitted to use a src from the parent directory, which is usually used when you have some common values shared across multiple Kustomizations.

So if you try to use something like this src: ../schemas, you will see an error like this:

Error: plugin Kubeval.v1beta1.examples.config.kubernetes.io/validate.[noNs] with mount path '../my_schemas' is not permitted; mount paths must be under the current kustomization directory

Level 3: Pass complex data types

If you want to pass more extra data it will be a bit complex. To be more accurate, the problem here is not more or complex data, but the plugin itself should work a bit differently if more data types are needed for the plugin to work correctly.

Let's take SopsSecretGenerator plugin as an example, which generates Kubernetes Secrets from sops-encrypted files. I like this plugin because you don't need to download SOPS binary; the plugin includes SOPS libs as part of it.

The SopsSecretGenerator plugin needs different files to work, but mainly it needs a GPG key and sops-encrypted files. The plugin itself supports KRM (I've added it in the PR #32). However, its container doesn't support running containerized KRM functions because there is no mechanism to import the GPG key into the container keyring.

Of course, that should be part of the upstream, but given that KRM support is still super young and not many plugins support KRM in the first place. But here, an example shows how much work is needed to make a plugin works as a containerized KRM function.

As I mentioned, the current container image of SopsSecretGenerator plugin doesn't support importing the GPG key, so we need to build a new image that's able to import the GPG key.

# syntax=docker/dockerfile:1

ARG SSG_UPSTREAM_VERSION=v1.6

FROM goabout/kustomize-sopssecretgenerator:$SSG_UPSTREAM_VERSION as ssg

FROM alpine

RUN apk add --no-cache gnupg

COPY --from=ssg /SopsSecretGenerator /SopsSecretGenerator

ENV GPG_KEYS_DIR=/mnt/gpg/keys

COPY <<-EOT /docker-entrypoint.sh
  # Create a writable new home dir because the KRM container works as "nobody" user with home "/".
  mkdir -p /tmp/user
  export HOME=/tmp/user
  gpg --quiet --import $GPG_KEYS_DIR/*
    /SopsSecretGenerator $@
EOT

ENTRYPOINT ["sh", "/docker-entrypoint.sh"]

Now let's build it using Docker BuildKit:

DOCKER_BUILDKIT=1 docker build -t sopssecretgenerator-custom .

As mentioned before, you cannot use bind type to mount files outside the kustomization directory. So the only option here is to copy them to a named volume and reference the volume name. To do so, and since Docker doesn't provide a way to copy data to volume directly, I spin a temporary container and use it to copy the GPG key to it.

docker run --rm \
    -v <PATH_TO_LOCAL_GPG_KYES_DIR>:/mnt \
    -v my-gpg-key-volume:/data busybox \
    cp -r /mnt /data

And here is the final manifest with the local volume for the GPG file and bind of current sops-encrypted secrets:

apiVersion: goabout.com/v1beta1
kind: SopsSecretGenerator
metadata:
  name: my-secret
  annotations:
    config.kubernetes.io/function: |
      container:
        image: sopssecretgenerator-custom
        mounts:
        - type: volume
          src: my-gpg-key-volume
          dst: /mnt/gpg
        - type: bind
          src: ./sops-encrypted-secrets
          dst: /mnt/sops-encrypted-secrets
envs:
- /mnt/sops-encrypted-secrets/my-secret.env

Conclusion

As you see, containerized KRM functions provide a super flexible way to extend Kustomize. However, particular limitations still make it hard to use in some cases. So, in addition to containerized KRM functions, it is inevitable also to use Exec KRM functions

Given that KRM functions in Kustomize are still alpha, I believe many things will improve over time, and I can't wait to see it stable!

---

That's it! Enjoy :-)

Powered by Blogger.

Hello, my name is Ahmed AbouZaid, I'm a passionate Tech Lead DevOps Engineer. 👋

With 16+ years of open-source contributions, 12+ years of hands-on experience in DevOps, and an M.Sc. in Data Engineering from Edinburgh Napier University (UK), I enjoy facilitating the growth of both businesses and individuals.

I specialize in Cloud-Native and Kubernetes. I'm also a Free/Open source geek and book author. My favorite topics are DevOps transformation, automation, data, and metrics.

Contact Me

Name

Email *

Message *

Start Your DevOps Engineer Journey!

Start Your DevOps Engineer Journey!
Start your DevOps career for free the Agile way in 2024 with the Dynamic DevOps Roadmap ⭐

Latest Post

2023 Highlights

Image generated with Craiyon . Finally, 2023 is over! What a year! One more crazy year, but it was the culmination ...

Popular Posts

Blog Archive