3 ways to customize off-the-shelf Helm charts with Kustomize - Kubernetes

If you are working with Kubernetes probably you know Helm and Kustomize. Both are used to manage Kubernetes' deployments and resources. Each one of them has its own philosophy and a different way of working.

Helm uses a template approach where a chart contains templates with placeholders that are replaced when it's rendered. On the other hand, Kustomize uses a template-free approach where it patches and merges YAML files. Also, it's been natively built into kubectl.

If you ask me, I'm in love with Kustomize ❤️
For me, the Kustomize style feels more natural to manage Kubernetes IaC (Infrastructure-as-Code). Probably that's because of my previous experience with configuration management tools like Ansible and SaltStack!

At the end of the day, both Helm and Kustomize are widely used in the Kubernetes ecosystem and probably you will use both of them. For a long time, Helm and Kustomize considered alternatives, but in reality, they work better together. In the next sections, we will explore 3 different solutions for the same challenge.

To fork or not to fork, that is the question!

As you may already know, many projects provide Helm charts for their applications and the template vars are controlled by the values file. That means you can only change certain predefined parts.

So the question has always been: How to customize upstream Helm charts (aka Off-The-Shelf charts)?
For example, adding or removing a K8s manifest to/from a Helm chart. Of course, if the change is generic, then contributing to upstream is the best option. But what if it's a custom change?

Usually forking is the common option! You fork the chart and do your extra changes on top of the chart in your repo. But doing that, again and again, would be a cumbersome, and error-prone task. Especially when chart needs to be updated or what's known as "last mile modification" to make generic charts fit your environment.

Here are 3 different ways to customize off-the-shelf Helm charts with Kustomize without forking. The main idea in all those approaches is that using Kustomize to do changes on the upstream Helm chart, but the main difference is how that is done.

Notes:

Here I will only cover direct workflows and will not go into some advanced workflows using Ship or Kots.
Versions used in the examples: Helm 3.3.1, Kustomize 3.8.1 and 4.1.0.

Method 1: Kustomize plugins

Kustomize offers a nice plugin ecosystem that allows extending Kustomize functionality. Working with Helm could be done via Kustomize built-in or external plugins

1.1 Kustomize built-in plugin

This is my favorite so far! By Kustomize v4.1.0, a built-in support for a Helm generator has been added. So Kustomize can use the helm command to inflate charts as a resource generator (the Helm CLI is still needed).

The top-level helmCharts specification was introduced first, but it had some limitations. Later on (I think around v4.1.3), the built-in plugin added HelmChartInflationGenerator which is IMO the best way to work with Helm charts in Kustomize!

Here is and example:

# helm-chart.yaml
apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: my-map
name: minecraft
repo: https://kubernetes-charts.storage.googleapis.com
version: v1.2.0
releaseName: test
namespace: testNamespace
valuesFile: values.yaml
IncludeCRDs: true

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
- helm-chart.yaml

That's it! Let's render the chart:

kustomize build --enable-helm .

You will see the Helm chart resources rendered and you actually can customize them using Kustomize as normal!

1.2 Kustomize external plugin

In the older versions of Kustomize (> v4.1.0), there is a non built-in plugin called ChartInflator which allows Kustomize to render Helm charts and apply any changes needed on the fly (it requires Helm binary to be installed). So let's start.

First install the plugin:

chartinflator_dir="$HOME/.config/kustomize/plugin/kustomize.config.k8s.io/v1/chartinflator"

chartinflator_bin="https://raw.githubusercontent.com/kubernetes-sigs/kustomize/kustomize/v3.8.1/plugin/someteam.example.com/v1/chartinflator/ChartInflator"

# Create the plugin dir.
mkdir -p ${chartinflator_dir}

# Download the plugin.
curl -L ${chartinflator_bin} > ${chartinflator_dir}/ChartInflator

# Set plugin the execute permission.
chmod u+x ${chartinflator_dir}/ChartInflator

Now we create the ChartInflator manifest and Helm values.yaml for Vault Helm Chart:

# ChartInflator manifest.
cat << EOF >> chartinflator-vault.yaml
apiVersion: kustomize.config.k8s.io/v1
kind: ChartInflator
metadata:
  name: vault-official-helm-chart
chartRepo: https://helm.releases.hashicorp.com
chartName: vault
chartRelease: hashicorp
chartVersion: 0.7.0
releaseName: vault
values: values.yaml
EOF

# Create values file for Helm chart.
helm repo add hashicorp https://helm.releases.hashicorp.com
helm show values --version 0.7.0 hashicorp/vault > values.yaml

# Create Kustomize file.
kustomize init
cat << EOF >> kustomization.yaml
generators:
- chartinflator-vault.yaml
EOF

# Add a common label for all resources.
kustomize edit add label env:dev

Now everything is ready. Let's render the chart:

kustomize build --enable_alpha_plugins .

As you see, the label has been applied to all resources in the chart which is done on-the-fly with no need to maintain any extra files.

However, there are 2 downsides of this approach.

Security, you should trust upstream 100% because any change in the upstream will end up at your side. But it's not just about trust! If the upstream has been hacked, you will get malicious code also!
Debugging, it's a bit hard to apply advanced customizations like patches. Especially when there are external changes. For example, in a recent change in kustomize 3.8.0, the order of output has been changed and all patchesJson6902 messed up!

Apart from that, I'm pretty happy with this way.

Finally, it worth mentioning that there is a community plugin called ChartRenderer which is written in Golang and does the same thing as "ChartInflator".

ChartRenderer provides some improvements like excluding some resources from the upstream chart, embedded values, and no need for Helm binary. However, the project status is not clear, and if it's still maintained.

Method 2: Kustomize with all-in-one chart file

Another way to customize OTS charts with Kustomize is using helm template to generate an all-in-one manifest of the chart. This way gives more control over the chart but it needs more work to handle updating and generating the all-in-one file from version to another.

Usually, I used Make as a helper for that case. So here is a quick demo to install the same Vault Helm chart.

# Makefile
CHART_REPO_NAME   := hashicorp
CHART_REPO_URL    := https://helm.releases.hashicorp.com
CHART_NAME        := vault
CHART_VERSION     := 0.7.0
CHART_VALUES_FILE := values.yaml

add-chart-repo:
    helm repo add ${CHART_REPO_NAME} ${CHART_REPO_URL}
    helm repo update

generate-chart-manifest: add-chart-repo
    helm template ${CHART_NAME} ${CHART_REPO_NAME}/${CHART_NAME} \
        --version ${CHART_VERSION} \
        --values ${CHART_VALUES_FILE} > ${CHART_NAME}.yaml

get-chart-values:
    @helm show values --version ${CHART_VERSION} \
    ${CHART_REPO_NAME}/${CHART_NAME}

generate-chart-values:
    @echo "Create values file: ${CHART_VALUES_FILE}"
    @$(MAKE) -s get-chart-values > ${CHART_VALUES_FILE}

diff-chart-values:
    @echo "Diff: Local <==> Remote"
    @$(MAKE) -s get-chart-values | \
    diff --suppress-common-lines --side-by-side ${CHART_VALUES_FILE} - || \
    exit 0

So to customize the upstream Vault Helm chart, we do the following:

# Init chart files.
make generate-chart-manifest generate-chart-values

# Create Kustomize file and add a common label for all resources.
kustomize init
kustomize edit add resource vault.yaml
kustomize edit add label env:dev

The same as the previous example, let's render the chart:

kustomize build .

As you see, you gain more control over the OTS chart, but at the same time, it needs more effort to update. Unlike the first method, where you just change the chart version and Kustomize will take care of the rest.

In this method you need to run the make command somehow/somewhere to generate updated all-in-one chart manifest. Also it could be a bit hard to integrate the update process with your GitOps workflow.

Method 3: Helm post rendering

Post Rendering was one of the new features that came with Helm 3. In the previous 2 methods, Kustomize was the main tool used to handle generating chart manifest, but here Kustomize works as a helper for Helm.

In other words, Helm controls the flow and that's helpful if Helm is your main deployment tool. Thus; you get all benefits of Helm like delete and rollback releases and so on.

So let's create the structure for that method:

# Create Kustomize file and add a common label for all resources.
kustomize init
kustomize edit add label env:dev

# Create Kustomize wrapper script that will be used by Helm.
cat << EOF > kustomize-wrapper.sh
#!/bin/bash
cat <&0 > chart.yaml
kustomize edit add resource chart.yaml
kustomize build . && rm chart.yaml
EOF
chmod +x kustomize-wrapper.sh

Now we can render or install the chart directly using Helm:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault --post-renderer ./kustomize-wrapper.sh

As is clear, the annoying part about this is managing extra script. Also, it's more or less the first method but using Helm itself to render the upstream chart.