Set OpenAPI patch strategy for Kubernetes Custom Resources - Kustomize

Kustomize supports 2 main client-side patching methods for Kubernetes manifests, JSON Patching and Strategic Merge Patch. In the JSON Patching method, you have a "meta" syntax that specifies operation/target/value. In the Strategic Merge Patch method, you can override values by providing a patch file with the same structure but with new values, and it will override the original values (it simply merges the 2 files with the same structure).

Each method has pros and cons but generally speaking, I would arguably say that Strategic Merge Patch is better for big changes/patches, and JSON Patching is better for smaller fine-grained patches. And for my use case, I will use Strategic Merge Patch, but I just faced a problem with patching Kubernetes Custom Resources!

ToC

• TL;DR
• 1. Task
• 2. Issue reproduction
• 3. Background
• 4. Solution
  • 4.1 Get the custom resource OpenAPI schema
  • 4.2 Find the desired key path
  • 4.3 Create the custom OpenAPI schema file
  • 4.4 Update kustomization.yaml with the OpenAPI schema file
• Conclusion

TL;DR

Kustomize's default patch strategy for the lists (arrays) is replace. That means the patch list will override the original list, which is not always the desired behavior. That behavior could be changed only if an OpenAPI schema for a Kubernetes resource is available to define the patch strategy.

The OpenAPI schema for Kustomize core resources (like Namespace, Deployment, Pod, etc.) is already part of Kustomize, so changing the patch strategy works out of the box for these resources. However, if you have a Kubernetes Custom Resource, you need to provide to Kustomize the OpenAPI schema of that custom resource. And that's only useful if the custom resource includes the OpenAPI extensions related to merging strategy.

This post shows how to add those extensions to have control over the patch strategy. You can jump directly to the solution section if you already know all these details.

1. Task

I want to use Kustomize to patch Kubernetes Custom Resources like Prometheus AlertmanagerConfig), and I want to use merge as a patch strategy for lists. That means the original lists in the same path should be merged, not overridden by the patch list. That works out of the box for Kustomize core resources but not for custom resources. First, let's see that in action, then dive into the explanation afterward.

2. Issue reproduction

Let's have a look at this example using the core resource Pod, given this Kustomization file:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- pod.yaml

patches:
- pod-patch01.yaml
- pod-patch02.yaml

And these resources and patches files:

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_01
      value: source

# pod-patch01.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_01
      value: patch 01

# pod-patch02.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_02
      value: patch 02

The kustomize build . will return:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_02
      value: patch 02
    - name: MY_ENV_VAR_01
      value: patch 01

As you see, the env key MY_ENV_VAR_01 overrode by the value from pod-patch01.yaml, and the env key MY_ENV_VAR_02 has just been added from pod-patch02.yaml. That's great; the lists are merged based on the name key.

...

However, if you tried to do that with a CustomResource like AlertmanagerConfig, it would not work! Let's give it a try! Given this Kustomization file:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- alertmanagerconfig.yaml

patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml

And these resources and patches files:

# alertmanagerconfig.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook01'
    webhookConfigs:
    - url: 'http://example.com/'

# alertmanagerconfig-patch01.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook01'
    webhookConfigs:
    - url: 'http://example01.com/'

# alertmanagerconfig-patch02.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook02'
    webhookConfigs:
    - url: 'http://example02.com/'

The kustomize build . will return:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: webhook02
    webhookConfigs:
    - url: http://example02.com/

As you see, the last patch from the file alertmanagerconfig-patch02.yaml replaced everything in the spec.receivers list, and that's the default behavior in Kustomize. The patch list will replace everything in the original list. Why? Because that's the safest choice since Kustomize doesn't know anything about AlertmanagerConfig schema! Before diving into the fix, let's learn more about the why.

3. Background

The Strategic Merge Patch is a client-side merge method that merges 2 or more Kubernetes manifests together based on the manifest apiVersion, kind, and metadata.name. To merge 2 YAML files, you need to decide the "merge strategy" for different data types, i.e., what should happen for the "string", "int", "list", "map", and so on? Should they merge together? Or do the patch values override the original values?

Also, each data type could be patched differently; for example, how to patch a list? Kustomize provides different patch formats like merge, replace, and delete. In fact, in a previous post (Delete a manifest from Kustomize base), I mentioned the delete patch strategy, which works out of the box with core Kubernetes primitive (namespace, deployment, pod, etc.), but not the CustomResources.

Why does it work with core resources only? Because of 2 things.

1. The Kubernetes project includes specific keys (extensions) in the core resources OpenAPI schema to deal with that. Namely the OpenAPI extensions x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key (see them in Kubernetes swagger.json).
2. The OpenAPI schema for Kubernetes' core resources is embedded in Kustomize.

If those keys are not included in the OpenAPI schema, and Kustomize doesn't have access to the OpenAPI schema, the default behavior will be applied, which in Kustomize, the patch list will fully replace the original list.

4. Solution

Now there are 2 cases, First, if the custom resource definition already has the x-kubernetes-patch-*`, and second, if the custom resource definition doesn't have them at all. The Kustomize supports "openapi" field, which specifies where Kustomize gets its OpenAPI schema.

For the first case, it's easy; you just need to point Kustomize to the OpenAPI schema and that's it!

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  # It could be also a URL.
  path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
[...]

However, for the second case, in a Platonic world, you should contact the upstream to add the OpenAPI extension keys x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key. But as you know, in reality, that will take ages, and in the best-case scenario, it will not happen overnight! So the pragmatic solution is to tell Kustomize how to deal with that custom resource via OpenAPI schema.

We will simply get the Custom Resource's OpenAPI schema and add the x-kubernetes-patch-* keys to it with the merge strategy, which can also be customized using different patch formats like merge, replace, and delete.

The following are the step to get the OpenAPI schema of a custom resource, clean it, add the merge strategy keys, and finally use it in kustomization.yaml file.

4.1 Get the custom resource OpenAPI schema

You can get the OpenAPI schema for the resource from the upstream project, or if you have already installed its CustomResourceDefinition, then you can get it directly by calling Kubernetes API. And since K8s API will return every definition it has (probably thousands of lines), we will use jq to get the exact custom resource OpenAPI schema.

Here is a snippet that will help to get the OpenAPI definition for a particular resource:

get_openapi_definition () {
    jq \
        --arg group "${1}" \
        --arg version "${2}" \
        --arg kind "${3}" \
        '.definitions | with_entries(select(.value."x-kubernetes-group-version-kind"[0] |
            .group==$group and
            .version==$version and
            .kind==$kind
        ))'
}

And we can get the schema for the exact resource from Kubernetes API by running the following (remember, you should have installed the CRD for that resource into your Kubernetes cluster to be able to do that):

kustomize openapi fetch | get_openapi_definition "monitoring.coreos.com" "v1alpha1" "AlertmanagerConfig" > alertmanagerconfig_openapi_schema_map.json

4.2 Find the desired key path

Here is most of the manual work, but the good news is that you need to do it once. We need 2 things, the data of the path spec.receivers and the key x-kubernetes-group-version-kind. Open the schema file and remove everything not under the hierarchy of the path we want to customize.

Here is what it looks like after removing everything unrelated:

# alertmanagerconfig_openapi_schema_map.json
{
  "com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
    "properties": {
      "spec": {
        "properties": {
          "receivers": {
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    "type": "object",
    "x-kubernetes-group-version-kind": [
      {
        "group": "monitoring.coreos.com",
        "kind": "AlertmanagerConfig",
        "version": "v1alpha1"
      }
    ]
  }
}

4.3 Create the custom OpenAPI schema file

Now we just need to put everything together adding x-kubernetes-patch-* keys and the definitions parent. The final result will look like the following:

# monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
{
  "definitions": {
    "com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
      "properties": {
        "spec": {
          "properties": {
            "receivers": {
              "x-kubernetes-patch-merge-key": "name",
              "x-kubernetes-patch-strategy": "merge",
              "type": "array"
            }
          },
          "type": "object"
        }
      },
      "type": "object",
      "x-kubernetes-group-version-kind": [
        {
          "group": "monitoring.coreos.com",
          "kind": "AlertmanagerConfig",
          "version": "v1alpha1"
        }
      ]
    }
  }
}

4.4 Update kustomization.yaml with the OpenAPI schema file

The final step, we need to tell Kustomize about our custom OpenAPI schema file as follows (it also accepts YAML files in case you like to convert it):

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json

resources:
- alertmanagerconfig.yaml

patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml

Now the kustomize build . will return:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: webhook02
    webhookConfigs:
    - url: http://example02.com/
  - name: webhook01
    webhookConfigs:
    - url: http://example01.com/

Great, it works as expected! 🎉️ And the custom resource list is merged based on the name key (you can choose any merge key based on your use case).

Conclusion

Kustomize is super powerful and has many capabilities to manage your entire Kubernetes infrastructure as code! And the most fantastic thing? It's now part of kubectl, so it's almost the standard way to deal with advanced Kubernetes manifest structure.

Enjoy :-)

Continue Reading »

Handling nullable nested values - Helm

I'd say this is one of the best tricks I've learned about Helm recently! For me, one of the most annoying things in Helm was that it's hard to handle nullable nested values!

For example, if you want to get the value of a nested key like

Continue Reading »