01/01/2023

Awesome Kustomize list - Kubernetes

After 3 years of using Kustomize intensively to manage Kubernetes manifests, I just created a list of "Awesome Kustomize" resources 🀩️

If you are using Kubernetes, this repo will be a pretty good starting point for Kustomize plugins, guides, tips and tricks, and more.

Kustomize is not a new tool, it started in 2018, and now it's built into "kubectl" (since v1.14), which means it's now considered the official method to deal with advanced Kubernetes scenarios (e.g., GitOps).

If you are SysOps, DevOps, SRE, or anyone working intensively with Kubernetes, you definitely should read more about Kustomize!

Also, if you just use "kubectl" or Helm, then consider using Kustomize. It could help you in many different ways (it's not a replacement for Helm, you can use both).

Follow the repo to get the updates πŸš€️

https://github.com/aabouzaid/awesome-kustomize

Continue Reading »

31/12/2022

2022 Highlights


Just a random image generated with AI!

Finally, 2022 is over! What a crazy year! In many countries, the Covid-19 pandemic is about to come to an end, but a global economic recession is almost at the door!

On a personal level, it wasn't an easy year for sure, but it was good in many different ways.

Top 5 highlights in 2022

  1. Career: Started the Distribution team at Camunda 🀩️ which is responsible for building and deploying the Camunda Platform 8 Self-Managed (now using an umbrella [Helm chart). Later on, there will be a Kubernetes Operator. That's a great career boost; I just started with many new and exciting challenges. And BTW, my team will begin hiring in 2023!

  2. Coding for Kubernetes: Big refactoring for Bank-Vaults operator which is the biggest open-source contribution to a project I don't own/manage. It polished my Golang skills, and I learned many new things (and had fun where I redesigned the operator logo πŸ˜‚️).

  3. Security Knowledge-sharing: In 2021, I got my CKS certificate. Then, at the beginning of 2022, I started a security initiative at Camunda to enhance security practices. Then, later on, I conducted a session about Kubernetes Security Best Practices (with some tips for the CKS exam) which was a great case that includes theory, applied practice, and knowledge-sharing!

  4. Advanced CI/CD Knowledge-sharing: I wrote a detailed post about my experience with custom step conditionalRetry, which handles failures on spot/preemptible infrastructure so you could save up to 90% of the costs and have stable builds as well! It's been released as open source, and you can use that in your pipeline!

  5. Activities: Helped more people in their careers, once by moderating the DevOps circle at [JobStack 2022, and also in the voluntary mentorship that I do from time to time.

Besides these highlights, I had some nice stuff during the year. For example:

  • Added more features to kubech (which is a tool to set kubectl context/namespace per shell/terminal )
  • Virtually attended KubeCon Europe 2022, and the content was great!
  • Reached my writing goal this year and wrote 12 blog posts in 2022!

And since we are on this topic, here are the top 5 visited blog posts in 2022!

Top 5 posts in 2022

  1. Delete a manifest from Kustomize base - Kubernetes

  2. 3 ways to customize off-the-shelf Helm charts with Kustomize - Kubernetes

  3. Validate, format, lint, secure, and test Terraform IaC - CI/CD

  4. Now I'm a Certified Kubernetes Application Developer + 10 exam tips

  5. Continuous Delivery and Maturity Model - DevOps

No wonder that Kustomize post is the hights post; that's because there is not much content about it even though it's built-in kubectl now! (since v1.14), I probably need to give it more attention since there is an increase in the demand for it.

For that reason, I just started Awesome Kustomize, which is a curated and collaborative list of awesome Kustomize resources πŸŽ‰️


Enjoy πŸš€️

Continue Reading »

12/12/2022

22/11/2022

Set OpenAPI patch strategy for Kubernetes Custom Resources - Kustomize

Kustomize supports 2 main client-side patching methods for Kubernetes manifests, JSON Patching and Strategic Merge Patch. In the JSON Patching method, you have a "meta" syntax that specifies operation/target/value. In the Strategic Merge Patch method, you can override values by providing a patch file with the same structure but with new values, and it will override the original values (it simply merges the 2 files with the same structure).

Each method has pros and cons but generally speaking, I would arguably say that Strategic Merge Patch is better for big changes/patches, and JSON Patching is better for smaller fine-grained patches. And for my use case, I will use Strategic Merge Patch, but I just faced a problem with patching Kubernetes Custom Resources!

ToC

TL;DR

Kustomize's default patch strategy for the lists (arrays) is replace. That means the patch list will override the original list, which is not always the desired behavior. That behavior could be changed only if an OpenAPI schema for a Kubernetes resource is available to define the patch strategy.

The OpenAPI schema for Kustomize core resources (like Namespace, Deployment, Pod, etc.) is already part of Kustomize, so changing the patch strategy works out of the box for these resources. However, if you have a Kubernetes Custom Resource, you need to provide to Kustomize the OpenAPI schema of that custom resource. And that's only useful if the custom resource includes the OpenAPI extensions related to merging strategy.

This post shows how to add those extensions to have control over the patch strategy. You can jump directly to the solution section if you already know all these details.

1. Task

I want to use Kustomize to patch Kubernetes Custom Resources like Prometheus AlertmanagerConfig), and I want to use merge as a patch strategy for lists. That means the original lists in the same path should be merged, not overridden by the patch list. That works out of the box for Kustomize core resources but not for custom resources. First, let's see that in action, then dive into the explanation afterward.

2. Issue reproduction

Let's have a look at this example using the core resource Pod, given this Kustomization file:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- pod.yaml

patches:
- pod-patch01.yaml
- pod-patch02.yaml

And these resources and patches files:

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_01
      value: source

# pod-patch01.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_01
      value: patch 01

# pod-patch02.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_02
      value: patch 02

The kustomize build . will return:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    env:
    - name: MY_ENV_VAR_02
      value: patch 02
    - name: MY_ENV_VAR_01
      value: patch 01

As you see, the env key MY_ENV_VAR_01 overrode by the value from pod-patch01.yaml, and the env key MY_ENV_VAR_02 has just been added from pod-patch02.yaml. That's great; the lists are merged based on the name key.

...

However, if you tried to do that with a CustomResource like AlertmanagerConfig, it would not work! Let's give it a try! Given this Kustomization file:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- alertmanagerconfig.yaml

patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml

And these resources and patches files:

# alertmanagerconfig.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook01'
    webhookConfigs:
    - url: 'http://example.com/'

# alertmanagerconfig-patch01.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook01'
    webhookConfigs:
    - url: 'http://example01.com/'

# alertmanagerconfig-patch02.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: 'webhook02'
    webhookConfigs:
    - url: 'http://example02.com/'

The kustomize build . will return:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: webhook02
    webhookConfigs:
    - url: http://example02.com/

As you see, the last patch from the file alertmanagerconfig-patch02.yaml replaced everything in the spec.receivers list, and that's the default behavior in Kustomize. The patch list will replace everything in the original list. Why? Because that's the safest choice since Kustomize doesn't know anything about AlertmanagerConfig schema! Before diving into the fix, let's learn more about the why.

3. Background

The Strategic Merge Patch is a client-side merge method that merges 2 or more Kubernetes manifests together based on the manifest apiVersion, kind, and metadata.name. To merge 2 YAML files, you need to decide the "merge strategy" for different data types, i.e., what should happen for the "string", "int", "list", "map", and so on? Should they merge together? Or do the patch values override the original values?

Also, each data type could be patched differently; for example, how to patch a list? Kustomize provides different patch formats like merge, replace, and delete. In fact, in a previous post (Delete a manifest from Kustomize base), I mentioned the delete patch strategy, which works out of the box with core Kubernetes primitive (namespace, deployment, pod, etc.), but not the CustomResources.

Why does it work with core resources only? Because of 2 things.

  1. The Kubernetes project includes specific keys (extensions) in the core resources OpenAPI schema to deal with that. Namely the OpenAPI extensions x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key (see them in Kubernetes swagger.json).
  2. The OpenAPI schema for Kubernetes' core resources is embedded in Kustomize.

If those keys are not included in the OpenAPI schema, and Kustomize doesn't have access to the OpenAPI schema, the default behavior will be applied, which in Kustomize, the patch list will fully replace the original list.

4. Solution

Now there are 2 cases, First, if the custom resource definition already has the x-kubernetes-patch-*`, and second, if the custom resource definition doesn't have them at all. The Kustomize supports "openapi" field, which specifies where Kustomize gets its OpenAPI schema.

For the first case, it's easy; you just need to point Kustomize to the OpenAPI schema and that's it!

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  # It could be also a URL.
  path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
[...]

However, for the second case, in a Platonic world, you should contact the upstream to add the OpenAPI extension keys x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key. But as you know, in reality, that will take ages, and in the best-case scenario, it will not happen overnight! So the pragmatic solution is to tell Kustomize how to deal with that custom resource via OpenAPI schema.

We will simply get the Custom Resource's OpenAPI schema and add the x-kubernetes-patch-* keys to it with the merge strategy, which can also be customized using different patch formats like merge, replace, and delete.

The following are the step to get the OpenAPI schema of a custom resource, clean it, add the merge strategy keys, and finally use it in kustomization.yaml file.

4.1 Get the custom resource OpenAPI schema

You can get the OpenAPI schema for the resource from the upstream project, or if you have already installed its CustomResourceDefinition, then you can get it directly by calling Kubernetes API. And since K8s API will return every definition it has (probably thousands of lines), we will use jq to get the exact custom resource OpenAPI schema.

Here is a snippet that will help to get the OpenAPI definition for a particular resource:

get_openapi_definition () {
    jq \
        --arg group "${1}" \
        --arg version "${2}" \
        --arg kind "${3}" \
        '.definitions | with_entries(select(.value."x-kubernetes-group-version-kind"[0] |
            .group==$group and
            .version==$version and
            .kind==$kind
        ))'
}

And we can get the schema for the exact resource from Kubernetes API by running the following (remember, you should have installed the CRD for that resource into your Kubernetes cluster to be able to do that):

kustomize openapi fetch | get_openapi_definition "monitoring.coreos.com" "v1alpha1" "AlertmanagerConfig" > alertmanagerconfig_openapi_schema_map.json

4.2 Find the desired key path

Here is most of the manual work, but the good news is that you need to do it once. We need 2 things, the data of the path spec.receivers and the key x-kubernetes-group-version-kind. Open the schema file and remove everything not under the hierarchy of the path we want to customize.

Here is what it looks like after removing everything unrelated:

# alertmanagerconfig_openapi_schema_map.json
{
  "com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
    "properties": {
      "spec": {
        "properties": {
          "receivers": {
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    "type": "object",
    "x-kubernetes-group-version-kind": [
      {
        "group": "monitoring.coreos.com",
        "kind": "AlertmanagerConfig",
        "version": "v1alpha1"
      }
    ]
  }
}

4.3 Create the custom OpenAPI schema file

Now we just need to put everything together adding x-kubernetes-patch-* keys and the definitions parent. The final result will look like the following:

# monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
{
  "definitions": {
    "com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
      "properties": {
        "spec": {
          "properties": {
            "receivers": {
              "x-kubernetes-patch-merge-key": "name",
              "x-kubernetes-patch-strategy": "merge",
              "type": "array"
            }
          },
          "type": "object"
        }
      },
      "type": "object",
      "x-kubernetes-group-version-kind": [
        {
          "group": "monitoring.coreos.com",
          "kind": "AlertmanagerConfig",
          "version": "v1alpha1"
        }
      ]
    }
  }
}

4.4 Update kustomization.yaml with the OpenAPI schema file

The final step, we need to tell Kustomize about our custom OpenAPI schema file as follows (it also accepts YAML files in case you like to convert it):

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json

resources:
- alertmanagerconfig.yaml

patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml

Now the kustomize build . will return:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: example
spec:
  receivers:
  - name: webhook02
    webhookConfigs:
    - url: http://example02.com/
  - name: webhook01
    webhookConfigs:
    - url: http://example01.com/

Great, it works as expected! πŸŽ‰️ And the custom resource list is merged based on the name key (you can choose any merge key based on your use case).

Conclusion

Kustomize is super powerful and has many capabilities to manage your entire Kubernetes infrastructure as code! And the most fantastic thing? It's now part of kubectl, so it's almost the standard way to deal with advanced Kubernetes manifest structure.

Enjoy :-)

Continue Reading »

11/11/2022

09/09/2022

How to create Makefile targets with dynamic parameters and autocompletion - Make

Make is one of the oldest build automation tools ever (the original Make was created in 1976!). And since then, it got many implementations as BSD make, GNU make, and Microsoft nmake. It uses a declarative syntax, and sometimes that's the best and worst thing about it!

You probably used Make at least once. It's widely adopted in the tech industry, or as someone said, "Make is the second best tool to automate anything!". πŸ˜„️

After

Continue Reading »

08/08/2022

2 ways to route Ingress traffic across namespaces - Kubernetes

The tech industry of full of workarounds, you are probably using or relying on some workaround. And there is no problem with that per se. But most important is that when you do a workaround you should be aware of that and change it to the standard way if it's more intuitive.

The Problem

A couple of years ago I had a use case where a single domain had 2 sub-paths each of them having its own service in different namespaces. Let's see this example:

example.com/app => service "backend" in namespace "app"
example.com/blog => service "wordpress" in namespace "blog"

The problem was that the Ingress object is able to send traffic to the services within the same namespace only and only one ingress object per host/domain was allowed. But at a certain version (I think that was v1.0.0) Nginx Ingress introduced what's known as Mergeable Ingress Resources. However, at that time I was working with an old version that doesn't support that (or there was an issue I cannot really recall now).

So at that time I found a generic solution which looks like a workaround. Actually, by thinking about it now, it was not a bad workaround. It depends on how you manage your infrastructure, and you can think about it as a centralized vs decentralized approach.

The Solution

So here are the 2 ways to route Ingress traffic across namespaces in Kubernetes. The 1st one you could call the standard way (which relies on the Ingress controller capabilities), and 2nd is the generic way that I used back in the days.

1. Mergeable Ingress Resources

If you took a look at the official Nginx docs you will find the Cross-namespace Configuration page suggests using Mergeable Ingress Resources. That approach relies on a simple idea, there is a single Ingress resource that has all configurations related to the host/domain and that resource is called "master", and any number of the Ingress resources handles the paths under that host/domain and each of these resources is called "minion".

Each one of the master or minion can or can not contain some of the Ingress annotations based on their role. Here I will use here the examples from the official documentation.

Config for shop.example.com like TLS and host-level annotations.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-ingress-master
  namespace: shop
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/mergeable-ingress-type: "master"
spec:
  tls:
  - hosts:
    - shop.example.com
    secretName: shop-secret
  rules:
  - host: shop.example.com

Config for shop.example.com/coffee which is in the coffee namespace and routes the traffic of the coffee-svc service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-ingress-coffee-minion
  namespace: coffee
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/mergeable-ingress-type: "minion"
spec:
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /coffee
        pathType: Prefix
        backend:
          service:
            name: coffee-svc
            port:
              number: 80

Config for shop.example.com/tea which is in the tea namespace and routes the traffic of the tea-svc service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-ingress-tea-minion
  namespace: tea
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/mergeable-ingress-type: "minion"
spec:
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc
            port:
              number: 80

As you see, the Ingress config is split into 2 parts, the host/domain config, and the paths config. Each one of them could be in a different namespace and handles the services in that namespace.

2. ExternalName Service

For one reason or another, that non-centralized way of managing Ingress resources (where the Ingress object is split across namespaces) might not fit all workloads. So here is another way I used it before and I find it much simpler for many use cases.

This method relies on native Kubernetes ExternalName Service which is simply a DNS CNAME! This method is centralized where it uses the normal Ingress object in addition to ExternalName Service within the same namespace as a bridge to the services in any other namespace.

The following is an example of that setup with a single Ingress resource and 2 ExternalName services.

Config for shop.example.com including the 2 sub-paths /coffee and /tea.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-ingress
  namespace: shop
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - shop.example.com
    secretName: shop-secret
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /coffee
        pathType: Prefix
        backend:
          service:
            name: coffee-svc-bridge
            port:
              number: 80
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc-bridge
            port:
              number: 80

The coffee-svc-bridge service in the shop namespace is a CNAME for the coffee-svc service in coffee namespace:

apiVersion: v1
kind: Service
metadata:
  name: coffee-svc-bridge
  namespace: shop
spec:
  type: ExternalName
  externalName: coffee-svc.coffee

The tea-svc-bridge service in the shop namespace is a CNAME for the tea-svc service in tea namespace:

apiVersion: v1
kind: Service
metadata:
  name: tea-svc-bridge
  namespace: shop
spec:
  type: ExternalName
  externalName: tea-svc.tea

As you see, the Ingress config comes in 1 part and is normal. And use the ExternalName services as a bridge to access the services in the other namespaces.

Conclusion

Maybe the second approach looks like a workaround, but for some workloads could be better and easier to follow and digest. But in general, it's good to have different ways to use what's fit better.

Enjoy :-)

Continue Reading »

27/07/2022

Notes about KRM Functions - Kustomize

Recently I dived into the new plugin system in Kustomize, KRM Functions, so I wanted to know more about it. Kubernetes Resource Model or KRM for short is simply a unified way to work with resources in Kubernetes ecosystem. For example, all plugins will have the same input and output format.

Here is a summary I found useful to share:

  • Kustomize decided to adapt KRM (Kubernetes Resource Model) functions from kpt ... and that's actually not new, it's been there for some time (around 2020).
  • The goal is to deprecate the old plugins style model. Kustomize already deprecated both Go Plugins and Exec plugins in favour of KRM style.
  • KRM Functions style has 2 ways for the plugins: Containerized KRM Functions and Exec KRM Functions.
  • The containerized KRM function is really useful one because you don't need to manage and download the Kustomize plugins (it was super annoying to manage plugins especially across multiple OS).

However, KRM functions are still alpha but look super promising, however, they are still buggy or incomplete for some use cases!

  • KRM exec has a bug which makes it almost unusable. In the PR no. #4654 I've a proposal to fix that issue.
  • KRM container has also some issues! It only works with KRM resources but not any external files (for example, if a plugin reads files from the disk, like creating ConfigMap from a text file, that will not work at the moment).

In May 2022, I decided to go a bit further and try to implement the KRM style to one of the existing plugins. So I've selected SopsSecretGenerator Kustomize plugin and introduced KRM support in the PR no. #32 So if you want to have an idea how KRM style looks like in action, then take a look at the change I made in that PR (It has been merged already).

In conclusion, KRM Functions look super promising but they are not that mature yet in Kustomize and they don't fit all the use cases.

Continue Reading »

02/07/2022

Kubernetes Security Best Practices with tips for the CKS exam - Presentation

At the end of last year (2021), and after a couple of years of Kubernetes production hands-on, I found that it was time to dive more into Kubernetes security, and after a lot of reading and practising, I got my CKS certificate. Also, for the last 3 quarters, security was one of the focus areas in my team, and I was taking care of it.

For that reason, I decided to consolidate that into a session which is a combination of Kubernetes Security Best Practices and tips for the Certified Kubernetes Security Specialist (CKS) exam to share the knowledge in my team as well across teams.

The session is just 15 Min in total. The first 6 Min are for everyone and the rest for Kubernetes specialists or anyone who wants to dive more into Kubernetes security topics. If you are just interested in the tools, then jump to section #5 Kubernetes Security Starter Kit. If you are just interested in the CKS exam tips, then jump to section #6 CKS Exam Overview and Tips.

Agenda:

  1. Introduction
  2. Shift-left and DevSecOps
  3. General Security Concepts
  4. The 4C's of Cloud Native Security
  5. Kubernetes Security Starter Kit
  6. CKS Exam Overview and Tips

Note: If you want more details about CKS, checkout my previous post for more info Now I'm a Certified Kubernetes Security Specialist + exam tips.



The recording of Kubernetes Security Best Practices session

Overview:

A dive into Kubernetes Security Best Practices and tips for the Certified Kubernetes Security Specialist (CKS) exam.

The 1-3 sections are for everyone and will cover the container era's security. So it doesn't matter your title or background; they are a good start for anyone.

The 4-6 sections will dive more into Kubernetes security, so DevOps engineers and SREs will probably find that more interesting. But in general, anyone interested in Kubernetes security is more than welcome.


That's it, enjoy :-)

Continue Reading »
Powered by Blogger.

Hello, my name is Ahmed AbouZaid and this is my "lite" technical blog!

I'm a passionate DevOps engineer, Cloud/Kubernetes specialist, Free/Open source geek, and an author.

I believe in self CI/CD (Continuous Improvements/Development), also that "the whole is greater than the sum of its parts".

DevOps transformation, automation, data, and metrics are my preferred areas. And I like to help both businesses and people to grow.

Popular Posts