I'd say this is one of the best tricks I've learned about Helm recently! For me, one of the most annoying things in Helm was that it's hard to handle nullable nested values!
For example, if you want to get the value of a nested key like
Make is one of the oldest build automation tools ever (the original Make was created in 1976!). And since then, it got many implementations as BSD make, GNU make, and Microsoft nmake. It uses a declarative syntax, and sometimes that's the best and worst thing about it!
You probably used Make at least once. It's widely adopted in the tech industry, or as someone said, "Make is the second best tool to automate anything!". 😄️
After
The tech industry is full of workarounds, and you probably rely on one or more. There is no problem with that per se, but it's important to review your workarounds from time to time because there could be a new standard/intuitive way to make it.
A couple of years ago I had a use case where a single domain has 2 sub-paths each of them using its own service in different namespaces. Let's see this example:
example.com/app => service "backend" in namespace "app" example.com/blog => service "wordpress" in namespace "blog"
The problem was that the Ingress object is namespaced which means that it interacts with services within the same namespace. Also, only one ingress object per host/domain is allowed.
So at that time I found a generic solution which looks like a workaround. Actually, by thinking about it now, it was not a bad workaround. It depends on how you manage your infrastructure, and you can think about it as a centralized vs decentralized approach.
So here are the 2 ways to route Ingress traffic across namespaces in Kubernetes. The 1st is a generic way that will work with any Ingress controller. The 2nd relies on the Ingress controller capabilities NGINX Ingress Controller by NGINX, Inc. (NOT Ingress-NGINX Controller by Kubernetes).
This method relies on native Kubernetes ExternalName Service which is simply a DNS CNAME! This method is centralized where it uses the normal Ingress object in addition to ExternalName Service within the same namespace as a bridge to the services in any other namespace.
The following is an example of that setup with a single Ingress resource and 2 ExternalName services (3 endpoints which are /, /coffee, and /tea).
Config for shop.example.com including the 2 sub-paths /coffee and /tea in addition to the root /.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress
namespace: shop
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- shop.example.com
secretName: shop-secret
rules:
- host: shop.example.com
http:
paths:
- path: /coffee
pathType: Prefix
backend:
service:
name: coffee-svc-bridge
port:
number: 80
- path: /tea
pathType: Prefix
backend:
service:
name: tea-svc-bridge
port:
number: 80
The coffee-svc-bridge service in the shop namespace is a CNAME for the coffee-svc service in coffee namespace:
apiVersion: v1 kind: Service metadata: name: coffee-svc-bridge namespace: shop spec: type: ExternalName externalName: coffee-svc.coffee
The tea-svc-bridge service in the shop namespace is a CNAME for the tea-svc service in tea namespace:
apiVersion: v1 kind: Service metadata: name: tea-svc-bridge namespace: shop spec: type: ExternalName externalName: tea-svc.tea
As you see, the Ingress config comes in 1 part and is normal. And use the ExternalName services as a bridge to access the services in the other namespaces.
The other option is using controller-specific capabilities to achieve that goal. There are dozens of Ingress controllers for Kubernetes like the Ingress-NGINX (by Kubernetes project), NGINX Ingress Controller (by NGINX, Inc.), Traefik, HAProxy, Istio, and many more.
Here I will cover only NGINX Ingress Controller by NGINX, Inc., but the idea is the same, using the controller-specific features.
If you took a look at the official Nginx docs you will find the Cross-namespace Configuration page suggests using Mergeable Ingress Resources.
That approach relies on a simple idea, there is a single Ingress resource that has all configurations related to the host/domain and that resource is called "master", and any number of the Ingress resources handles the paths under that host/domain and each of these resources is called "minion".
Each one of the master or minion can or can not contain some Ingress annotations based on their role. Here I will use here the examples from the official documentation.
Config for shop.example.com like TLS and host-level annotations.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-master
namespace: shop
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "master"
spec:
tls:
- hosts:
- shop.example.com
secretName: shop-secret
rules:
- host: shop.example.com
Config for shop.example.com/coffee which is in the coffee namespace and routes the traffic of the coffee-svc service.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-coffee-minion
namespace: coffee
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /coffee
pathType: Prefix
backend:
service:
name: coffee-svc
port:
number: 80
Config for shop.example.com/tea which is in the tea namespace and routes the traffic of the tea-svc service.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-tea-minion
namespace: tea
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /tea
pathType: Prefix
backend:
service:
name: tea-svc
port:
number: 80
As you see, the Ingress config is split into 2 parts, the host/domain config, and the paths config. Each one of them could be in a different namespace and handles the services in that namespace.
Maybe the first approach looks like a workaround, but for many workloads could be better and easier to follow and digest. But in general, it's good to have different ways to use what's fit better.
Enjoy :-)
Recently I dived into the new plugin system in Kustomize, KRM Functions, so I wanted to know more about it. Kubernetes Resource Model or KRM for short is simply a unified way to work with resources in Kubernetes ecosystem. For example, all plugins will have the same input and output format.
Here is a summary I found useful to share:
However, KRM functions are still alpha but look super promising, however, they are still buggy or incomplete for some use cases!
In May 2022, I decided to go a bit further and try to implement the KRM style to one of the existing plugins. So I've selected SopsSecretGenerator Kustomize plugin and introduced KRM support in the PR no. #32 So if you want to have an idea how KRM style looks like in action, then take a look at the change I made in that PR (It has been merged already).
In conclusion, KRM Functions look super promising but they are not that mature yet in Kustomize and they don't fit all the use cases.
At the end of last year (2021), and after a couple of years of Kubernetes production hands-on, I found that it was time to dive more into Kubernetes security, and after a lot of reading and practising, I got my CKS certificate. Also, for the last 3 quarters, security was one of the focus areas in my team, and I was taking care of it.
For that reason, I decided to consolidate that into a session which is a combination of Kubernetes Security Best Practices and tips for the Certified Kubernetes Security Specialist (CKS) exam to share the knowledge in my team as well across teams.
The session is just 15 Min in total. The first 6 Min are for everyone and the rest for Kubernetes specialists or anyone who wants to dive more into Kubernetes security topics. If you are just interested in the tools, then jump to section #5 Kubernetes Security Starter Kit. If you are just interested in the CKS exam tips, then jump to section #6 CKS Exam Overview and Tips.
Note: If you want more details about CKS, checkout my previous post for more info Now I'm a Certified Kubernetes Security Specialist + exam tips.
A dive into Kubernetes Security Best Practices and tips for the Certified Kubernetes Security Specialist (CKS) exam.
The 1-3 sections are for everyone and will cover the container era's security. So it doesn't matter your title or background; they are a good start for anyone.
The 4-6 sections will dive more into Kubernetes security, so DevOps engineers and SREs will probably find that more interesting. But in general, anyone interested in Kubernetes security is more than welcome.
That's it, enjoy :-)
Last Satruday (18.06.2022), I had a great chance to moderate and participate the DevOps circle in JobStack 2022 by Talents Arena. JobStack is the biggest virtual tech job fair in the region (MENA) and this was the 4rd edition.
With Hussein El-Sayed (Software engineer III at AWS) and Mohamed Radwan (Sr. Cloud Architect at T-Systems) ... we answered many different questions about DevOps. The circle or the AMA session was heavily based on a previous collaborative session between us in 2022 (DevOps! What, Why, and How? - Arabic)
The whole event was great and had a lot of fruitful sessions and discussions.
Kustomize is a template-free declarative management tool for Kubernetes resources. Kustomize has 2 main concepts: Generators and Transformers. In short, the first able to create K8s manifests, and the second is able to manipulate K8s manifests. In this post, I'm interested in the Kustomize Transformers.
In Q1 2022, my friend Islam Wazery and I were working on an interesting enhancement for the open-source Vault Kubernetes Operator, Bank-Vaults.
It's one of my biggest open-source contributions recently. In this meta post, I like to share some details about the problem we were trying to solve, goal, available solutions, implementation details, and challenges during working on the new feature.
Anyone who used Kubernetes knows that the Secret resources are encoded, not encrypted, so you probably need another solution to manage your secrets and sensitive data. HashiCorp Vault is one of the best tools for that purpose.
In case you didn't use Vault before, here is a short intro from its docs:
Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, or certificates. Vault provides encryption services that are gated by authentication and authorization methods. Using Vault's UI, CLI, or HTTP API, access to secrets and other sensitive data can be securely stored and managed, tightly controlled (restricted), and auditable.
HashiCorp already provides resources to install Vault on Kubernetes as well as a Helm chart for Vault. However, there is no official solution from HashiCorp to manage Vault itself on Kubernetes. And here comes Bank-Vaults, the Vault Swiss Army Knife!
Bank-Vaults by Banzai Cloud is an open-source umbrella project which provides various tools (Operator, Configurer, Vault Env injector, and more) for ease of use and operation of Hashicorp Vault.
The most exciting part here is the Vault Operator by Bank-Vaults. Which allows you to manage Vault on Kubernetes. And based on my research, it's the only operator in the market for Vault, so I started a PoC to use it in production.
After the initial setup, It seemed that the Bank-Vaults operator was mature and production-ready. It had many features like bootstrap, sealing, unsealing, cloud backend, and all the features we need, but it's missing an important feature, it didn't support full Vault management! It handled the creation of Vault's config (like policies, secrets engines, auth methods, etc.), but it didn't handle the removal of the config! And that was confirmed by the issue no. #605 which had been unresolved for more than 2 years! (Aug 2019)
Next, I've checked the operator code, and it turned out that the operator works only with create/update, but it doesn't have any mechanism to work the config removal. No one fixed that because it's a full feature that needs much work (well, that's why it needed more than 2 years to fix).
That means the operator doesn't fully manage Vault! Unfortunately, this is a deal-breaker to use the operator in production. And to fix that, there are several ways to remove the unmanaged config. In the following sections, I dive more into the available mechanisms to handle the config removal, but first, let's set the goal.
I already experienced managing Vault using Terraform but doing that on Kubernetes would be a snowflake where it needs an extra stateful tool! I like to use Terraform for the infrastructure like Kubernetes clusters but not for the apps. Hence, I don't want to go that way.
So the ultimate goal is that the operator should be able to manage Vault completely. It should add and remove Vault config like policies, secrets engines, auth methods, etc. And that should be done using the Kubernetes ecosystem in a cloud-native approach.
Based on my previous experience with code, infrastructure as code, and configuration management tools, there are several ways to achieve config removal, and each one of them has pros and cons.
The first mechanism is simply the purge approach, where the operator removes anything not in the configuration. This mechanism compares Bank-Vaults config and Vault config and removes the differences.
So this approach is somewhat radical. It doesn't allow any manual changes, and any change outside the configuration will be removed. But the good side is that, well, it doesn't allow any manual changes! So the configuration is the source of truth. However, there is mitigation to allow some manual changes by excluding some of the configs. I will discuss it in the implementation section.
The second mechanism is the last-diff approach, where the operator compares the old and the new config and removes anything not in the new config. This way is considered "semi-stateful" where you need to have the old config and the new config to compare them. This approach allows manual changes outside the operator, but the operator is only aware of the last changes.
The third mechanism is the diff approach, where the operator maintains a state of all its operations, and with any new change, it compares the changes with the state (this is the Terraform way). This way is fully-stateful, which allows for tracking the changes done by the operator and allows manual changes outside the operator.
Finally, the fourth mechanism is the flag approach, where the operator manages the config according to a config flag. For example, each policy in the operator config could have a field called "state", and its value could be "present" or "absent" (this is the style of config management tools like Ansible). In this solution, it's possible to have managed, and unmanaged config but the biggest downside is that you need to deal with the config on the individual level.
In the cloud-native era, the first style looks the most suitable approach where full management is assumed. So anything that is not in the config would be removed. And to mitigate that behavior, it's 'possible to exclude some sections like policies, auth methods, etc., so they could have manual changes if needed.
Vault has main 7 configuration sections:
Each section already has the "add" mechanism, and it's able to create the config in Vault, and the goal is to add the "remove" mechanism to have full CRUD (Create, read, update and delete). However, the "adding" code wasn't follow Golang style and it needed to be refactored. So for each section, the code is refactored first, then the "removing" code is added.
Let's take policies as an example (which will be the same way for all 7 configs mentioned above); the "removing" part works as the following:
The final step is creating E2E tests to run in the CI (Github Actions). The tests simply check different cases like removing a config while the purge option is disabled/enabled fully/partially. Now let's take a look at the challenges I had while working on this feature.
In the following sections, I'd like to share the top challenges while introducing full Vault management in the Bank-Vaults operator.
Bank-Vaults is not just the operator; it's an umbrella project to work with Vault. It's a mono repo with many shared parts. For example, the operator relies on a CLI tool with the same name.
Hence, the first challenge was to understand the project structure and where exactly to change, and how the changes could affect the rest of the project.
After a thoughtful dive into the project, it was clear what and where I should change to fully manage Vault by Bank-Vaults (so it can add and remove config in Vault). However, the write path code in the operator (that's responsible for creating and updating managed config) doesn't follow the Golang style. It was more like Python written in Golang. It reminded me of when I wrote Golang for the first time, coming from a Python background.
Leaving the write path code as it is would make the code oddly bad and redundant. So the first step was refactoring the "write path" code, then adding the "remove path" code (which is responsible for removing any unmanaged config). And this was the second challenge to solve before the actual implementation.
Another challenge was that the part I wanted to change didn't have any unit tests, but only generic acceptance tests were available. Which makes things harder to change. I needed to pay extra attention to ensure I didn't break anything while refactoring the existing code and introducing the new feature. That also means I should write some E2E tests to avoid this situation in the future.
As I mentioned before, this feature is a bit big, and it would be implemented by 2 people (my friend Wazery and me). At the same time, it's a new project we didn't work on before, and we didn't work together before. So we needed to make sure that everything was clear and both of us were aligned to deliver this feature in high quality.
With the PR no. #1538, and Bank-Vaults v1.15.1 was able to fully or partially purging unmanaged configuration in Vault.
The user has the option to fully or partially purge unmanaged config as shown here:
purgeUnmanagedConfig:
# This will purge any unmanaged config in Vault.
enabled: true
# This will prevent purging unmanaged config for secret engines in Vault.
exclude:
secrets: true
To avoid behavior change, and since this feature is destructive, it was safe to make it disabled by default. The user needs to enable it explicitly in Bank-Vaults config. And as usual, it's recommended to test it in a non-production environment first.
Wazery PRs:
My PRs:
That's it! It was a pretty exciting journey for me 🤩️
In 2021, I wrote a high-level blog post about how a small task force revamped and modernized a gigantic CI pipeline Which was an overview of the challenges we had while building advanced declarative pipelines.
In today's post, I dive more into the technical details and implementation and how we overcame long-running job limitations in Jenkins Declarative Pipelines.
About 10 years ago, I used SaltStack extensively and wrote my top 10 best practices for SaltStack formulas . Today, I do the ...
