As a DevOps engineer, I worked with many programming languages on a daily basis in CI/CD, infrastructure, and so on. But when it comes to DevOps tools, I usually work with Python and Golang.
Recently we have a use case to integrate external secret management systems with Kubernetes. e.g. we needed to get some credentials from Vault or Google Secret Manager and pass them to K8s.
We actually use consul-template (the name is a bit misleading for historical reasons, but it actually works with Consul and Vault) to access Vault secrets within K8s Pods. But doing that outside pods like creating K8s Secret to be used in imagePullSecrets was a bit hacky and we needed a better solution for that.
We found 2 solutions that could fit our use case, one of them was vault-secrets-webhook and the other was kubernetes-external-secrets. We eventually selected the second one and here is why.
1. Find the best fit!
1.1 Vault Secrets Webhook
Vault Secrets Webhook (VSW for shortening) is simply a K8s admission controller webhook which triggered with each change of K8s Secret and if that secret has a certain annotation, VSW will inject secrets from Vault.
Pros:
- It doesn't need any extra CustomResources. It uses core K8s primitives.
- It's part of a bigger project called Bank Vaults by Banzai Cloud (which a great project by the way). That makes it more sustainable.
- It's written in Golang and has pretty nice docs.
Cons:
- It works with Vault only. So it will not fit for another sources like Google Secret Manager.
- It uses admission controller webhook and if you don't monitor them closely they could make bigger issues in terms of security or unexpected behavior (because they can alter requests against K8s API).
Nice pros, but deal-breaker cons! So we decided to find another solution.
1.2 Kubernetes External Secrets
Kubernetes External Secrets (KES for shortening) is a typical K8s operator with a controller and CustomResourceDefinition. So it adds a new CRD called ExternalSecret, then users can create a CustomResource of that type with all needed config to pull secrets from an external resource and apply it in the K8s cluster.
Pros:
- It works with many backends! Vault, Google Secret Manager, AWS Secrets Manager, and even more.
- It's pretty straightforward and easy to use.
- It doesn't mess with admission controllers.
Cons:
- It's written in JavaScript! (really why? 😂️)
- It's a side project by GoDaddy engineers and not the main product and doesn't have much focus.
- It doesn't support multitenancy! So it's one operator per cluster! Which is not the best option security wise. (it has some ways to scope the access but they are really limited!)
KES was the best fit for our use case. But it needed a bit of polishing on the security side to make it multitenancy friendly. For example, deploy multi instances per cluster and scope the access to each instance which means limiting the permissions of the ServiceAccount to make sure it doesn't have access more than it needs.
2. Fix the best fit!
As you see, KES is really cool but needs some work. On one hand, it would be easier for me if KES was written in Python or Golang. But on the other hand, it's also a nice chance to have some hands-on with JavaScript for the first time. Cool, challenge accepted! 😀️
I like to learn from experts, and I have many as friends 😀️ So I've asked Ahmed Ayoub, a Software Engineer at Zalando SE. Not only because he is an expert with JavaScript, but also because he is the founder of M3ntorship! M3ntorship is a pretty nice initiative that helps newcomers in web development to work in a team and to build real production-ready projects.
I've asked him to help me and review my changes before an post the PR to the upstream repo. Because I don't know anything about JavaScript paradigm and probably I will simply write JavaScript in Python style! 😀️
And the result was those 2 PRs:
- Multitenancy: Allow to watch ExternalSecrets using annotation #549
- Multitenancy: Allow to watch ExternalSecrets in specific namespaces #548
I had a lot of fun with JavaScript, both the actual implementation or unit/e2e tests! I also learned many things about the language and this new paradigm while before I was barely was able to read JavaScript code!
Funny thing is that, a few days after I've submitted my PRs, 3 projects decided to consolidate their effort and union under one project called External Secrets and it would be rewritten in Golang! KES was of them! That was a piece of good news even better than my 2 PRs 😂️
...
I think that's it, and this nice journey could be summarized as the following: If you want to get secrets into K8s from an external source, kubernetes-external-secrets is your way to go till External Secrets project is ready!
