Overview
In KubeZero (an open-source out-of-the-box Platform Orchestrator with GitOps designed for multi-environment Cloud Native setup), virtual clusters are created using vCluster. The main GitOps tool used in KubeZero is Argo CD, so we needed to automate provisioning the cluster and adding it to Argo CD.
If you used Argo CD before, you probably know that Argo CD provides a method for declarative setup (like for GitOps) where you can add new K8s clusters credentials by storing them in secrets, just like repositories or repository credentials.
However, to automate that, you need some way to extract the vClusters credentials and format them as an Argo CD config. There are many ways to do that, I prefer to use a declarative method, which is External Secrets Operator, namely PushSecret and ClusterSecretStore.
Flow
The flow is simple: when a K8s cluster is created via vCluster, the cluster credentials are created as a Secret object in the same namespace as the virtual cluster. Then, using PushSecret templating capabilities, it will read the secret, reformat it, and then push it to the Argo CD cluster using ClusterSecretStore.
vCluster supports multiple installation methods. We use vCluster Helm chart, so the PushSecret is created within the Helm chart to further automate it. Using Helm here is not mandatory; you can use any other installation method you like.
Prerequisites
- Host K8s cluster (v1.31)
- Argo CD (v3.0.6)
- External Secrets Operator
(v0.17.0)
- ClusterSecretStore to store the reformated Secret object in Argo CD namespace.
Assuming you deploy the virtual cluster using vCluster (v4.3.0) Helm chart, you just need this extra Helm values file (here I just copy the example from KubeZero repo):
---
experimental:
deploy:
host:
manifestsTemplate: |
---
# Push the vCluster credentails to KubeZero ClusterSecretStore,
# which will save it as a Secret in the KubeZero namespace to be used as an Argo CD cluster config
# (just a secret with a specific label).
# https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: argo-cd-{{ .Release.Name }}-credentials
namespace: {{ .Release.Name }}
spec:
refreshInterval: 5m
secretStoreRefs:
- name: kubezero-management
kind: ClusterSecretStore
selector:
secret:
name: vc-{{ .Release.Name }}
data:
- match:
secretKey: name
remoteRef:
remoteKey: argo-cd-{{ .Release.Name }}-credentials
property: name
- match:
secretKey: server
remoteRef:
remoteKey: argo-cd-{{ .Release.Name }}-credentials
property: server
- match:
secretKey: config
remoteRef:
remoteKey: argo-cd-{{ .Release.Name }}-credentials
property: config
template:
engineVersion: v2
metadata:
annotations:
managed-by: external-secrets
labels:
argocd.argoproj.io/secret-type: cluster
data:
name: {{ .Release.Name }}
server: https://{{ .Release.Name }}.{{ .Release.Namespace }}.svc:443
config: |
{
"tlsClientConfig": {
"insecure": false,
"caData": "{{ printf "{{ index . "certificate-authority" | b64enc }}" }}",
"certData": "{{ printf "{{ index . "client-certificate" | b64enc }}" }}",
"keyData": "{{ printf "{{ index . "client-key" | b64enc }}" }}",
"serverName": "{{ .Release.Name }}"
}
}
That will create the reformated Secret object in the Argo CD namespace, where the Argo CD controller will read it as an config because of the lable argocd.argoproj.io/secret-type: cluster. The actual output will be something like this:
apiVersion: v1
kind: Secret
metadata:
annotations:
managed-by: external-secrets
labels:
argocd.argoproj.io/secret-type: cluster
name: argo-cd-k0-credentials
namespace: argo-cd
# The base64 is decoded for the sake of the example.
data:
name: argo-cd-k0
server: https://argo-cd-k0.mgmt-demo.svc:443
config: |
{
"tlsClientConfig": {
"insecure": false,
"caData": "<base64 encoded from vCluster secret>",
"certData": "<base64 encoded from vCluster secret>",
"keyData": "<base64 encoded from vCluster secret>",
"serverName": "argo-cd-k0"
}
}
That's it! Enjoy, and don't forget to star the KubeZero project on GitHub :-)
