Automate adding vCluster to Argo CD using External Secrets Operator - GitOps

Overview

In KubeZero (an open-source out-of-the-box Platform Orchestrator with GitOps designed for multi-environment Cloud Native setup), virtual clusters are created using vCluster. The main GitOps tool used in KubeZero is Argo CD, so we needed to automate provisioning the cluster and adding it to Argo CD.

If you used Argo CD before, you probably know that Argo CD provides a method for declarative setup (like for GitOps) where you can add new K8s clusters credentials by storing them in secrets, just like repositories or repository credentials.

However, to automate that, you need some way to extract the vClusters credentials and format them as an Argo CD config. There are many ways to do that, I prefer to use a declarative method, which is External Secrets Operator, namely PushSecret and ClusterSecretStore.

Flow

The flow is simple: when a K8s cluster is created via vCluster, the cluster credentials are created as a Secret object in the same namespace as the virtual cluster. Then, using PushSecret templating capabilities, it will read the secret, reformat it, and then push it to the Argo CD cluster using ClusterSecretStore.

vCluster supports multiple installation methods. We use vCluster Helm chart, so the PushSecret is created within the Helm chart to further automate it. Using Helm here is not mandatory; you can use any other installation method you like.

Prerequisites

• Host K8s cluster (v1.31)
• Argo CD (v3.0.6)
• External Secrets Operator (v0.17.0)
  • ClusterSecretStore to store the reformated Secret object in Argo CD namespace.

Assuming you deploy the virtual cluster using vCluster (v4.3.0) Helm chart, you just need this extra Helm values file (here I just copy the example from KubeZero repo):

---
experimental:
  deploy:
    host:
      manifestsTemplate: |
        ---
        # Push the vCluster credentails to KubeZero ClusterSecretStore,
        # which will save it as a Secret in the KubeZero namespace to be used as an Argo CD cluster config
        # (just a secret with a specific label).
        # https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
        apiVersion: external-secrets.io/v1alpha1
        kind: PushSecret
        metadata:
          name: argo-cd-{{ .Release.Name }}-credentials
          namespace: {{ .Release.Name }}
        spec:
          refreshInterval: 5m
          secretStoreRefs:
            - name: kubezero-management
              kind: ClusterSecretStore
          selector:
            secret:
              name: vc-{{ .Release.Name }}
          data:
            - match:
                secretKey: name
                remoteRef:
                  remoteKey: argo-cd-{{ .Release.Name }}-credentials
                  property: name
            - match:
                secretKey: server
                remoteRef:
                  remoteKey: argo-cd-{{ .Release.Name }}-credentials
                  property: server
            - match:
                secretKey: config
                remoteRef:
                  remoteKey: argo-cd-{{ .Release.Name }}-credentials
                  property: config
          template:
            engineVersion: v2
            metadata:
              annotations:
                managed-by: external-secrets
              labels:
                argocd.argoproj.io/secret-type: cluster
            data:
              name: {{ .Release.Name }}
              server: https://{{ .Release.Name }}.{{ .Release.Namespace }}.svc:443
              config: |
                {
                  "tlsClientConfig": {
                    "insecure": false,
                    "caData": "{{ printf "{{ index . "certificate-authority" | b64enc }}" }}",
                    "certData": "{{ printf "{{ index . "client-certificate" | b64enc }}" }}",
                    "keyData": "{{ printf "{{ index . "client-key" | b64enc }}" }}",
                    "serverName": "{{ .Release.Name }}"
                  }
                }

That will create the reformated Secret object in the Argo CD namespace, where the Argo CD controller will read it as an config because of the lable argocd.argoproj.io/secret-type: cluster. The actual output will be something like this:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    managed-by: external-secrets
  labels:
    argocd.argoproj.io/secret-type: cluster
  name: argo-cd-k0-credentials
  namespace: argo-cd
# The base64 is decoded for the sake of the example.
data:
  name: argo-cd-k0
  server: https://argo-cd-k0.mgmt-demo.svc:443
  config: |
    {
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 encoded from vCluster secret>",
        "certData": "<base64 encoded from vCluster secret>",
        "keyData": "<base64 encoded from vCluster secret>",
        "serverName": "argo-cd-k0"
      }
    }

That's it! Enjoy, and don't forget to star the KubeZero project on GitHub :-)