Use Kubernetes annotations to relabel Prometheus metrics with prefix/suffix - Monitoring

Prometheus becomes the de facto standard for Kubernetes monitoring. And either you use Prometheus Operator or manage Prometheus yourself on K8s, it works great.

Many applications now expose metrics in Prometheus format. However, some applications don't follow Prometheus best practices for metric and label naming. And therefore, you can end up with unclear metrics in your Prometheus!

Actually, that was the case with one of my favorite tools, Kubernetes External Secrets. It was exposing metrics called sync_calls and last_state, which don't even tell where they are coming from!

Of course, fixing issues in the upstream is the best (and that what I did in the PR no. #540 Better naming for Prometheus metrics), but if you are working with open-source, you already know that things take time. Thus, I needed to fix it in my cluster.

So the idea is simple, using K8s __meta_ labels and Prometheus relabeling capabilities to rewrite metric names.


I highly recommend to read the blog post Life of a Label to get better understanding of the difference between relabel_configs and metric_relabel_configs. It's a great post with flowcharts that makes it easier to digest the flow.

Prometheus config

Let's start with prometheus.​yml config file. Here is an example for prefix, and the same goes for suffix.

- job_name: my-cluster-kubernetes-service-endpoints
  # First step: Convert Kubernetes annotation 'prometheus.io/metrics-prefix' to an interim label.
  # That's needed because all '__meta_*' labels will be removed and not available to metric_relabel_configs.
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metrics_prefix]
    separator: ;
    regex: (.*)
    target_label: metrics_prefix
    replacement: $1
    action: replace
  # Second step: Concatenate the metric name with the prefix and save it as a new metric name.
  - source_labels: [metrics_prefix, __name__]
    separator: ;
    regex: (.+);(.+)
    target_label: __name__
    replacement: ${1}_${2}
    action: replace
  # Third step: Drop the interim label since there is no actual need for it.
  - separator: ;
    regex: metrics_prefix
    replacement: $1
    action: labeldrop

Kubernetes config

Now we just need to add the annotation to the application service

apiVersion: v1
kind: Service
  name: kubernetes-external-secrets
    prometheus.io/scrape: "true"
    prometheus.io/port: "3001"
    prometheus.io/metrics-prefix: "kubernetes_external_secrets"

By doing that, all metrics exposed by that app will get the configured prefix. Therefore, if there is a metric call sync_calls, then it will be kubernetes_external_secrets_sync_calls.

That's it! Prometheus relabeling is a powerful tool to rewrite not only the labels but also metric names dynamically.

It's worthwhile to mention that the relabeling is not needed anymore for Kubernetes External Secrets since it's been fixed already.

Powered by Blogger.

Hello, my name is Ahmed AbouZaid and this is my "lite" technical blog!

I'm a passionate DevOps engineer, Cloud/Kubernetes specialist, Free/Open source geek, and an author.

I believe in self CI/CD (Continuous Improvements/Development), also that "the whole is greater than the sum of its parts".

DevOps transformation, automation, data, and metrics are my preferred areas. And I like to help both businesses and people to grow.

Popular Posts