Use Kubernetes annotations to relabel Prometheus metrics with prefix/suffix - Monitoring

Prometheus became the de facto standard for Kubernetes monitoring. And either you use Prometheus Operator or manage Prometheus yourself on K8s, it works great.

Many applications now expose metrics in Prometheus format. However, some applications don't follow Prometheus best practices for metric and label naming. And therefore, you can end up with unclear metrics in your Prometheus!

Actually, that was the case with one of my favorite tools, Kubernetes External Secrets. It was exposing metrics called sync_calls and last_state, which don't even tell where they are coming from!

Of course, fixing issues in the upstream is the best (and that what I did in the PR no. #540 Better naming for Prometheus metrics), but if you are working with open-source, you already know that things take time. Thus, I needed to fix it in my cluster.

So the idea is simple, using K8s __meta_ labels and Prometheus relabeling capabilities to rewrite metric names.


I highly recommend to read the blog post Life of a Label to get better understanding of the difference between relabel_configs and metric_relabel_configs. It's a great post with flowcharts that makes it easier to digest the flow.

Prometheus config

Let's start with prometheus.​yml config file. Here is an example for prefix, and the same goes for suffix.

- job_name: my-cluster-kubernetes-service-endpoints
  # First step: Convert Kubernetes annotation 'prometheus.io/metrics-prefix' to an interim label.
  # That's needed because all '__meta_*' labels will be removed and not available to metric_relabel_configs.
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metrics_prefix]
    separator: ;
    regex: (.*)
    target_label: metrics_prefix
    replacement: $1
    action: replace
  # Second step: Concatenate the metric name with the prefix and save it as a new metric name.
  - source_labels: [metrics_prefix, __name__]
    separator: ;
    regex: (.+);(.+)
    target_label: __name__
    replacement: ${1}_${2}
    action: replace
  # Third step: Drop the interim label since there is no actual need for it.
  - separator: ;
    regex: metrics_prefix
    replacement: $1
    action: labeldrop

Kubernetes config

Now we just need to add the annotation to the application service

apiVersion: v1
kind: Service
  name: kubernetes-external-secrets
    prometheus.io/scrape: "true"
    prometheus.io/port: "3001"
    prometheus.io/metrics-prefix: "kubernetes_external_secrets"

By doing that, all metrics exposed by that app will get the configured prefix. Therefore, if there is a metric call sync_calls, then it will be kubernetes_external_secrets_sync_calls.

That's it! Prometheus relabeling is a powerful tool to rewrite not only the labels but also metric names dynamically.

It's worthwhile to mention that the relabeling is not needed anymore for Kubernetes External Secrets since it's been fixed already.

Powered by Blogger.

Hello, my name is Ahmed AbouZaid, I'm a passionate Tech Lead DevOps Engineer. 👋

With 16+ years of open-source contributions, 12+ years of professional hands-on experience in DevOps, and an M.Sc. in Data Engineering from Edinburgh Napier University (UK), I enjoy facilitating the growth of both businesses and individuals.

I specialize in Cloud-Native and Kubernetes. I'm also a Free/Open source geek and book author. My favorite topics are DevOps transformation, automation, data, and metrics.

Contact Me


Email *

Message *

Start Your DevOps Engineer Journey!

Start Your DevOps Engineer Journey!
Start your DevOps career for free the Agile way in 2024 with the Dynamic DevOps Roadmap ⭐

Latest Post

Gomplate v4 is here! - Tools

This year, one of my nice discoveries was gomplate , a fast template renderer supporting many data sources and hundre...

Popular Posts

Blog Archive