Prometheus became the de facto standard for Kubernetes monitoring. And either you use Prometheus Operator or manage Prometheus yourself on K8s, it works great.
Many applications now expose metrics in Prometheus format. However, some applications don't follow Prometheus best practices for metric and label naming. And therefore, you can end up with unclear metrics in your Prometheus!
Actually, that was the case with one of my favorite tools, Kubernetes External Secrets. It was exposing metrics called sync_calls and last_state, which don't even tell where they are coming from!
Of course, fixing issues in the upstream is the best (and that what I did in the PR no. #540 Better naming for Prometheus metrics), but if you are working with open-source, you already know that things take time. Thus, I needed to fix it in my cluster.
So the idea is simple, using K8s __meta_ labels and Prometheus relabeling capabilities to rewrite metric names.
Prerequisite
I highly recommend to read the blog post Life of a Label to get better understanding of the difference between relabel_configs and metric_relabel_configs. It's a great post with flowcharts that makes it easier to digest the flow.
Prometheus config
Let's start with prometheus.yml config file. Here is an example for prefix, and the same goes for suffix.
scrape_configs:
- job_name: my-cluster-kubernetes-service-endpoints
[...]
relabel_configs:
# First step: Convert Kubernetes annotation 'prometheus.io/metrics-prefix' to an interim label.
# That's needed because all '__meta_*' labels will be removed and not available to metric_relabel_configs.
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metrics_prefix]
separator: ;
regex: (.*)
target_label: metrics_prefix
replacement: $1
action: replace
metric_relabel_configs:
# Second step: Concatenate the metric name with the prefix and save it as a new metric name.
- source_labels: [metrics_prefix, __name__]
separator: ;
regex: (.+);(.+)
target_label: __name__
replacement: ${1}_${2}
action: replace
# Third step: Drop the interim label since there is no actual need for it.
- separator: ;
regex: metrics_prefix
replacement: $1
action: labeldrop
Kubernetes config
Now we just need to add the annotation to the application service
apiVersion: v1
kind: Service
metadata:
name: kubernetes-external-secrets
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3001"
prometheus.io/metrics-prefix: "kubernetes_external_secrets"
[...]
By doing that, all metrics exposed by that app will get the configured prefix. Therefore, if there is a metric call sync_calls, then it will be kubernetes_external_secrets_sync_calls.
That's it! Prometheus relabeling is a powerful tool to rewrite not only the labels but also metric names dynamically.
It's worthwhile to mention that the relabeling is not needed anymore for Kubernetes External Secrets since it's been fixed already.
