Control collecting Prometheus client library metrics - Golang

Recently I played with Prometheus more deeply. I wanted to expose some metrics from a REST service written in Golang. I needed to deal with Prometheus instrumentation (it's the way you expose your metrics in Prometheus format).

If you are new to Prometheus, what you do is simply using Prometheus client library, you gather metrics in your app/service and expose them in an endpoint (by convention it's /metrics). Then you scrape that endpoint with Prometheus to collect the metrics.

So what is the problem?
The problem is default Prometheus HTTP handler exposes some internal metrics about the client library! In Golang they are ~35 metrics in addition to your app/service metrics.

Those internal metrics are really helpful for debugging, but mostly you don't really need them every day as your main app/service metrics. Of course you can drop that metrics from Prometheus server side. Also there is a way to make "only" your app/service metrics available.

But what's better? Having the control to toggle client library metrics and enable them whenever you want! Let's take an example for that approach. This simple golang app has 2 endpoints.

  • First one is /count which suppose to have the business logic.
  • Second one is /metrics which should expose a single metric in Prometheus format.

So whenever you hit /count endpoint, you should see an increase of the counter metric which's called my_app_http_hit_total. Also Prometheus client library metrics could be exposed using env var is called PROMETHEUS_CLIENT_LIBRARY_METRICS.

Here is a working example:

package main

import (

var (
 // Initial count.
 currentCount = 0

 // If value of env var "PROMETHEUS_CLIENT_LIBRARY_METRICS" is true, set this var to be "true", if not it's "fasle".
 collectPromClientLibMetrics = strings.ToLower(os.Getenv("PROMETHEUS_CLIENT_LIBRARY_METRICS")) == "true"

 // Prometheus Registry to register metrics.
 prometheusRegistry = prometheus.NewRegistry()

 // The Prometheus metric that will be exposed.
 httpHits = prometheus.NewCounter(
   Name: "my_app_http_hit_total",
   Help: "Total number of http hits.",

 // Add all metrics that will be resisted
 metricsList = []prometheus.Collector{

func init() {
 // Register metrics that will be exposed.

// Count, display, log number of hits.
func count() func(w http.ResponseWriter, r *http.Request) {
 return func(w http.ResponseWriter, r *http.Request) {
  message := fmt.Sprint("Current count is: ", currentCount)

// HTTP handler for prometheus metrics. The handler return registered metrics.
// Also internal metrics from prometheus lib could be returned like golang gc and process.
func metricsHandler() http.Handler {
 if !collectPromClientLibMetrics {
  return promhttp.HandlerFor(
    ErrorHandling: promhttp.ContinueOnError,
 return promhttp.Handler()

func main() {
 r := mux.NewRouter()
 r.Handle("/metrics", metricsHandler())
 r.HandleFunc("/count", count())
 log.Fatal(http.ListenAndServe(":8080", r))

Let's try this:

$ go run main.go

$ curl localhost:8080/count
Current count is: 1

$ curl localhost:8080/metrics
# HELP my_app_http_hit_total Total number of http hits.
# TYPE my_app_http_hit_total counter
my_app_http_hit_total 1

Try to repeat it again but let's collect the internal metrics this time:
$ go run main.go

In addition to your metric "my_app_http_hit_total", you will get a full list of internal lib metrics like:


That's it!
Now you have more control about the emitted internal metrics and you can turn on/off them as you need.

Powered by Blogger.

Hello, my name is Ahmed AbouZaid, I'm a passionate Tech Lead DevOps Engineer. 👋

With 16+ years of open-source contributions, 12+ years of professional hands-on experience in DevOps, and an M.Sc. in Data Engineering from Edinburgh Napier University (UK), I enjoy facilitating the growth of both businesses and individuals.

I specialize in Cloud-Native and Kubernetes. I'm also a Free/Open source geek and book author. My favorite topics are DevOps transformation, automation, data, and metrics.

Contact Me


Email *

Message *

Start Your DevOps Engineer Journey!

Start Your DevOps Engineer Journey!
Start your DevOps career for free the Agile way in 2024 with the Dynamic DevOps Roadmap ⭐

Latest Post

2023 Highlights

Image generated with Craiyon . Finally, 2023 is over! What a year! One more crazy year, but it was the culmination ...

Popular Posts

Blog Archive