Control collecting Prometheus client library metrics - Golang

Recently I played with Prometheus more deeply. I wanted to expose some metrics from a REST service written in Golang. I needed to deal with Prometheus instrumentation (it's the way you expose your metrics in Prometheus format).

If you are new to Prometheus, what you do is simply using Prometheus client library, you gather metrics in your app/service and expose them in an endpoint (by convention it's /metrics). Then you scrape that endpoint with Prometheus to collect the metrics.

So what is the problem?
The problem is default Prometheus HTTP handler exposes some internal metrics about the client library! In Golang they are ~35 metrics in addition to your app/service metrics.

Those internal metrics are really helpful for debugging, but mostly you don't really need them every day as your main app/service metrics. Of course you can drop that metrics from Prometheus server side. Also there is a way to make "only" your app/service metrics available.

But what's better? Having the control to toggle client library metrics and enable them whenever you want! Let's take an example for that approach. This simple golang app has 2 endpoints.

• First one is /count which suppose to have the business logic.
• Second one is /metrics which should expose a single metric in Prometheus format.

So whenever you hit /count endpoint, you should see an increase of the counter metric which's called my_app_http_hit_total. Also Prometheus client library metrics could be exposed using env var is called PROMETHEUS_CLIENT_LIBRARY_METRICS.

Here is a working example:

package main

import (
 "fmt"
 "github.com/gorilla/mux"
 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
 "log"
 "net/http"
 "os"
 "strings"
)

var (
 // Initial count.
 currentCount = 0

 // If value of env var "PROMETHEUS_CLIENT_LIBRARY_METRICS" is true, set this var to be "true", if not it's "fasle".
 collectPromClientLibMetrics = strings.ToLower(os.Getenv("PROMETHEUS_CLIENT_LIBRARY_METRICS")) == "true"

 // Prometheus Registry to register metrics.
 prometheusRegistry = prometheus.NewRegistry()

 // The Prometheus metric that will be exposed.
 httpHits = prometheus.NewCounter(
  prometheus.CounterOpts{
   Name: "my_app_http_hit_total",
   Help: "Total number of http hits.",
  },
 )

 // Add all metrics that will be resisted
 metricsList = []prometheus.Collector{
  httpHits,
 }
)

func init() {
 // Register metrics that will be exposed.
 prometheusRegistry.MustRegister(httpHits)
}

// Count, display, log number of hits.
func count() func(w http.ResponseWriter, r *http.Request) {
 return func(w http.ResponseWriter, r *http.Request) {
  currentCount++
  message := fmt.Sprint("Current count is: ", currentCount)
  httpHits.Inc()
  w.Write([]byte(message))
  log.Printf(message)
 }
}

// HTTP handler for prometheus metrics. The handler return registered metrics.
// Also internal metrics from prometheus lib could be returned like golang gc and process.
func metricsHandler() http.Handler {
 if !collectPromClientLibMetrics {
  return promhttp.HandlerFor(
   prometheusRegistry,
   promhttp.HandlerOpts{
    ErrorHandling: promhttp.ContinueOnError,
   },
  )
 }
 return promhttp.Handler()
}

func main() {
 r := mux.NewRouter()
 r.Handle("/metrics", metricsHandler())
 r.HandleFunc("/count", count())
 log.Fatal(http.ListenAndServe(":8080", r))
}

Let's try this:

$ go run main.go

$ curl localhost:8080/count
Current count is: 1

$ curl localhost:8080/metrics
# HELP my_app_http_hit_total Total number of http hits.
# TYPE my_app_http_hit_total counter
my_app_http_hit_total 1

Try to repeat it again but let's collect the internal metrics this time:

$ export PROMETHEUS_CLIENT_LIBRARY_METRICS="true"
$ go run main.go

In addition to your metric "my_app_http_hit_total", you will get a full list of internal lib metrics like:

go_threads
go_gc_duration_seconds
go_memstats_alloc_bytes
process_virtual_memory_bytes
process_cpu_seconds_total
promhttp_metric_handler_requests_total

That's it!
Now you have more control about the emitted internal metrics and you can turn on/off them as you need.