Metrics collision with internal metrics

[scythargon]

Hi! I'm getting the same error whenever I download a prebuilt executable or run in via docker image:

When I open http://localhost:9108/metrics

An error has occurred while serving metrics:

6 error(s) occurred:
* collected metric process_cpu_seconds_total counter:<value:0.2 >  has help "Total user and system CPU time spent in seconds." but should have "Graphite metric process_cpu_seconds_total"
* collected metric process_virtual_memory_bytes gauge:<value:7.31475968e+08 >  has help "Virtual memory size in bytes." but should have "Graphite metric process_virtual_memory_bytes"
* collected metric process_resident_memory_bytes gauge:<value:1.3062144e+07 >  has help "Resident memory size in bytes." but should have "Graphite metric process_resident_memory_bytes"
* collected metric process_start_time_seconds gauge:<value:1.63203198914e+09 >  has help "Start time of the process since unix epoch in seconds." but should have "Graphite metric process_start_time_seconds"
* collected metric process_open_fds gauge:<value:12 >  has help "Number of open file descriptors." but should have "Graphite metric process_open_fds"
* collected metric process_max_fds gauge:<value:1.048576e+06 >  has help "Maximum number of open file descriptors." but should have "Graphite metric process_max_fds"

And I'm also unable to build it from source:

WARN [runner] Can't run linter goanalysis_metalinter: buildir: failed to load package tsdb: could not load export data: no export data for "github.com/prometheus/prometheus/tsdb" 
ERRO Running error: buildir: failed to load package tsdb: could not load export data: no export data for "github.com/prometheus/prometheus/tsdb" 
make: *** [Makefile.common:192: common-lint] Error 3

Thank you!

[matthiasr]

Not sure about the build error (and don't have access to a computer to reproduce until next week). Does go build ./cmd/graphite_exporter work?

For the metric problem, I think you have a collision between the metrics of the exporter itself (e.g. how much CPU the exporter has used) and metrics ingested via Graphite. Can you share your mappings and what metrics you are sending?

The immediate workaround/fix is to use (map to) different metric names for your metrics. In principle we could split the exporter-internal metrics from the mapped metrics, but this is a very invasive change that affects everyone, so I would rather not.

[matthiasr]

I changed the title to reflect what is happening, since the exporter does run. I don't believe this is a build issue.

[matthiasr]

X-ref #79 which is about collisions between different mapped metrics

[scythargon]

Hi @matthiasr and thank your for your quick response!
Now that makes sense!
I was using Python client's Graphite bridge, which was overriding these metrics and producing this error, now this mapping file has solved it for me:

mappings:
  - match: "process_.*"
    match_type: regex
    action: drop
    name: "dropped"

go build ./cmd/graphite_exporter Also finished without errors. I'm not familiar with Go development - so have no idea what is happening here.

[scythargon]

Can I also ask another thing here - either Graphite Bridge or Graphite Exporter turns metric names from

ia_satisfaction_histogram_sum{facility_slug="Fox Trail Senior Living"} 1.0

into

ia_satisfaction_histogram_sum_facility_slug_Fox_Trail_Senior_Living 1

It there a way I can prevent it and keep them in Prometheus curly brackets format?

Thank you!

[matthiasr]

Ah, that explains it. I would strongly discourage this – the primary purpose of the Prometheus client libraries is to be scraped directly by Prometheus. The Graphite bridge is intended to help the transition if you have a Graphite server that should hold the metrics.

When you use the Graphite bridge with the Graphite exporter, you transform from the Prometheus semantics into Graphite semantics and back, this is inherently lossy.

I am going to close the issue, since the collision happens due to this unrecommended chaining of formats. Please reach out to the regular community channels who will be able to help with alternative solutions for your use case.