prometheus – Tarik Billa

How to rename label within a metric in Prometheus

April 9, 2024 by Tarik

you can use the label_replace function in promQL, but it also add the label, don’t replace it label_replace( <vector_expr>, “<desired_label>”, “$1”, “<existing_label>”, “(.+)” ) label_replace( node_systemd_unit_state{instance=”server-01″,job=”node-exporters”,name=”kubelet.service”,state=”active”}, “unit_name”,”$1″,”name”, “(.+)” ) So, to avoid the repetition you can add: sum(label_replace( node_systemd_unit_state{instance=”server-01″,job=”node-exporters”,name=”kubelet.service”,state=”active”}, “unit_name”,”$1″,”name”, “(.+)” ) )by(unit_name)

How to add https url on target prometheus

April 6, 2024 by Tarik

Use the following configuration: – job_name: ‘domain-job’ scheme: https static_configs: – targets: [‘myDomain.dev’]

Monitor custom kubernetes pod metrics using Prometheus

April 2, 2024 by Tarik

You have to add this three annotation to your pods: prometheus.io/scrape: ‘true’ prometheus.io/path: ‘/data/metrics’ prometheus.io/port: ’80’ How it will work? Look at the kubernetes-pods job of config-map.yaml you are using to configure prometheus, – job_name: ‘kubernetes-pods’ kubernetes_sd_configs: – role: pod relabel_configs: – source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true – source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ … Read more

How to display all metrics that don’t have a specific label

January 4, 2024 by Tarik

Try this: {__name__=~”.+”,container=””} There needs to be at least one non-empty matcher (hence the + in the __name__ regular expression, * wouldn’t cut it). And the way you query for a missing label is by checking for equality with the empty string.

Get total and free disk space using Prometheus

January 3, 2024 by Tarik

According to my Grafana dashboard, the following metrics work nicely for alerting for available space, 100 – ((node_filesystem_avail_bytes{mountpoint=”https://stackoverflow.com/”,fstype!=”rootfs”} * 100) / node_filesystem_size_bytes{mountpoint=”https://stackoverflow.com/”,fstype!=”rootfs”}) The formula gives out the percentage of available space on the pointed disk. Make sure you include the mountpoint and fstype within the metrics.

How dangerous are high-cardinality labels in Prometheus?

December 25, 2023 by Tarik

High-cardinality labels (e.g. labels with big number of unique values) aren’t dangerous on their own. The danger is in the total number of active time series. A single Prometheus instance can handle up to ten millions of active time series according to https://www.robustperception.io/why-does-prometheus-use-so-much-ram when running on a host with >100GB of RAM. An example: suppose … Read more

Prometheus (in Docker container) Cannot Scrape Target on Host

December 24, 2023 by Tarik

While not a very common use case.. you can indeed connect from your container to your host. From https://docs.docker.com/docker-for-mac/networking/ I want to connect from a container to a service on the host The host has a changing IP address (or none if you have no network access). From 18.03 onwards our recommendation is to connect … Read more

increase() in Prometheus sometimes doubles values: how to avoid?

December 23, 2023 by Tarik

This is known as aliasing and is a fundamental problem in signal processing. You can improve this a bit by increasing your sample rate, a 4m range is a bit short with a 2m range. Try a 10m range. Here for example the query executed at 1515722220 only sees the [email protected] and [email protected] samples. That’s … Read more

Why there are both counters and gauges in Prometheus if gauges can act as counters?

December 13, 2023 by Tarik

From a conceptual point of view, gauge and counter have different purposes a gauge typically represent a state, usually with the purpose of detecting saturation. the absolute value of a counter is not really meaningful, the real purpose is rather to compute an evolution (usually a utilization) with functions like irate/rate(), increase() … Those evolution … Read more

Prometheus vs ElasticSearch. Which is better for container and server monitoring? [closed]

December 10, 2023 by Tarik

ELK is a general-purpose no-sql stack that can be used for monitoring. We’ve successfully deployed one on production and used it for some aspects of our monitoring system. You can ship metrics into it (if you wish) and use it to monitor them, but its not specifically designed to do that. Nor does the non-commercial … Read more