Prometheus: grouping metrics by metric names
The following query lists all available metrics: sum by(__name__)({app=”bar”}) Where bar is the application name, as you can see in the log entries posted in the question.
prometheus
Why does increase() return a value of 1.33 in prometheus?
The challenge with calculating this number is that we only have a few data points inside a time range, and they tend not to be at the exact start and end of that time range (1 minute here). What do we do about the time between the start of the time range and the first … Read more
Is there a way to monitor kube cron jobs using prometheus
I’m using these rules with kube-state-metrics: groups: – name: job.rules rules: – alert: CronJobRunning expr: time() -kube_cronjob_next_schedule_time > 3600 for: 1h labels: severity: warning annotations: description: CronJob {{$labels.namespaces}}/{{$labels.cronjob}} is taking more than 1h to complete summary: CronJob didn’t finish after 1h – alert: JobCompletion expr: kube_job_spec_completions – kube_job_status_succeeded > 0 for: 1h labels: severity: warning … Read more
How can I visualize a histogram with Promdash or Grafana?
Grafana v5+ provides direct support for representing Prometheus histograms as heatmap. http://docs.grafana.org/features/panels/heatmap/#histograms-and-buckets Heatmaps are preferred over histogram because a histogram does not show you how the trend changes over time. So if you have a time-series histogram, then use the heatmap panel to picture it. To get you started, here is an example (for Prometheus … Read more
Most recent value or last seen value
All you need is my_metric, which will by default return the most recent value no more than 5 minutes old.
How to use the selected period of time in a query?
There are two ways that I know: You can use the $__interval variable like this: increase(http_requests_total[$__interval]) There is a drawback that the $__interval variable’s value is adjusted by resolution of the graph, but this may also be helpful in some situations. This approach should fit your case better: Go to Dashboard’s Templating settings, create new … Read more
How can I alert for container restarted?
I used the following Prometheus alert rule for finding container restarts in an hour(can be modified to max time), It may be helpful for you. Prometheus Alert Rule Sample ALERT ContainerRestart/PodRestart IF rate(kube_pod_container_status_restarts[1h]) * 3600 > 1 FOR 5s LABELS {action_required = “true”, severity=”critical/warning/info”} ANNOTATIONS {DESCRIPTION=”Pod {{$labels.namespace}}/{{$labels.pod}} restarting more than once during last one hours.”, … Read more
Relabel instance to hostname in Prometheus
I just came across this problem and the solution is to use a group_left to resolve this problem. You can’t relabel with a nonexistent value in the request, you are limited to the different parameters that you gave to Prometheus or those that exists in the module use for the request (gcp,aws…). So the solution … Read more
Getting error “Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused”
Since the targets are not running inside the prometheus container, they cannot be accessed through localhost. You need to access them through the host private IP or by replacing localhost with docker.for.mac.localhost or host.docker.internal. On Windows: host.docker.internal (tested on win10, win11) On Max docker.for.mac.localhost
Monitoring log files using some metrics exporter + Prometheus + Grafana
Take a look at Telegraf. It does support tailing logs using input plugins logparser and tail. To export metrics as prometheus endpoint use prometheus_client output plugin. You also may apply on the fly aggregations. I’ve found it simpler to configure for multiple log files than grok_exporter or mtail