Try this: {__name__=~”.+”,container=””} There needs to be at least one non-empty matcher (hence the + in the __name__ regular expression, * wouldn’t cut it). And the way you query for a missing label is by checking for equality with the empty string.
High-cardinality labels (e.g. labels with big number of unique values) aren’t dangerous on their own. The danger is in the total number of active time series. A single Prometheus instance can handle up to ten millions of active time series according to https://www.robustperception.io/why-does-prometheus-use-so-much-ram when running on a host with >100GB of RAM. An example: suppose … Read more
This is known as aliasing and is a fundamental problem in signal processing. You can improve this a bit by increasing your sample rate, a 4m range is a bit short with a 2m range. Try a 10m range. Here for example the query executed at 1515722220 only sees the [email protected] and [email protected] samples. That’s … Read more
The following query lists all available metrics: sum by(__name__)({app=”bar”}) Where bar is the application name, as you can see in the log entries posted in the question.
The challenge with calculating this number is that we only have a few data points inside a time range, and they tend not to be at the exact start and end of that time range (1 minute here). What do we do about the time between the start of the time range and the first … Read more
All you need is my_metric, which will by default return the most recent value no more than 5 minutes old.
I used the following Prometheus alert rule for finding container restarts in an hour(can be modified to max time), It may be helpful for you. Prometheus Alert Rule Sample ALERT ContainerRestart/PodRestart IF rate(kube_pod_container_status_restarts[1h]) * 3600 > 1 FOR 5s LABELS {action_required = “true”, severity=”critical/warning/info”} ANNOTATIONS {DESCRIPTION=”Pod {{$labels.namespace}}/{{$labels.pod}} restarting more than once during last one hours.”, … Read more
I just came across this problem and the solution is to use a group_left to resolve this problem. You can’t relabel with a nonexistent value in the request, you are limited to the different parameters that you gave to Prometheus or those that exists in the module use for the request (gcp,aws…). So the solution … Read more
The endpoint for that is http://localhost:9090/api/v1/label/__name__/values API Reference
You currently can’t configure the metrics_path per target within a job but you can create separate jobs for each of your targets so you can define metrics_path per target. Your config file would look something like this: scrape_configs: – job_name: ‘example-target-1’ scrape_interval: 5s metrics_path: /target-1-path-to-metrics static_configs: – targets: [‘localhost:8090’] labels: group: ‘dummy’ – job_name: ‘example-target-2’ … Read more