monitoring

you can use the label_replace function in promQL, but it also add the label, don’t replace it label_replace( <vector_expr>, “<desired_label>”, “$1”, “<existing_label>”, “(.+)” ) label_replace( node_systemd_unit_state{instance=”server-01″,job=”node-exporters”,name=”kubelet.service”,state=”active”}, “unit_name”,”$1″,”name”, “(.+)” ) So, to avoid the repetition you can add: sum(label_replace( node_systemd_unit_state{instance=”server-01″,job=”node-exporters”,name=”kubelet.service”,state=”active”}, “unit_name”,”$1″,”name”, “(.+)” ) )by(unit_name)

Probably the default scrape_timeout value is too short for you [ scrape_timeout: <duration> | default = 10s ] Set a bigger value for scrape_timeout. scrape_configs: – job_name: ‘prometheus’ scrape_interval: 5m scrape_timeout: 1m Take a look here https://github.com/prometheus/prometheus/issues/1438

Edit the prometheus.service file vi /etc/systemd/system/prometheus.service add “–storage.tsdb.retention.time=1y” below to “ExecStart=/usr/local/bin/prometheus \” line. So the config will look like bellow for 1 year of data retention. [Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ –config.file /etc/prometheus/prometheus.yml \ –storage.tsdb.path /var/lib/prometheus/ \ –web.console.templates=/etc/prometheus/consoles \ –web.console.libraries=/etc/prometheus/console_libraries \ –web.external-url=http://34.89.26.156:9090 \ –storage.tsdb.retention.time=1y [Install] WantedBy=multi-user.target