Prometheus Notes: Difference between revisions
Line 157: | Line 157: | ||
</pre> | </pre> | ||
== | == On Cardinality and Metrics == | ||
https://www.robustperception.io/which-are-my-biggest-metrics | references: | ||
* https://www.robustperception.io/which-are-my-biggest-metrics | |||
* https://promlabs.com/blog/2020/12/17/promql-queries-for-exploring-your-metrics/ | |||
topk(10, count by (__name__)({__name__=~".+"})) | topk(10, count by (__name__)({__name__=~".+"})) |
Revision as of 14:21, 5 June 2023
PromQL
node exporter:
node_memory_MemAvailable_bytes{job=~"myjob.*"} / on ( instance ) node_memory_MemTotal_bytes{job=~"myjob.*"}
node_memory_MemFree_bytes{job=~"myjob.*"} / on ( instance ) node_memory_MemTotal_bytes{job=~"myjob.*"}
sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_capacity_cpu_cores) * 100
topk( 10, count({job="prometheus"}) by (__name__) )
renaming metrics
scrape_configs: - job_name: sql targets: [172.21.132.39:41212] metric_relabel_configs: - source_labels: ['prometheus_metric_name'] target_label: '__name__' regex: '(.*[^_])_*' replacement: '${1}' - regex: prometheus_metric_name action: labeldrop
turns this:
query_result_dm_os_performance_counters{ counter_instance="ex01", counter_name="log file(s) size (kb)", prometheus_metric_name="sqlserver_databases", }
into :
sqlserver_databases{ counter_instance="ex01", counter_name="log file(s) size (kb)", }
dirty install node exporter
curl -L -o /tmp/node_exporter-1.0.1.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz tar zxvf /tmp/node_exporter-1.0.1.linux-amd64.tar.gz -C /tmp/ cp /tmp/node_exporter-1.0.1.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter curl -L -o /tmp/node_exporter-1.0.1.linux-armv6.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-armv6.tar.gz tar zxvf /tmp/node_exporter-1.0.1.linux-armv6.tar.gz cp /tmp/node_exporter-1.0.1.linux-armv6/node_exporter /usr/bin/prometheus-node-exporter chmod 755 /usr/bin/prometheus-node-exporter chown root:root /usr/bin/prometheus-node-exporter cat << EOF > /etc/default/prometheus-node-exporter ARGS="--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ \ --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \ --collector.netclass.ignored-devices=^lo$ \ --collector.systemd EOF chown root:root /etc/default/prometheus-node-exporter chmod 644 /etc/default/prometheus-node-exporter cat << EOF > /lib/systemd/system/prometheus-node-exporter.service [Unit] Description=Prometheus exporter for machine metrics Documentation=https://github.com/prometheus/node_exporter [Service] Restart=always User=nobody EnvironmentFile=/etc/default/prometheus-node-exporter ExecStart=/usr/bin/prometheus-node-exporter $ARGS ExecReload=/bin/kill -HUP $MAINPID TimeoutStopSec=20s SendSIGKILL=no [Install] WantedBy=multi-user.target EOF chown root:root /lib/systemd/system/prometheus-node-exporter.service chmod 644 /lib/systemd/system/prometheus-node-exporter.service systemctl daemon-reload systemctl enable prometheus-node-exporter.service systemctl start prometheus-node-exporter.service
Exposing metrics
with python use:
import prometheus_client
example output:
# HELP go_memstats_frees_total Total number of frees. # TYPE go_memstats_frees_total counter go_memstats_frees_total 21217 # HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. # TYPE go_memstats_gc_sys_bytes gauge go_memstats_gc_sys_bytes 307200
cpu usage from cpu seconds
usage by job:
100 - (avg by (job) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
reference: https://www.robustperception.io/understanding-machine-cpu-usage
cont. cpus on system by job:
avg(count(node_cpu_seconds_total)without (cpu))by(job)
when you have tagged you node pools by colour:
100 - (avg by (colour) (irate(node_cpu_seconds_total{job="kubernetes-node-exporter",mode="idle"}[5m])) * 100)
Plotting more than one metrics
label_replace( max(process_open_handles{kubernetes_namespace="mynamespace"}), "aggregation", "max", "", "" ) or label_replace( quantile(0.95, process_open_handles{kubernetes_namespace="mynamespace"}), "aggregation", "p95", "", "" ) or label_replace( quantile(0.5, process_open_handles{kubernetes_namespace="mynamespace"}), "aggregation", "p50", "", "" ) or label_replace( avg(process_open_handles{kubernetes_namespace="mynamespace"}), "aggregation", "avg", "", "" )
On Cardinality and Metrics
references:
- https://www.robustperception.io/which-are-my-biggest-metrics
- https://promlabs.com/blog/2020/12/17/promql-queries-for-exploring-your-metrics/
topk(10, count by (__name__)({__name__=~".+"}))
agg by job:
topk(10, count by (__name__, job)({__name__=~".+"}))
From: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/889
It is also useful to perform the following query in order to determine scrape targets exporting the maximum number of metrics:
max_over_time(scrape_samples_scraped[1h])
topk(10,sum(max_over_time(scrape_samples_scraped[1h]))by(job))
synonym?
topk(10, count by (job)({__name__=~".+"}))
Additionally to that the following query could be useful for determining scrape targets that introduce the most of time series churn rate:
max_over_time(scrape_series_added[1h])
topk(10,sum(max_over_time(scrape_series_added[1h]))by(job))
check your config
you have to do this between the time the container comes up and when it fails because it read the config file.
kns exec "cluster-metrics-prometheus-server-0" -c prometheus-server -- /bin/promtool check config /etc/config/prometheus.yml
kns is just kubectl that uses the $KNS env var for it's namespace. A little hack I created.
help
resources
https://timber.io/blog/promql-for-humans/
https://www.weave.works/blog/promql-queries-for-the-rest-of-us/
https://promcon.io/2018-munich/slides/taking-advantage-of-relabeling.pdf
https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085
https://www.robustperception.io/extracting-full-labels-from-consul-tags
https://blog.freshtracks.io/prometheus-relabel-rules-and-the-action-parameter-39c71959354a