Prometheus & Grafana & Netdata

Before a while, I had written a post about monitoring single linux system. It was the magic of netdata that was supplying us those good visuals. You can look here.

Netdata is good for basic info about a single system, but what about an enterprise infrastructure, mostly with lots of machines? Let’s broaden the concept some more: It is not only basic data we want to see, but also metrics of enterprise applications such as cache retrieve miss count of Apache web servers, table IO waits of Mysql servers or DIMM status of physical servers. A central solution could be useful for these all. Here is where Prometheus comes into play.

What is Prometheus? As described here

Prometheus is an open-source systems monitoring and alerting toolkit.

And it is a monitoring system and time series database. Have a look at github!
Some applications supply metrics to Prometheus directly and some application metrics are collected by exporters.

When you first install Prometheus, you get a screen similar to this.

If you write something about metric into the expression area or open dropdown list, you see the metrics being collected. You may list both tabular data or graph of the metric. Prometheus provides a functional expression language (PromQL) that lets the user select and aggregate time series data in real time.

I have been monitoring my computer with netdata. I noticed that (Netdata with Prometheus) netdata is able to supply Prometheus metrics without exporters. The only thing I had to do was add these lines to prometheus configuration data:


  - job_name: 'netdata-scrape'

    metrics_path: "/api/v1/allmetrics?format=prometheus"

    static_configs:
      - targets: ['{ip_of_vm}:19999']

After having Prometheus page with my netdata metric, I used the sample query and had the graph.

This gives me what i need as a sample but there is no need to restrict ourselves though a great visualization tool exists: Grafana.

Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics.

I added Prometheus as a data source. Then in a new dashboard, added `system_cpu_user` as query to a panel. And able to get the graph here:

Grafana offers us a variety of graphs. I added process counts of some users as a graph and gauges.

Grafana officially supports the following data sources:

Graphite
Elasticsearch
CloudWatch
InfluxDB
OpenTSDB
KairosDB
Prometheus

What can be done is rather flexible. If you want to monitor anything in enterprise, if it supports metrics to Prometheus or has exporters, then the rest depends on your imagination. (Of course custom exporters can also be written :))

In this post, I used netdata metrics in Prometheus and then used them in my grafana visual objects. For this working environment, I did not install prometheus and grafana to my computer. Instead, I used docker images prom/prometheus

docker run \
	-p 9090:9090 \
	-v /usr/local/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
	prom/prometheus

and grafana/grafana

 docker run -i -p 3000:3000 grafana/grafana

.

Good day…

References:
Advertisements