System monitoring with Grafana (InfluxDB, CollectD)

Introduction

System monitoring with Grafana allows you to query, visualize, alert on and understand your metrics. We live in a world of data, big data. Tracking everything in logs or monitoring tools is somewhat tiring. Especially if you need constant input. Grafana has some disadvantages when compared with Kibana (for instance full-text data search), or it’s maybe better to say that they have a different purposes. Grafana is great with metrics analysis, compared with Kibana which is more for exploring logged data. All in all, it’s an eye opener. Pleasing esthetics, ease of use, you’ll hardly want to go back to a rusty old terminal.. on the other hand some habits die hard..

As mentioned, grafana is here just for esthetics, a data representation tool. We need couple more things, like a collector (that gathers all the data points from a system/service) and a storage service compatible with grafana (a data source). As for the collectors, there are many options:

• statsd
• Telegraf
• collectd
• zabbix
• NetData
• Munin

Officially supported datasources for grafana:

• Graphite
• Elasticsearch
• CloudWatch
• InfluxDB
• OpenTSDB
• Prometheus
• MySQL
• Postgres
• Microsoft SQL Server (MSSQL)

Here, we’ll rely on collectd & InfluxDB. The thing we’ll make, in rough sketch:

InfluxDB Setup

We’ll get things ready before we turn to collectd. Install InfluxDB:

Debian/Ubuntu

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Red Hat/CentOS

# cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF

# sudo yum install influxdb
# sudo service influxdb start or systemctl start influxdb (if you're using systemd)

Steps are also available on Influx site. After installation edit /etc/influxdb/influxdb.conf , set:

[http]
enabled=true
bind-address=":8086"

[collectd]
enabled = true
bind-address = ":8096"
database = "collectd"
typesdb = "/usr/share/collectd/types.db"

We defined input/output points and we’re ready for the collectd. Before you continue, start/restart influx:

# service influxdb start [or restart]

Collectd Setup

InfluxDB is not waiting for our data, continue with Collectd:

Red Hat/CentOS

Through EPEL repository:

# yum install epel-release
# yum install collectd

Debian/Ubuntu

# sudo apt-get install collectd

Adjustments

Collectd generates/gathers the stats (System/Services), but we need to put it somewhere for grafana to use it. We need to define which data to collect, and where to put it. Somewhere in /etc is collectd config. Probably /etc/collectd/collectd.conf :

Hostname "YourHostNameHere"
FQDNLookup true
BaseDir "/var/lib/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib64/collectd"
TypesDB "/usr/share/collectd/types.db"

With basic info set, we now need to enable some plugins. For example: syslog, disk, interface, disk, load, memory, network are some most frequently used, so uncomment them. Aside from network, default values are fine. For network, the plugin will send data to some collector (InfluxDB in this case), so we’ll need to point it towards our InfluxDB server:

<Plugin network>
Server "127.0.0.1" "8096"
</Plugin>

Don’t forget to start/restart the service:

# service collectd start [or restart]

Grafana Setup

System monitoring with Grafana is great, even with all this “hassle”, I’m still optimistic. For any additional help, if needed, check grafana documentation.

Grafana Ubuntu [Debian)

Add the following line to your /etc/apt/sources.list file.

# deb https://packagecloud.io/grafana/stable/debian/ stretch main

Then add the Package Cloud key. This allows you to install signed packages.

# curl https://packagecloud.io/gpg.key | sudo apt-key add -

Update your Apt repositories and install Grafana

# sudo apt-get update
# sudo apt-get install grafana

Grafana CentOS [RedHat]

Directly, find a path/file on grafana website:

$ sudo yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.1.4-1.x86_64.rpm

or through YUM repository. Add the following to a new file at /etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Then install Grafana via the yum command.

$ sudo yum install grafana

Adjustments

Grafana is listening by default on the port 3000. Check your firewall setup if needed. Default config file is somewhere in /etc/grafana* , most likely on /etc/grafana/grafana.ini . There are a number of options to play with, so look around. Something useful that comes into mind is setting a nginx pass thing (a Reverse Proxy), so your grafana shows on a neat path like: <domain>/grafana/ .

root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/

or

#root_url = http://localhost:3000
#root_url = %(protocol)s://%(domain)s:/grafana/

and within Nginx config:

location /grafana/ {
  proxy_pass http://<ip/domain>:3000/;
  rewrite ^/grafana/(.*) /$1 break;
  proxy_set_header Host $host;
}

When you’re done with the adjustments, go login to grafana (<ip/domain>:3000). you can create your own dashboard, or you can download it from grafana labs. For example, here we’ll use Host Overview. When you login to your grafana, add the datasource/InfluxDB we’ve already configure above:

• Type: InfluxDB
• URL: http://localhost:8086
• Database: collectd

Import the dashboard we mentioned earlier (Host Overview), and with that you should immediately see what collectd gathered in InfluxDB:

Three types of Grafana plugins :

1. Panel plugins – These allow new data visualization to be added to Grafana.
2. Data Source – Grafana Data Source plugins.
3. App – Bundles of panels, data source, dashboard with new UI.

To install, Grafan provides a cmd tool grafana-cli:

# grafana-cli

Get a list of all available plugins in the repository:

# grafana-cli plugins list-remote

To install:

# grafana-cli plugins install <plugin>

Restart grafana after you finish installing plugin(s):

# service restart grafana-server

To list installed plugins on your system:

# grafana-cli plugins ls

To remove plugin:

# grafana-cli plugins remove <plugin>

Grafana SSL setup

To setup SSL (Let’s encrypt in this case) you should add SSL keys in your /etc/grafana/grafana.ini config file. Grafana probably doesn’t have enough permissions to access those files directly, and a quick solution is to simply copy those keys/files to some custom folder within grafana’s reach (alter permissions):

protocol = https

cert_file = /etc/grafana/ssl/fullchain.pem
cert_key = /etc/grafana/ssl/privkey.pem

Restart grafana-server.

Grafana – Restart Admin Password

In case you need this (as I did), you can try:

# grafana-cli admin reset-admin-password --homepath "/usr/share/grafana/" newpass

but that didn’t work for me at the time. The approach that did:

1. stop grafana
2. setup admin user and password in grafana.ini
3. delete /var/lib/grafana/grafana.db
4. restart

Conclusion

There is definitely a lot of things to cover here (CollectD and different DBs/Data sources, setup of different sources Grafana could use, etc.) . This was just a quick intro, so you could get a rough picture on how things work. We’ll try to keep things here fresh by updating some info as we dig up something interesting. Most likely there will be a lot of  new/related posts, so stay tunned.

Things to work on:

• Secure your ports/access
• Play around with Plugins
• Monitor Nginx or some other service/log
• Maintenance & Limits (InfluxDB/CollectD)
• Vulnerability check: Grafana, InfluxDB, CollectD