Monitoring a Raspberry Pi with Prometheus & node_exporter in Dokku

The last weeks, I've been having some issues with my Raspberry Pi, which unexpectedly stopped working after some days, so I asked a question on twitter, and the following stack was one of the proposals.

Prometheus is one of the most popular options when you think in a monitoring solution (time series based, alerting, etc), and one of their sub-projects is node_exporter, which shows a list of metrics that will be consumed by Prometheus.

Dokku is The smallest PaaS implementation you've ever seen, and I've blogged about it several times because I'm using it for all my personal projects hosted in the smallest dropplet in Digital Ocean (5$/month).

I've divided this article in two different posts, and these are all the parts:

1. Export the metrics from the Raspberry Pi using node_exporter.
2. Do the metrics accessible from internet (as we'll run Prometheus in a Digital Ocean dropplet).
3. Install Prometheus in dokku.
4. Install Grafana in dokku.
5. Connect all the pieces.
6. Importing a node_exporter dashboard.

1. Exporting the metrics

node_exporter is a small application that exposes several metrics in a format that Prometheus is able to pull.
The installation is straightforward, and consists in download the package from the releases tab in Github, and execute it (these commands are run in your RPi).

# Download the package
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0/node_exporter-1.0.0.linux-armv7.tar.gz

# Extract the package
sudo tar -xvf node_exporter-1.0.0.linux-armv7.tar.gz -C /usr/local/bin/ --strip-components=1

# Run the node_exporter
node_exporter

Once you run the command, you could visit  http://<raspberry_IP>:9100/, and press in the metrics button, in my case it will link to http://192.168.0.166:9100/metrics.

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 9.0365e-05
go_gc_duration_seconds{quantile="0.25"} 0.000183073
go_gc_duration_seconds{quantile="0.5"} 0.000413749
go_gc_duration_seconds{quantile="0.75"} 0.0005413
go_gc_duration_seconds{quantile="1"} 0.012291414
go_gc_duration_seconds_sum 2.396249156
go_gc_duration_seconds_count 4209
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.3"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.897568e+06

...
...

You don't want to run this command manually every time that your raspberry needs to restart, so you can add an entry to your crontab:

crontab -e

# Add the following line to your crontab and save the file
@reboot /usr/local/bin/node_exporter

2. Doing accessible node_exporter from Internet

As we'll have our Prometheus hosted in an external server (Digital Ocean in my case), we have to do our RPi accessible from the internet.

Doing that could be different depending on the router you have, so I gonna share a screenshot of my router's configuration, where it's only needed to set up the port to be mapped to the RPi, in my case, accessing to <my_ip>:9101 from the internet, will be mapped to the port 9100 on my RPi.

3. Install Prometheus in Dokku

I'm going to explain how to deploy Prometheus in that droplet using Dokku.

# Create the dokku application
dokku apps:create prometheus

# Create the directory where we'll store the prometheus data
mkdir -p /var/lib/dokku/data/storage/prometheus/

# The prometheus container uses the user `nobody`, so we change the permissions to do that directory writable
# Related: https://github.com/prometheus/prometheus/blob/e963d953eac7d5181024eba2215740835b65c0a9/Dockerfile#L18-L21
sudo chown -R 65534:65534 /var/lib/dokku/data/storage/prometheus/

In our local, we create a new repository and setup a remote with our dokku host.

# Initialize the repository
git init

# Add a new remote called `dokku` with our dokku host
git remote add dokku dokku@maroto.me:prometheus

Now, we have to create the minimum docker environment:

• Dockerfile (using the prometheus image as base, and just copying the config).
• prometheus.yml (with the config to scrape).
• app.json (optional, but I'm using the dokku-require plugin, this file configures the mounted volumes).

FROM prom/prometheus

COPY ./prometheus.yml /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s # Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "Raspberry"
    static_configs:
      - targets: ["<your_public_ip_address>:9101"]

{
  "dokku": {
    "volumes": [
      {
        "host": "/var/lib/dokku/data/storage/prometheus/data",
        "app": "/data/",
        "phases": "run,deploy"
      }
    ]
  }
}

After commit all these files, and push to the dokku server, you should be able to access to the prometheus in the default domain value, but maybe you prefer to access in a more friendly url, like https://prometheus.yourdomain.com:9091/.

# Configure your desired domain
dokku domains:add prometheus prometheus.yourdomain.com

# Enable the HTTPS using the letsencrypt plugin
dokku letsencrypt prometheus

# Set the proxy ports
dokku proxy:ports-set prometheus http:9090:9090 https:9091:9090

The last step, is just to configure the start parameter that prometheus will use to run the container (more info about the available parameters in the prometheus website):

dokku config:set prometheus DOKKU_DOCKERFILE_START_CMD="--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/data --storage.tsdb.retention.size=1GB --storage.tsdb.retention.time=30d"

After that, if you open https://prometheus.yourdomain.com:9091/ in your browser, you should see something similar to this under the Status > Targets menu.

And you could start to play with the available metrics from the Graph menu (prometheus query examples).

Recap

In this article we've learned how to consume the metrics that node_exporter generates from our Raspberry Pi, and querying these using prometheus.

In the next article I'll explain how to install Grafana in Dokku, and how to connect all the pieces.

References

• Prometheus :: Storage Configuration.
• Tutorial: Node Exporter in Raspberry Pi.
• Github: raspberry_exporter.