Monitoring with Grafana & Prometheus
In this tutorial, we are going to create a grafana dashboard for our stake pool that have 3 nodes relays and 1 core node (block producer).
One of the relay nodes, the relay3, will be used as grafana host, it will be the server where installing and deploying grafana monitoring web. So for this server, the configuration steps will be a bit different, and the rest relays and core nodes will follow the same steps.
We will monitor both the cardano-node generated metrics and the general server metrics of the Prometheus node-exporter service.
Relay3 host
Relay1 & relay2 & core

Setup environment

Install Grafana

Add key to apt :
1
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
Copied!
Add repository to apt:
1
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
Copied!
Update the packages (now we have grafana repo and verifiable with the key):
1
sudo apt update
Copied!
Install grafana:
1
sudo apt install grafana
Copied!
Change port where grafana web will be deployed:
1
sudo vim /etc/grafana/grafana.ini
Copied!
Choose the port of your choice(you can use the default port):
http_port = <port>
Start grafana service and enable to start on reboot:
1
sudo systemctl start grafana-server
2
sudo systemctl enable grafana-server
Copied!
Check grafana status:
1
sudo systemctl status grafana-server
Copied!
Others options:
1
sudo systemctl restart grafana-server
2
sudo systemctl stop grafana-server
Copied!
Open the port used in grafana.ini with ufw:
1
sudo ufw allow proto tcp from any to any port <port>
Copied!
Now, everyone can access to the login grafana page throw a browser:http://<ip-address>:<port>
1est time, use 'admin' as user & password, then, change the password.
You can give a domain to the ip-address to give it a more friendly access.
Finally, install the Clock plugin:
1
grafana-cli plugins install grafana-clock-panel
Copied!

Install Prometheus & node exporter

1
sudo apt-get install -y prometheus prometheus-node-exporter prometheus-alertmanager
Copied!
1
sudo find / -name prometheus-node-exporter.service
2
vim path-to/prometheus-node-exporter.service
Copied!
Choose the port fo your choice, change the line from:
1
ExecStart=/usr/local/bin/node_exporter
Copied!
To:
1
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<port>
Copied!
Start the services and enable them to init when reboot:
1
sudo systemctl start prometheus
2
sudo systemctl start prometheus-node-exporter
3
sudo systemctl enable prometheus
4
sudo systemctl enable prometheus-node-exporter
Copied!
Once all the services are installed and working, lets edit the prometheus conf file to add the relays and core metrics data source:
1
sudo find / -name prometheus.yml
2
sudo vim <path-to>/prometheus.yml
Copied!

Add metrics data sources to Prometheus: relays & core

Before continue, be sure to complete all relays and core steps.

Delete all content and add the new conf:
1
# global configs
2
global:
3
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
4
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
5
# scrape_timeout is set to the global default (10s).
6
7
scrape_configs:
8
- job_name: 'cardano' # To scrape data from the running cardano-node
9
scrape_interval: 15s
10
static_configs:
11
- targets: ['<ip-address1>:12798']
12
labels:
13
alias: 'relay1'
14
type: 'cardano-node'
15
- targets: ['<ip-address2>:12798']
16
labels:
17
alias: 'relay2'
18
type: 'cardano-node'
19
- targets: ['<ip-address3>:12798']
20
labels:
21
alias: 'core'
22
type: 'cardano-node'
23
- targets: ['127.0.0.1:12798']
24
labels:
25
alias: 'relay3-host'
26
type: 'cardano-node'
27
28
- job_name: 'node' # To scrape data from a node exporter to monitor the linux host metrics.
29
scrape_interval: 15s
30
static_configs:
31
- targets: ['<ip-address1>:<node-exporter-port>']
32
labels:
33
alias: 'relay1'
34
type: 'host-system'
35
- targets: ['<ip-address2>:<node-exporter-port>']
36
labels:
37
alias: 'relay2'
38
type: 'host-system'
39
- targets: ['<ip-address3>:<node-exporter-port>']
40
labels:
41
alias: 'core'
42
type: 'host-system'
43
- targets: ['127.0.0.1:<node-exporter-port>']
44
labels:
45
alias: 'relay3-host'
46
type: 'host-system'
Copied!
Note that, the relay3 is the host of grafana monitoring tool, so we just need to specify the localhost ip 127.0.0.1 for this source, because they are in the same machine.
Be sure to replace <ip-address> and <node-exporter-port> with your own values.
Restart the service to load the new conf:
1
sudo systemctl restart prometheus
Copied!

Setup environment

Install Prometheus & node exporter

1
sudo apt-get install -y prometheus-node-exporter
Copied!
1
sudo find / -name prometheus-node-exporter.service
2
vim path-to/prometheus-node-exporter.service
Copied!
Choose the port fo your choice, change the line from:
1
ExecStart=/usr/local/bin/node_exporter
Copied!
To:
1
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<node-exporter-port>
Copied!
Start the services and enable them to init when reboot:
1
sudo systemctl start prometheus-node-exporter.service
2
sudo systemctl enable prometheus-node-exporter
Copied!

Cardano config: Make cardano-node metrics accessible all over the world:

Lets find the cardano-node configuration file, then edit prometheus binding ip address:
1
sudo find / -name mainnet-config.json
2
vim path-to/mainnet-conf.json
Copied!
Change field hasPrometheus from:
"hasPrometheus": [ "127.0.0.1", 12798 ]
To:
"hasPrometheus": [ "0.0.0.0", 12798 ]
After editing configuration, you must restart the cardano-node service to get the new config values.

Firewall: restrict cardano-node port 12798 just to grafana host server

Open port to the specific ip address, in this case, just the host relay can request metrics:
1
sudo ufw allow from <host-ip-address3> to any port 12798
2
sudo ufw allow from <host-ip-address3> to any port <node-exporter-port>
Copied!
Congrats! we have the cardano-node generating accesible metrics from port 12798 and the server metrics exposed in <node-exporter-port> .

Grafana monitor web

Add Prometheus source:

In grafana left menu, go to Configuration > Data Sources > Add data source and select Prometheus.

Create a new stake pool dashboard from scratch:

In grafana left menu, go to + > Create > Dashboard.
Create a graph about connected peers:

1. Select Prometheus data source

2. Select a specific metric

Create a chart about connected peers:
Example output:

3. Set chart properties

Load a dashboard from file.

In grafana panel, Go to + > Create > Import > enter json content.
Example panel version 1:
Support our work delegating to Ada Booster.