Monitoring with Grafana & Prometheus

In this tutorial, we are going to create a grafana dashboard for our stake pool that have 3 nodes relays and 1 core node (block producer).
One of the relay nodes, the relay3, will be used as grafana host, it will be the server where installing and deploying grafana monitoring web. So for this server, the configuration steps will be a bit different, and the  rest relays and core nodes will follow the same steps.
We will monitor both the cardano-node generated metrics and the general server metrics of the Prometheus node-exporter service.
Relay3 host
Relay1 & relay2 & core
Setup environment
Install Grafana
Add key  to apt :
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
Add repository to apt:
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
Update the packages (now we have grafana repo and verifiable with the key):
sudo apt update
Install grafana:
sudo apt install grafana
Change port where grafana web will be deployed:
sudo vim /etc/grafana/grafana.ini
Choose the port of your choice(you can use the default port):
http_port = <port>
Start grafana service and enable to start on reboot:
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
Check grafana status:
sudo systemctl status grafana-server
Others options:
sudo systemctl restart grafana-server
sudo systemctl stop grafana-server
Open the port used in grafana.ini with ufw:
sudo ufw allow proto tcp from any to any port <port>
Now, everyone can access to the login grafana page throw a browser:http://<ip-address>:<port>
1est time, use 'admin' as user & password, then, change the password.
You can give a domain to the ip-address to give it a more friendly access.
Finally, install the Clock plugin:
grafana-cli plugins install grafana-clock-panel
Install Prometheus & node exporter
sudo apt-get install -y prometheus prometheus-node-exporter prometheus-alertmanager
sudo find / -name  prometheus-node-exporter.service
vim path-to/prometheus-node-exporter.service
Choose the port fo your choice, change the line from:
ExecStart=/usr/local/bin/node_exporter
To:
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<port>
Start the services and enable them to init when reboot:
sudo systemctl start prometheus
sudo systemctl start prometheus-node-exporter
sudo systemctl enable prometheus
sudo systemctl enable prometheus-node-exporter
Once all the services are installed and working, lets edit the prometheus conf file to add the relays and core metrics data source:
sudo find / -name  prometheus.yml
sudo vim <path-to>/prometheus.yml
Add metrics data sources to Prometheus: relays & core
Before continue, be sure to complete all relays and core steps.
Delete all content and add the new conf:
# global configs
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
​
scrape_configs:
  - job_name: 'cardano' # To scrape data from the running cardano-node
    scrape_interval: 15s
    static_configs:
      - targets: ['<ip-address1>:12798']
        labels:
          alias: 'relay1'
          type:  'cardano-node'
      - targets: ['<ip-address2>:12798']
        labels:
          alias: 'relay2'
          type:  'cardano-node'
      - targets: ['<ip-address3>:12798']
        labels:
          alias: 'core'
          type:  'cardano-node'
      - targets: ['127.0.0.1:12798']
        labels:
          alias: 'relay3-host'
          type:  'cardano-node'
​
  - job_name: 'node' # To scrape data from a node exporter to monitor the linux host metrics.
    scrape_interval: 15s
    static_configs:
      - targets: ['<ip-address1>:<node-exporter-port>']
        labels:
          alias: 'relay1'
          type:  'host-system'
      - targets: ['<ip-address2>:<node-exporter-port>']
        labels:
          alias: 'relay2'
          type:  'host-system'
      - targets: ['<ip-address3>:<node-exporter-port>']
        labels:
          alias: 'core'
          type:  'host-system'
      - targets: ['127.0.0.1:<node-exporter-port>']
        labels:
          alias: 'relay3-host'
          type:  'host-system'
Note that, the relay3 is the host of grafana monitoring tool, so we just need to specify the localhost ip 127.0.0.1 for this source, because they are in the same machine.
Be sure to replace <ip-address> and <node-exporter-port> with your own values.
Restart the service to load the new conf:
sudo systemctl restart prometheus
Setup environment
Install Prometheus & node exporter
sudo apt-get install -y prometheus-node-exporter
sudo find / -name  prometheus-node-exporter.service
vim path-to/prometheus-node-exporter.service
Choose the port fo your choice, change the line from:
ExecStart=/usr/local/bin/node_exporter
To:
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<node-exporter-port>
Start the services and enable them to init when reboot:
sudo systemctl start prometheus-node-exporter.service
sudo systemctl enable prometheus-node-exporter
Cardano config: Make cardano-node metrics accessible all over the world:
Lets find the cardano-node configuration file, then edit prometheus binding ip address:
sudo find / -name  mainnet-config.json
vim path-to/mainnet-conf.json
Change field hasPrometheus from:
"hasPrometheus": [ "127.0.0.1", 12798 ]
To:
"hasPrometheus": [ "0.0.0.0", 12798 ]
After editing configuration, you must restart the cardano-node service to get the new config values.
Firewall: restrict cardano-node port 12798 just to grafana host server
Open port to the specific ip address, in this case, just the host relay can request metrics:
sudo ufw allow from <host-ip-address3> to any port 12798
sudo ufw allow from <host-ip-address3> to any port <node-exporter-port>
Congrats! we have the cardano-node generating accesible metrics from port 12798 and the server metrics exposed in <node-exporter-port> .
Grafana monitor web
Add Prometheus source:
In grafana left menu, go to Configuration > Data Sources > Add data source and select Prometheus.
Create a new stake pool dashboard from scratch:
In grafana left menu, go to + > Create > Dashboard.
Create a graph about connected peers:
1. Select Prometheus data source
2. Select a specific metric
Create a chart about connected peers:
Example output:
3. Set chart properties
​
Load a dashboard from file.
In grafana panel, Go to + > Create > Import > enter json content.
Example panel version 1:
​
Support our work delegating to Ada Booster.
​
Última actualización 2yr ago