Setup Prometheus and Grafana on Ubuntu

1. Introduction

In this guide, we will demonstrate how to set up Prometheus and Grafana on Ubuntu 22.04.

Written in the Go language, Prometheus is a free and open-source monitoring and alerting solution that collects metrics data from services and stores them in a time-series database. It scrapes metric data from HTTP endpoints, either directly or via an intermediary push gateway for systems not compatible with push-gateway scraping.

It’s a reliable tool for collecting numeric time-series data and provides support for multi-dimensional data collection and querying. Once data has been collected, it can be visualized using Grafana or other API tools.

Key features provided by Prometheus include:

Prometheus Server: Scrapes and stores time-series data.

PromQL (Prometheus Query Language): This is a powerful and flexible query language that aggregates time series data.

Alert Manager: Triggers alerts when they meet certain rules.

Client libraries: Used for instrumenting application code.

Push Gateway: Gathers services and metrics from external systems that are not compatible with pull-based scraping.

Special Purpose exporters: This allows Prometheus to scrape metrics from third-party systems. E.g node_exporter.

Grafana is an open-source data analysis and visualization platform that visualizes data in interactive dashboards in the form of charts, and graphs from supported data sources. This allows for easier interpretation and understanding of data. It enables monitoring of application and service performance and pinpoints errors or any issues that impact performance.

2. Install Prometheus 

To get started, we will first install and configure Prometheus, then focus on installing Grafana and creating beautiful dashboards for visualizing data collected by Prometheus.

2.1. Setup Prometheus System

First, we are going to create dedicated non-login system account users for Prometheus and Node exporters. Having individual users for each service acts as a security measure to reduce the attack surface in case of an incident. It also allows you to track what resources are associated with which service.

To create Prometheus and node_exporter users, run the following commands

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter

Let’s break down the command options.

• --system: Creates system accounts.
• --no-create-home: Omits the creation of a home directory for Prometheus and node_exporter system accounts. 
• --shell /bin/false: Denies login access to the system account users.

Next, create Prometheus directories for storing Prometheus settings and libraries as shown.

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

Now we will have to update the group and user ownership on the newly created directories to Prometheus.

To do so, run the following commands:

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

2.2. Download Prometheus Binary File

You can find the latest version of Prometheus from the official Prometheus website.

Here we are downloading Prometheus 2.44 version, which is the latest version at the time of writing this guide.

Navigate to /opt directory.

cd /opt/

Download the Prometheus setup using wget command:

wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz

This will download the Prometheus archive file to the current directory.

2.3. Install Prometheus

First, we will use sha256sum command to generate a checksum of the downloaded Prometheus archive file. It will ensure that our downloaded file is not a corrupted file.

sha256sum prometheus-2.44.0.linux-amd64.tar.gz

The output should match with the sha256sum checksum on the official Prometheus download page.

Now we can extract the archive file

tar -xvf prometheus-2.44.0.linux-amd64.tar.gz

Navigate to Prometheus extracted folder

cd prometheus-2.44.0.linux-amd64

To check the list of setup files run the command.

ls -l

There are two library files in the directory named prometheus and promtool. We will have to copy both libraries to the  /usr/local/bin directory.

Copying library files to /usr/local/bin/ directory:

sudo cp /opt/prometheus-2.44.0.linux-amd64/prometheus /usr/local/bin/
sudo cp /opt/prometheus-2.44.0.linux-amd64/promtool /usr/local/bin/

Change the ownership and group to "prometheus" for the library files:

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Next, copy the console and console_libraries directories to /etc/prometheus directory:

sudo cp -r /opt/prometheus-2.44.0.linux-amd64/consoles /etc/prometheus

sudo cp -r /opt/prometheus-2.44.0.linux-amd64 /console_libraries /etc/prometheus

sudo cp -r /opt/prometheus-2.44.0.linux-amd64 /prometheus.yml /etc/prometheus

We will update the user and group ownership on the console and console library directories and Prometheus YAML file.

Recursively change ownership and group to "prometheus" for consoles and console_libraries directory. Also, change the ownership for prometheus.yml file.

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

Done !! Prometheus is successfully installed. We can verify it by checking the version.

To verify the Prometheus version from the command line, type:

prometheus --version

promtool --version

2.4. Create Prometheus Systemd Service File

You can find the configuration file under /etc/prometheus - which we have already copied /opt/prometheus-2.26.0.linux-amd64/prometheus.yml file /etc/prometheus directory. Modify it as per your requirement.

cat /etc/prometheus/prometheus.yml

Now we will create Prometheus systemd service file in /etc/systemd/system directory named prometheus.service.

sudo nano /etc/systemd/system/prometheus.service

Add the following lines and save the file.

[Unit]
Description=Prometheus
Wants=network-online. Target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

To use the newly created service we will have to reload the daemon services

sudo systemctl daemon-reload

Start and enable Prometheus service:

sudo systemctl start prometheus
sudo systemctl enable prometheus

Make sure the prometheus service is active (running) by running the following command.

sudo systemctl status prometheus

From the output, we can confirm that Prometheus service is up.

2.5. Accessing Prometheus

Prometheus should now be installed and running on your Ubuntu system. You access the Prometheus web interface on any web browser, using:

http://server-IP-or-Hostname:9090

If you are running firewall, make sure to enable 9090 port. For example to enable Prometheus port in the ufw firewall, type:

sudo ufw allow 9090/tcp
sudo ufw reload

3. Install Grafana

Add the Grafana GPG key in Ubuntu using wget:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Now add the Grafana repository in Ubuntu using APT

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update system packages

sudo apt update

Now Install Grafana using the apt command.

sudo apt install grafana

Once installed make sure Grafana service status running:

sudo systemctl status grafana-server.service

Enable the Grafana service which will automatically start the Grafana on boot.

sudo systemctl enable grafana-server.service

To access Grafana Dashboard open your favorite browser, type server IP or hostname followed by Grafana default port 3000.

http://your_ip:3000

You will see the Grafana login page. Use the following default login credentials to access the dashboard.

Username – admin
Password – admin

Once you press ‘Login’ you will be required to set a new password.

4. Install Node Exporter

Node Exporter is a Prometheus exporter that collects system-level metrics such as CPU, memory, disk, and network usage. First, you need to install Node Exporter on the target machine.

To install Node Exporter, first, navigate to Prometheus official download page, scroll down and you will get node exporter section and then select Linux OS for amd64.

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz

Once download, extract node exporter from the archive

tar -xvf node_exporter-1.6.0.linux-amd64.tar.gz

Navigate to the node_exporter folder

cd  node_exporter*/

Next, move the node_exporter binary file to the /usr/local/bin directory.

sudo mv node_exporter  /usr/local/bin

Once the binary file is moved, you can delete extracted directory to keep the system clean.

Installation done. You can check the node exporter version using the following command:

node_exporter --version

4.1. Create Node Exporter Systemd File

Now we can go ahead creating a node exporter service file in /lib/systemd/system directory and named it node_exporter.service.

sudo vi /etc/systemd/system/node_exporter.service

Paste the following lines.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

Save the changes and exit the nano editor. Next, reload systemd service:

sudo systemctl daemon-reload

Now let's enable and start the node_exporter service using the following commands:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Be sure to verify that node exporter service is up and running.

sudo systemctl status node_exporter

4.2. Configure Node Exporter as a Prometheus target

In the Prometheus configuration file, add a new job under the scrape_configs section.

Open the Prometheus configuration file using any editor:

sudo nano /etc/prometheus/prometheus.yml

Navigate to static_configs section - Replace <node_exporter_ip> with the IP address or hostname of the machine running Node Exporter. If Node Exporter is running on the same machine as Prometheus, you can use localhost or 127.0.0.1 as the IP address.

- targets: [‘localhost:9090’, ‘localhost:9100’]

To apply the changes, restart the Prometheus service.

sudo systemctl restart prometheus

5. Creating Grafana Dashboard

Now let's build a dashboard in Grafana - we will visualize the metrics of the Linux system. Let's log in to Grafana.

Before you can create a dashboard, you need to add a data source. So, click ‘Add your first data source’

Next, click ‘Prometheus’ since this is the service that is collecting and storing time-series metric data from the server.

Specify the name and URL of your server.

Then scroll all the way down and click ‘Save and Test’. 

If all goes well you should get a message saying "Data source is working".

Grafana provides an array of custom dashboards that you can import and use to visualize metrics from your systems.  These include heat maps, histograms, geo maps, line graphs, etc. You can check them out at the Grafana dashboards page.

So, we will import a dashboard. To do so, click the ‘Add’ button then ‘Import from library’.

Here we will use the 14513 dashboard ID and click ‘Load’

Next, select the name of the Prometheus data source and click ‘Import’.

Shortly after, the system metrics will start being displayed on the dashboards.