Server Monitoring Guide: Using Prometheus, Grafana And Node_Exporter For Easy Server Monitoring

Server monitoring is really essential if we want to make sure that our applications are running smoothly. It is a basic need for every DevOps engineer, System Administrator and Developers as well. It shows us the health of our server and the insights provided by monitoring tool can prevent us from heading into a critical issue.

Prometheus and grafana are really popular monitoring tools that have a very large community. We will setup all the necessary tools in ubuntu server.

Page Contents

• Part 1: Installing Docker
• Part 2: Installing Docker-Compose
• Part 3: Setting up prometheus and grafana:

Part 1: Installing Docker

Before you install Docker Engine for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.

Use these commands and you can also check screenshots for reference:

sudo apt update

Next, install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt install apt-transport-https ca-certificates curl software-properties-common

Then add the GPG key for the official Docker repository to your system:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the Docker repository to APT sources:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

This will also update our package database with the Docker packages from the newly added repo.

Make sure you are about to install from the Docker repo instead of the default Ubuntu repo:

apt-cache policy docker-ce

Notice that docker-ce is not installed, but the candidate for installation is from the Docker.

Finally, install Docker:

sudo apt install docker-ce

Check the status of docker:

sudo systemctl status docker

Initially you won’t have any images and containers.

To view images:

sudo docker images

To view running containers:

sudo docker ps

Part 2: Installing Docker-Compose

We don’t need to do much while installing docker-compose. Just follow these steps:

sudo apt-get update -y
sudo apt-get install docker-compose -y

Check your docker-compose version after its installed:

sudo docker-compose --version

Part 3: Setting up prometheus and grafana:

After installing docker-compose we can easily setup prometheus and grafana using docker-compose.yml file. While we are doing this let’s try to monitor our host machine (server) as well. We need node_exporter for this.

Lets configure docker-compose.yml and start up our containers

sudo mkdir prometheus

cd prometheus
sudo nano docker-compose.yml

Now insert the following code into it:

version: '3.3'
volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./config:/etc/prometheus/
      - prometheus-data:/prometheus
    networks:
      - prometheus-network
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - prometheus-network
    ports:
      - "3000:3000"
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    ports:
      - "9100:9100"
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
    networks:
      - prometheus-network
networks:
  prometheus-network:
    driver: bridge

Our configuration file looks like this:

In our docker-compose file:

version: Version 3.3 is specified.

volumes: Two volumes prometheus-data and grafana-data is created locally. It is created automatically in directory /var/lib/docker/volumes/

services: Two services prometheus and grafana is created; image and container name is specified for them respectively.

restart: restart is set to unless stopped i.e. if the container was running before the reboot, the container would be restarted once the system restarted.

volumes:

For Prometheus: – ./config:/etc/prometheus/ : A folder named config (inside our current folder of host/server) is mapped with /etc/Prometheus (inside the container prometheus) – prometheus-data:/prometheus : Our previously created volume prometheus-data is mapped with /Prometheus (inside the container prometheus)

For grafana: – grafana-data:/var/lib/grafana : Our previously created volume grafana-data is mapped with /var/lib/grafana (inside the container grafana)

For node_exporter: – ‘/:/host:ro,rslave’ : Root directory of the host i.e. ‘/’ is mapped as ‘/host’ in readonly mode as a slave (inside the container node_exporter).

pid: pid of node_exporter is shared with host. This is done for process monitoring.

command: In case of node_exporter. Command ‘–path.rootfs=/host’ is specified and it is executed while our container starts . It is done for monitoring the root disk of our host.

networks: All containers reside in same network i.e. prometheus-network

ports: Port mapping is done from host port to container port.

networks: A network named prometheus-network is defined whose driver mode is bridge.

Also, we need a config file for Prometheus. So let’s create it.

mkdir config

sudo nano config/prometheus.yml

Copy the following code inside it:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  
  # scrape_timeout is set to the global default (10s).
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

 Now lets start our containers:

docker-compose up --build –d

and check the status:

docker-compose ps

Looks like everything is running fine. Now go to http://yourserverip:9090/ to access Prometheus:

Go to Status->targets.

Here we can see that both our targets prometheus and node_exporter is up.

It means that prometheus can scrape data from these endpoints.

However, we won’t be discussing about prometheus for now. We will be using grafana to display the metrics that is scraped by prometheus.

Let’s login to our grafana server.

Goto http://yourserverip:3000/

Default username and password is admin.

After logging in click on Add your first data source:

Select Prometheus:

Set your data source name and enter url for making connection to prometheus.

In our case set this url:

http://prometheus:9090

This above url works only because grafana and prometheus are in same network in docker.

Click on Save and test.

Now let’s setup our dashboard for host monitoring.

Go to import:

Now paste 1860 and click on Load

This ID helps us to export full node_exporter dashboard from grafana dashboard library. You can browser more dashboards on https://grafana.com/grafana/dashboards/.

Click on load.

Set name, set the correct preometheus and click on import:

After importing you can see a dashboard. This dashboard is readymade dashboard and contains a lot of information.

Your dashboard has been loaded along with various metrics such as CPU utilization, memory utilization, network traffic utilization, etc. There are a lot of various metrics that you can explore and customize as you need.

Thats it!! Its really easy to monitor your server using this method.