What is Prometheus?

Prometheus is a popular Opensource system monitoring and alerting tool Originally developed by SoundCloud, then made open source and accepted as the second project in the Cloud Native Computing Foundation (CNCF) in 2016. It is similar in design to Google’s Borgmon monitoring system, and a relatively modest system can handle collecting hundreds of thousands of metrics every second.

Prometheus Architecture diagram :

Prometheus is a multi-component system some of which are optional

• Prometheus Server (Scrapes and stores the time series data)
• Client libraries (Instrumenting application code)
• Push gateway (To support metric collections from short-lived jobs)
• Special purpose exporters (for services like HAProxy, StatsD, Graphite, etc.)
• Alert manager (to handle alerts)
• Additional support tools

Prometheus architecture diagram

How does it work?

Prometheus is meant to be used to collect metrics from an exposed HTTP endpoint. It stores all the collected metrics locally and runs rules over the data to aggregate and record new time series from existing data or generate alerts. API consumers like Grafana can be used to visualize

• In addition to stored time series from the source are generated by queries. These series are identified by metric labels.
• Queries produced by PromQL (Prometheus Query Language) enables users to select and aggregate time-series data in real-time.
• PromQL (Prometheus Query Language) also allows you to set up triggers that will send push notifications to other sources, like Slack, PagerDuty, or Email.

Minimum Requirements to Install Prometheus :

• 2 CPU cores
• 4 GB of memory
• 20 GB of free disk space

Installation Process :

• This guide explains how to install and configure the latest Prometheus on a Linux VM.

Step 1:

• Create a Prometheus user, required directories, and make prometheus user as the owner of those directories.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Step 2:

• Download the source using Curl, untar it and rename the extracted folder to prometheus-files

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz
tar -xvf prometheus-2.28.1.linux-amd64.tar.gz
mv prometheus-2.18.1.linux-amd64 prometheus-files

Step 3:

• Copy prometheus and promtool binaries from prometheus-files folder to /usr/local/bin and change the ownership to prometheus user.

sudo cp prometheus-files/prometheus /usr/local/bin/
sudo cp prometheus-files/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

• Move the consoles and console_libraries directories from prometheus-files to /etc/prometheus folder and change the ownership to prometheus user.

sudo cp -r prometheus-files/consoles /etc/prometheussudo cp -r prometheus-files/console_libraries /etc/prometheussudo chown -R prometheus:prometheus /etc/prometheus/consolessudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

In the /etc/prometheus directory, need to create a configuration file named prometheus.yml.

sudo cat << EOF > /etc/prometheus/ prometheus.yml
global:
 external_labels:
  region: us-central
  monitor: infra
alerting:
 alertmanagers:
 - static_configs:
    - targets:
     - “localhost:9093”
rule_files:
 - “rules.yml”
scrape_configs:
 - job_name: prometheus
   scrape_interval: 5s
   static_configs:
    - targets:
       - “localhost:9090”
 - job_name: nsq_exporter
   scrape_interval: 5s
   static_configs:
   - targets:
     - “<Target_Server_IP>:9127”
EOF

Step 4: Setup Prometheus Systemd Service

• Create a prometheus service file.

sudo cat << EOF > /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /monitoring/prometheus/prometheus_data/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.external-url=https://prom.example.com \
  --web.listen-address=:9090 \
  --web.enable-lifecycle \
  --web.enable-admin-api \
  --log.level=info
[Install]
WantedBy=multi-user.target
EOF

• — config.file the directory containing the Prometheus configuration file
  — storage.tsdb.path Where Prometheus writes its database
  — web.console.templates Prometheus Console templates path
  — web.console.libraries Prometheus Console libraries path
  — web.external-url Prometheus External URL
  — web.listen-addres Prometheus running port
  — web.enable-lifecycle If you use the Prometheus /-/reload HTTP endpoint to automatically reload your Prometheus config when it changes, these endpoints are disabled by default for security reasons in Prometheus 2.0. To enable them, set the — web.enable-lifecycle flag
  — web.enable-admin-api controls access to the administrative HTTP API
• Reload the system service to register the prometheus service and start the prometheus service

sudo systemctl daemon-reload
sudo systemctl start prometheus

• Check the prometheus service status using the following command.

sudo systemctl status prometheus

Alert Manager Integration

AlertManager is used to handle alerts for client applications(like Prometheus). It also takes care of alerts deDuplicating, grouping and then routes them to different receivers such as E-mail, Slack, Pager Duty.

Installation

• The installation part of AlertManager is not a fancy thing, we just need simply to download the latest binary of Alertmanager from here.

Configuration

• Need to configure after the tar file is extracted and the alertmanager binary file is placed at the right location. Although Alertmanager extracted directory contains the configuration file as well, it is not of our use. So we will create our own configuration. Let’s start by creating a directory for configuration.

mkdir /etc/alertmanager/

• Then the configuration file will take place.

sudo cat << EOF > /etc/alertmanager/alertmanager.yml
route:
group_by: [‘alertname’, ‘cluster’, ‘service’]
group_wait: 30s
group_interval: 2m
repeat_interval: 3m
receiver: slack_general
routes:
- match:
severity: slack
receiver: slack_general
receivers:
- name: slack_general
  slack_configs:
  - api_url:‘https://hooks.slack.com/services/T596DT44S/xxxxxxxxxxxxxxxxxxxxx'
channel: ‘#monitoring-alerts’
send_resolved: true
EOF

• Once the configuration part is done we just have to create a storage directory where Alertmanager will store its data.

mkdir /var/lib/alertmanager

Creating Alertmanager Systemd service

• Create an alermanager.service file to start with systemd

sudo cat << EOF > /etc/systemd/system/alertmanager.service
[Unit]
Description=AlertManager Server Service
Wants=network-online.target
After=network-online.target[Service]
User=root
Group=root
Type=Simple
ExecStart=/usr/local/bin/alertmanager \
--config.file /etc/alertmanager/alertmanager.yml
[Install]
WantedBy=multi-user.target
EOF

• Then reload the daemon and start the alertmanager service.

systemctl daemon-reload
systemctl start alertmanager
systemctl enable alertmanager

Prometheus Node Exporter Setup

• Node exporter is the best way to collect all the Linux server related metrics and statistics for monitoring

Prerequisites

• Prometheus Node Exporter needs the Prometheus server to be up and running.
• Port 9100 opened in server firewall as Prometheus reads metrics on this port.
• Setup Node Exporter Binary

Download the latest node exporter package and unpack it

cd /tmpcurl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Move the node export binary to /usr/local/bin

sudo mv node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin

• Create a Node Exporter systemd Service

sudo useradd -rs /bin/false node_exporter
sudo cat << EOF > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF

• Reload the system daemon and start the node exporter service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Now, the node exporter would be exporting metrics on port 9100.

You can see all the server metrics by visiting your server URL on /metrics as shown below.

http://localhost:9100/metrics

Configure the Server as Target on Prometheus Server

• Now that we have the node exporter up and running on the server, we have to add this server as a target on the Prometheus server configuration.

Note: This configuration should be done on the Prometheus server.

Step 1:

• Login to the Prometheus server and open the prometheus.yml file.

sudo vi /etc/prometheus/prometheus.yml

Step 2:

• Under the scrape config section add the node exporter target as shown below. Change 1.0.0.0.2 with your server IP where you have setup node exporter.
• Job name can be your server hostname or IP for identification purposes.

- job_name: ‘node_exporter_metrics’
   scrape_interval: 5s
   static_configs:
    - targets: [‘TARGET_SERVER_IP:9100’]

• Restart the prometheus service for the configuration changes to take place.

sudo systemctl restart prometheus