What is Prometheus?
Prometheus is a popular Opensource system monitoring and alerting tool Originally developed by SoundCloud, then made open source and accepted as the second project in the Cloud Native Computing Foundation (CNCF) in 2016. It is similar in design to Google’s Borgmon monitoring system, and a relatively modest system can handle collecting hundreds of thousands of metrics every second.
Prometheus Architecture diagram :
Prometheus is a multi-component system some of which are optional
- Prometheus Server (Scrapes and stores the time series data)
- Client libraries (Instrumenting application code)
- Push gateway (To support metric collections from short-lived jobs)
- Special purpose exporters (for services like HAProxy, StatsD, Graphite, etc.)
- Alert manager (to handle alerts)
- Additional support tools
Prometheus architecture diagram
How does it work?
Prometheus is meant to be used to collect metrics from an exposed HTTP endpoint. It stores all the collected metrics locally and runs rules over the data to aggregate and record new time series from existing data or generate alerts. API consumers like Grafana can be used to visualize
- In addition to stored time series from the source are generated by queries. These series are identified by metric labels.
- Queries produced by PromQL (Prometheus Query Language) enables users to select and aggregate time-series data in real-time.
- PromQL (Prometheus Query Language) also allows you to set up triggers that will send push notifications to other sources, like Slack, PagerDuty, or Email.
Minimum Requirements to Install Prometheus :
- 2 CPU cores
- 4 GB of memory
- 20 GB of free disk space
Installation Process :
- This guide explains how to install and configure the latest Prometheus on a Linux VM.
- Create a Prometheus user, required directories, and make prometheus user as the owner of those directories.
sudo useradd --no-create-home --shell /bin/false prometheus sudo mkdir /etc/prometheus sudo mkdir /var/lib/prometheus sudo chown prometheus:prometheus /etc/prometheus sudo chown prometheus:prometheus /var/lib/prometheus
- Download the source using Curl, untar it and rename the extracted folder to prometheus-files
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz tar -xvf prometheus-2.28.1.linux-amd64.tar.gz mv prometheus-2.18.1.linux-amd64 prometheus-files
- Copy prometheus and promtool binaries from prometheus-files folder to /usr/local/bin and change the ownership to prometheus user.
sudo cp prometheus-files/prometheus /usr/local/bin/ sudo cp prometheus-files/promtool /usr/local/bin/ sudo chown prometheus:prometheus /usr/local/bin/prometheus sudo chown prometheus:prometheus /usr/local/bin/promtool
- Move the consoles and console_libraries directories from prometheus-files to /etc/prometheus folder and change the ownership to prometheus user.
sudo cp -r prometheus-files/consoles /etc/prometheussudo cp -r prometheus-files/console_libraries /etc/prometheussudo chown -R prometheus:prometheus /etc/prometheus/consolessudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
In the /etc/prometheus directory, need to create a configuration file named prometheus.yml.
sudo cat << EOF > /etc/prometheus/ prometheus.yml global: external_labels: region: us-central monitor: infra alerting: alertmanagers: - static_configs: - targets: - “localhost:9093” rule_files: - “rules.yml” scrape_configs: - job_name: prometheus scrape_interval: 5s static_configs: - targets: - “localhost:9090” - job_name: nsq_exporter scrape_interval: 5s static_configs: - targets: - “<Target_Server_IP>:9127” EOF
Step 4: Setup Prometheus Systemd Service
- Create a prometheus service file.
sudo cat << EOF > /etc/systemd/system/prometheus.service [Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /monitoring/prometheus/prometheus_data/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --web.external-url=https://prom.example.com \ --web.listen-address=:9090 \ --web.enable-lifecycle \ --web.enable-admin-api \ --log.level=info [Install] WantedBy=multi-user.target EOF
- — config.file the directory containing the Prometheus configuration file
— storage.tsdb.path Where Prometheus writes its database
— web.console.templates Prometheus Console templates path
— web.console.libraries Prometheus Console libraries path
— web.external-url Prometheus External URL
— web.listen-addres Prometheus running port
— web.enable-lifecycle If you use the Prometheus /-/reload HTTP endpoint to automatically reload your Prometheus config when it changes, these endpoints are disabled by default for security reasons in Prometheus 2.0. To enable them, set the — web.enable-lifecycle flag
— web.enable-admin-api controls access to the administrative HTTP API
- Reload the system service to register the prometheus service and start the prometheus service
sudo systemctl daemon-reload sudo systemctl start prometheus
- Check the prometheus service status using the following command.
sudo systemctl status prometheus
Alert Manager Integration
AlertManager is used to handle alerts for client applications(like Prometheus). It also takes care of alerts deDuplicating, grouping and then routes them to different receivers such as E-mail, Slack, Pager Duty.
- The installation part of AlertManager is not a fancy thing, we just need simply to download the latest binary of Alertmanager from here.
- Need to configure after the tar file is extracted and the alertmanager binary file is placed at the right location. Although Alertmanager extracted directory contains the configuration file as well, it is not of our use. So we will create our own configuration. Let’s start by creating a directory for configuration.
- Then the configuration file will take place.
sudo cat << EOF > /etc/alertmanager/alertmanager.yml route: group_by: [‘alertname’, ‘cluster’, ‘service’] group_wait: 30s group_interval: 2m repeat_interval: 3m receiver: slack_general routes: - match: severity: slack receiver: slack_general receivers: - name: slack_general slack_configs: - api_url:‘https://hooks.slack.com/services/T596DT44S/xxxxxxxxxxxxxxxxxxxxx' channel: ‘#monitoring-alerts’ send_resolved: true EOF
- Once the configuration part is done we just have to create a storage directory where Alertmanager will store its data.
Creating Alertmanager Systemd service
- Create an alermanager.service file to start with systemd
sudo cat << EOF > /etc/systemd/system/alertmanager.service [Unit] Description=AlertManager Server Service Wants=network-online.target After=network-online.target[Service] User=root Group=root Type=Simple ExecStart=/usr/local/bin/alertmanager \ --config.file /etc/alertmanager/alertmanager.yml [Install] WantedBy=multi-user.target EOF
- Then reload the daemon and start the alertmanager service.
systemctl daemon-reload systemctl start alertmanager systemctl enable alertmanager
Prometheus Node Exporter Setup
- Node exporter is the best way to collect all the Linux server related metrics and statistics for monitoring
- Prometheus Node Exporter needs the Prometheus server to be up and running.
- Port 9100 opened in server firewall as Prometheus reads metrics on this port.
- Setup Node Exporter Binary
Download the latest node exporter package and unpack it
cd /tmpcurl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
Move the node export binary to /usr/local/bin
sudo mv node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin
- Create a Node Exporter systemd Service
sudo useradd -rs /bin/false node_exporter sudo cat << EOF > /etc/systemd/system/node_exporter.service [Unit] Description=Node Exporter After=network.target [Service] User=node_exporter Group=node_exporter Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target EOF
- Reload the system daemon and start the node exporter service.
sudo systemctl daemon-reload sudo systemctl start node_exporter sudo systemctl enable node_exporter
Now, the node exporter would be exporting metrics on port 9100.
You can see all the server metrics by visiting your server URL on /metrics as shown below.
Configure the Server as Target on Prometheus Server
- Now that we have the node exporter up and running on the server, we have to add this server as a target on the Prometheus server configuration.
Note: This configuration should be done on the Prometheus server.
- Login to the Prometheus server and open the prometheus.yml file.
sudo vi /etc/prometheus/prometheus.yml
- Under the scrape config section add the node exporter target as shown below. Change 184.108.40.206.2 with your server IP where you have setup node exporter.
- Job name can be your server hostname or IP for identification purposes.
- job_name: ‘node_exporter_metrics’ scrape_interval: 5s static_configs: - targets: [‘TARGET_SERVER_IP:9100’]
- Restart the prometheus service for the configuration changes to take place.
sudo systemctl restart prometheus