Let's continue from where we left off. In this blog, we are going to monitor Docker by creating a dashboard.
Task:
1) Install Docker and start docker service on a Linux EC2 through USER DATA .
2) Create 2 Docker containers and run any basic application on those containers (A simple todo app will work).
3) Now intregrate the docker containers and share the real time logs with Grafana (Your Instance should be connected to Grafana and Docker plugin should be enabled on grafana).
4) Check the logs or docker container names on Grafana UI.
- For installing docker you can use the below commands. As in our last blog, we have a ready instance with docker and Grafana installed, we are going to use that.
How do you install it through USER DATA has been explained by me in this blog:
#!/bin/bash
# Update the system
apt-get update
apt-get upgrade -y
# Install Docker dependencies
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Update package information and install Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
# Start Docker service
systemctl start docker
# Enable Docker to start on system boot
systemctl enable docker
- Now, we will create 2 docker containers
docker run -d -p 8001:8001 pmgoriya/pankaj:latest
docker run -d nginx
# also you can restart the loki container
docker ps -a
docker restart <container_name_or_id>
Integrating Docker containers with Grafana involves collecting and visualizing logs and metrics generated by Docker containers. In this scenario, we'll use Telegraf as the data collector, InfluxDB as the time-series database, and Grafana for log visualization.
This will be explained in the Grafana UI below.
Configure a data source in Grafana
by adding Loki, as previously demonstrated in the referenced article. After completing this step, proceed to the Explore section to generate metrics.
In this process, we aim to create metrics that specifically highlight logs containing the term "Docker."
Follow these steps:
Open the Explore section.
Create a metric with the following specifications:
Label Filters: jobs, varlogs
Line Contains: docker
Execute the query by clicking on "Run Query."
Once you obtain the desired output, organize the results into a new dashboard. Prior to this, assign the name "Docker Logs" to the upcoming dashboard.
Setting up Telegraf
It is a crucial step in establishing a robust data collection system for monitoring, observability, and performance analysis. Telegraf serves as a cornerstone for creating scalable and efficient monitoring solutions in diverse environments.
Follow these steps to configure Telegraf on your EC2 instance:
Install Telegraf through the apt package manager:
sudo apt-get install telegraf
Verify the status of the Telegraf service:
sudo systemctl status telegraf
Modify the Telegraf configuration file to enable the collection of Docker logs:
sudo vim /etc/telegraf/telegraf.conf
After making the necessary changes, restart the Telegraf service and verify its status:
sudo service telegraf restart
sudo service telegraf status
To ensure proper permissions, grant access to the Docker socket:
sudo chmod 666 /var/run/docker.sock
These steps will effectively configure Telegraf on your EC2 instance, allowing it to collect Docker logs and contribute to a comprehensive monitoring solution.
Configuring InfluxDB
To establish the backend storage for Telegraf, follow the steps below for configuring InfluxDB on your EC2 instance:
Install InfluxDB using the Ubuntu apt package manager:
sudo apt install influxdb
Download the InfluxDB client:
sudo apt install influxdb-client
Open the InfluxDB shell by executing the following command in your terminal:
influx
Within the InfluxDB shell, create the "telegraf" database using the following command:
CREATE DATABASE telegraf
Navigate to the Telegraf configuration file and enable the database configuration to establish the connection between Telegraf and InfluxDB:
vim /etc/telegraf/telegraf.conf
After making the necessary changes, restart the Telegraf service to apply the configuration updates:
sudo service telegraf restart
By following these steps, you've effectively configured InfluxDB as the backend storage for Telegraf, allowing you to store and analyze time series data for monitoring, performance analysis, and observability of your systems and applications.
Dashboard Creation and Configuration
Before creation Configure the data source in Grafana which we have installed in the above steps:
Open Grafana and navigate to the Data Sources section.
Add a new data source, select "InfluxDB," and enter the InfluxDB URL (e.g.,
http://localhost:8086).
Set the database name to "Telegraf" and configure authentication if needed.
Click "Save & Test" to confirm the connection.
Now, InfluxDB is configured as a data source in Grafana with the Telegraf database.
Creating the Dashboard: Displaying Key Metrics
We will create a comprehensive dashboard with the following data:
Total Containers:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In the SELECT section, choose n_containers to show the total number of containers on the server.
Running Containers:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In the SELECT section, choose n_containers_running to display the total number of running containers on the server.
Stopped Containers:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In the SELECT section, choose n_containers_stopped to show the total number of stopped containers on the server.
Images:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In the SELECT section, choose n_images to display the total number of images on the server.
Containers Memory:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In the SELECT section, choose field(usage_percent) & last(). This shows the usage percentage of containers last used according to the set time.
Containers Uptime:
Choose the data source as InfluxDB.
In the FROM section, select docker to configure Docker data.
In SELECT, choose field(uptime_ns) last() alias(uptime_ns). This displays the uptime of each container currently running on the server.
Finalizing the Dashboard
Finally, add all the individual unit graphs and tables to the dashboard. Your comprehensive dashboard is now ready for monitoring various aspects of your Docker environment!
Thanks for reading until here. See you in the next one.