Deploying a Prometheus and Grafana Cluster Using Application Centric Deployment

ACD and its benefits
The in-house developed application centric deployment (short: ACD) plugin helps you deploy multi host applications with the click of a button. Applications are based on Ansible playbooks and application definitions.

Generally, an application consists of multiple services and therefore requires different kinds of hosts connected to each other. These hosts have dependencies on each other, for example, an application server might require a database server. The number of deployed hosts depends on the expected usage of the application. You can scale an application instance vertically via compute profile or horizontally by choosing multiple hosts to run the same service to handle increased workloads.  Refer to the application centric deployment guide for more information. This example shows how to deploy an application consisting of two services: Prometheus to collect metrics and Grafana to visualize data.

Prometheus and Grafana
Prometheus and Grafana is a monitoring stack to store and visualize time series data. Prometheus collects and stores metrics. Grafana provides a browser based interface to analyze and visualize data.
The dashboard contains any numbers of widgets to display metrics and resulting key performance indicators, for example “Performance as handled requests per second” or “Utilization as percentage of used CPU and memory resources”. Monitoring metrics leads to a better understanding of your computing environment and better decision making.

Requirements for this example

• orcharhino configured to deploy hosts running CentOS 7

• configured host group for CentOS 7, including necessary content for hosts, e.g. packages that will be installed via an Ansible task

• one time root access to orcharhino via ssh to put the Ansible playbook into /etc/foreman/plugins/foreman_acd/ansible-playbooks/

• orcharhino enabled to perform smart proxy functionality with remote execution for its subnet

Preparations in the terminal

1. Download the ACD playbook from github.

   ssh root@orcharhino.example.com
   cd /tmp/
   wget https://github.com/ATIX-AG/acd_playbook_for_prometheus_grafana/archive/refs/heads/main.zip
   
   unzip -d /etc/foreman/plugins/foreman_acd/ansible-playbooks/ main.zip
   rm -f /tmp/main.zip
   

2. Write down the path of the directory and the Ansible playbook.

    ls /etc/foreman/plugins/foreman_acd/ansible-playbooks/acd_playbook_for_prometheus_grafana-main/
   /etc/foreman/plugins/foreman_acd/ansible-playbooks/acd_playbook_for_prometheus_grafana-main/
   play_prometheus_grafana.yml
   

Setting up the Ansible Playbook in orcharhino

1. Navigate to Applications > Ansible Playbooks and click the New Ansible Playbook button.

2. Enter a name and both the path and name of the Ansible playbook file as shown above.

3. Click the Import groups button to import Ansible group variables after submitting the Ansible playbook to orcharhino. 

Setting up an Application Definition in orcharhino

1. Navigate to Applications > App Definitions and click the New Application Definition button.

2. Enter a name and select the Ansible playbook from the step earlier.

3. Define two services: prometheus and grafana.

   We recommend using a host group with CentOS 7 for both services.
   The used compute profile (mostly CPU and RAM) depends on the size and interval of scraping metrics.

4. Click Submit to save your application definition to orcharhino.

The two steps above are usually handled by an administrative user.
Once this has been prepared, users with restricted rights can conveniently define and deploy multi host applications as follows:

Define an application instance in orcharhino

1. Navigate to Applications > App Instances and click the New Application Instance button.

2. Enter a name and select the application definition from the step earlier.

3. Using the + sign, add hosts to the two services from the application definition.

In this example, we create two hosts: one Prometheus node and one Grafana node.

Deploy an application instance

1. Navigate to Applications > App Instances and click the Deploy button.
   Grab a cup of coffee while your hosts are being deployed and configured.
   This process might take a while depending on your environment, compute resource provider, and network connectivity.

2. View the last deployment by clicking the Show Report link.

You now have a fully functional Prometheus and Grafana cluster consisting of one Prometheus node ready to collect data and one Grafana node to visualize your data.
Start collecting metrics using the node exporter or third party software that stores metrics which are fetched by Prometheus.
There are plenty of community created solutions to export metrics to Prometheus.

Comment or tweet us @orcharhino which metrics you are collecting and how this helps you. One community example is to monitor a compute resource like a VMware vSphere. Refer to the orcharhino documentation for more information.