📊 Observability & Telemetry

This document details the configuation and design of the cluster-wide telemetry scraping infrastructure utilizing Prometheus, Grafana, and native Node Exporter.

📈 Observability Architecture

Prometheus scrapes node metrics periodically via Node Exporter agents listening on port 9100 across the bridge network:

graph TD
    %% My Color Palette
    classDef hostNode fill:#161d1c,stroke:#415854,color:#f8f8f2,stroke-width:2px;
    classDef ipdNode fill:#2b3b38,stroke:#ff9580,color:#ff9580,stroke-width:1.5px;
    classDef vmNode fill:#2b3b38,stroke:#70a99f,color:#f8f8f2,stroke-width:1.5px;
    classDef obsNode fill:#2b3b38,stroke:#8aff80,color:#8aff80,stroke-width:1.5px;

    VM1["🔑 freeipa.lab.local (172.30.1.85)<br>LDAP / Kerberos / BIND DNS"]:::ipdNode

    subgraph Nodes ["Monitored Nodes (Port: 9100)"]
        HostOS["🖥️ Hypervisor Host (172.30.1.200)"]:::hostNode
        VM2["📄 portfolio VM (172.30.1.93)"]:::vmNode
        VM3["⚔️ minecraft VM (172.30.1.91)"]:::vmNode
        VM4["🎵 navidrome VM (172.30.1.92)"]:::vmNode
    end

    subgraph PortfolioServices ["portfolio VM Services"]
        Prom["📈 Prometheus TSDB"]:::obsNode
        Grafana["📊 Grafana Dashboard"]:::obsNode
    end

    HostOS & VM1 & VM2 & VM3 & VM4 -.->|Node Exporter Scrape| Prom
    Prom -->|Data Source Query| Grafana

    Nodes --->|DNS Lookups| VM1

    %% Subgraph Colors
    style Nodes fill:#212c2a,stroke:#70a99f,stroke-width:1px;
    style PortfolioServices fill:#161d1c,stroke:#415854,stroke-width:2px;

📄 Service Configurations

Telemetry collectors are automated via ansible/playbooks/07_observability.yml:

1. Prometheus Node Exporter (Daemon Node)

Engine: Downloads the native node_exporter-1.8.2.linux-amd64 release binary and places it under /usr/local/bin/node_exporter.
Systemd Integration (node_exporter.service): Deploys a background daemon unit configured to run the exporter immediately after network interfaces load.
SELinux Contexts (AlmaLinux): Automatically runs restorecon to preserve SELinux contexts for the binary and service files.
Security: Opens port 9100/tcp on firewalld on RedHat-family systems to allow scrapers to read metric inputs.

2. Prometheus Engine (`prometheus.yml.j2`)

Engine: Containerized using docker.io/prom/prometheus:latest running with --net=host on the portfolio VM.
Storage Mounts: Maps the host folder /home/sho/containers/prometheus/data to /prometheus with tag properties :z,U(ensuring rootless Podman SELinux permissions map correctly to local storage directories).
Scrape Loop Specifications: Sets a scrape and evaluation interval of 15 seconds:

global:
    scrape_interval: 15s
    evaluation_interval: 15s

scrape_configs:
    - job_name: 'homelab-nodes'
      static_configs:
        - targets:
            - '172.30.1.200:9100' # Host
            - '172.30.1.85:9100' # freeipa
            - '172.30.1.93:9100' # portfolio
            - '172.30.1.91:9100' # minecraft
            - '172.30.1.92:9100' # navidrome

3. Grafana Dashboard

Engine: Runs docker.io/grafana/grafana-oss:latest in a container mapping port 3000:3000.
Data Persistence: Mounts /home/sho/containers/grafana/data to preserve custom dashboards, datasources, and user configurations between restarts.

🚀 Execution & Monitoring

To deploy the observability stack across the cluster nodes, run the following:

ansible-playbook site.yml --tags "observability" --ask-vault-pass

Checking Scraping Sinks

Prometheus Targets Console: Access the Prometheus TUI interface by opening http://172.30.1.93:9000/targets and verify that all 5 target hosts report UP. 2 Grafana Portal: Navigate to http://172.30.1.93:3000 to create custom query dashboards. (Default port is mapped externally via Cloudflared Tunnel).

3.9 KiB Raw Blame History