Application Performance Monitoring is the ultimate level of observability in your systems. This comes on top of infra, network and other types of monitoring, providing info about the health and perfrmance of your applications or services.

OpenTelemetry is CNCF project providing standard standard way to collect telemetry data. Supports metrics, traces, and logs with vendor-neutral APIs and SDKs. It goes together with Grafana supported stack to store and visualize signals collected from monitored systems. The stack is made of OpenTelemetry SDK/API, Opentelemetry Collector to collect and transfer signals from applications, DBs, servers and other components of your system. Othe part is made of Grafana stack to store, search and visualize signals: Mimir (Prometheus) to store metrics, Loki to store logs and Tempo to store traces.

Signals collectd by this platform will be correlated so it will be easy for developer and other teams involved in troubleshooting or performance tuning to understand the state of the app/service or to find root cause for the failure.

This is essential tool for modern, distributed, systems like microservices deployed in Kubernetes environment. In this scenario, PODs or components could die or new instances could be created, so IP addresses and traditional approach do not apply. Monitoring must focus on service (by name), not on servers and IPs.

Here is summary how to install and configure the observability platform to collect signals (metrics, traces, logs) from your OpenTelemetry instrumented application (zero-code or manual instrumentation).

My configuration is based on Hetzner Cloud running Ubuntu sevrer. On top of Ubuntu, there is MicroK8s Kubernetes cluster for the apps.

Helm Charts

# Provides Grafana, Loki, Tempo, K8s-monitoring
helm repo add grafana https://grafana.github.io/helm-charts

# Provides Prometheus database for Metrics
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create Namespace for Observability System
kubectl create namespace monitoring

Grafana Stack

Grafana

Download default values.yaml:

https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml

Enable persistence and create manually Persistent Volume and Persistent Volume Claim:

# PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: "/var/lib/grafana"


# PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi  # Matches the size in values.yaml
  volumeName: grafana-pv  # Must match your PV name

Check the status:

kubectl get pv -n monitoring
kubectl get pvc -n monitoring

# status: bound

Install Grafana into your k8s cluster:


helm install grafana grafana/grafana -n monitoring -f ./values.yaml

Loki

https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml

Helm failed to start without hostpath-storage. This is used to auto provision storage using PVC, PV and storageclass.

microk8s enable hostpath-storage

Then, install:

helm install loki grafana/loki -n monitoring -f ./values.yaml

Loki will start with auth enabled so all queries must come with specific HTTP header set: X-Scope-OrgId. Grafana will not be able to add it as datasource without this additional setting.

Tempo

https://github.com/grafana/helm-charts/blob/main/charts/tempo/values.yaml

helm install tempo grafana/tempo -n monitoring -f ./values.yaml

Tempo must be configured to support service graphs. This will generate and send additional metrics to Prometheus.

metricsGenerator:
    enabled: true
    remoteWriteUrl: "http://prometheus-server.monitoring.svc.cluster.local/api/v1/write"

K8s-monitoring

https://artifacthub.io/packages/helm/grafana/k8s-monitoring/0.2.10?modal=values

helm install alloy grafana/k8s-monitoring -n monitoring -f ./values.yaml

Prometheus

https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml

helm install prometheus prometheus-community/prometheus -n monitoring -f ./values.yaml

This will install Prometheus, Alertmanager, kube-state-metrics, push-gateway and node-exporter.

APM Instrumentation