Alertmanager is part of Mimir. It will store rules and check them against the received metrics. When the rule is triggered it will send notification to defined notification chanels.

It provides API so you can automate alerts provisioning. You could keep alerts under source control and create them from CICD.

Alert

Alerts are defined in yaml files. Here is sample:

# alert.yaml
groups:
  - name: cpu_alerts
    interval: 30s
    rules:
      - alert: HighCPUUsage
        expr: system_cpu_time_seconds > 100
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU time exceeded threshold"

Curl Mimir API

curl -X POST http://<MIMIR_URL>/api/v1/rules/cpu_alerts \
     -H "Content-Type: application/yaml" \
     --data-binary @alert.yaml