K8s-Monitoring Self Monitoring

Intro

k8s-monitoring will deploy several Alloy components to collect signals from the Kubernetes cluster. Each Alloy components will expose own metrics in :12345/metrics. There is no scrape job enabled by default, so they are not sent together with other metrics. Here you have useful info about total number of sent metrics, logs, traces or total number of failed metrics, logs, traces.

Config

You can add scraper in the extraConfig section under your components.

collectors:
  alloy-metrics:
    enabled: true
  
    extraConfig: |
      prometheus.scrape "alloy_self" {
        targets = [{
          __address__ = "localhost:12345",
        }]
        job_name        = "alloy"
        scrape_interval = "30s"
        forward_to      = [otelcol.receiver.prometheus.otel_gateway.receiver]
      }

Verify

Query your Mimir or Prometheus for one of the otelcol_ metrics.

otelcol_exporter_sent_metric_points_total

You should see result similar to:

otelcol_exporter_sent_metric_points_total{cluster="docker-desktop", component_id="otelcol.exporter.otlphttp.otel_gateway", component_path="/", exporter="otlphttp/otelcol.exporter.otlphttp.otel_gateway", instance="localhost:12345", job="alloy", k8s_cluster_name="docker-desktop"}

Interesting Metrics

otelcol_exporter_send_failed_metric_points_total
otelcol_exporter_send_failed_log_records_total
otelcol_exporter_send_failed_spans_total

otelcol_receiver_failed_log_records_total
otelcol_receiver_failed_metric_points_total
otelcol_receiver_failed_spans_total