Intro #
k8s-monitoring will deploy several Alloy components to collect signals from the Kubernetes cluster. Each Alloy components will expose own metrics in :12345/metrics. There is no scrape job enabled by default, so they are not sent together with other metrics. Here you have useful info about total number of sent metrics, logs, traces or total number of failed metrics, logs, traces.
Config #
You can add scraper in the extraConfig section under your components.
collectors:
alloy-metrics:
enabled: true
extraConfig: |
prometheus.scrape "alloy_self" {
targets = [{
__address__ = "localhost:12345",
}]
job_name = "alloy"
scrape_interval = "30s"
forward_to = [otelcol.receiver.prometheus.otel_gateway.receiver]
}
Verify #
Query your Mimir or Prometheus for one of the otelcol_ metrics.
otelcol_exporter_sent_metric_points_total
You should see result similar to:
otelcol_exporter_sent_metric_points_total{cluster="docker-desktop", component_id="otelcol.exporter.otlphttp.otel_gateway", component_path="/", exporter="otlphttp/otelcol.exporter.otlphttp.otel_gateway", instance="localhost:12345", job="alloy", k8s_cluster_name="docker-desktop"}
Interesting Metrics #
otelcol_exporter_send_failed_metric_points_total
otelcol_exporter_send_failed_log_records_total
otelcol_exporter_send_failed_spans_total
otelcol_receiver_failed_log_records_total
otelcol_receiver_failed_metric_points_total
otelcol_receiver_failed_spans_total