Intro

k8s-monitoring will deploy several Alloy components to collect signals from the Kubernetes cluster. Each Alloy components will expose own metrics in :12345/metrics. There is no scrape job enabled by default, so they are not sent together with other metrics. Here you have useful info about total number of sent metrics, logs, traces or total number of failed metrics, logs, traces.

Config

You can add scraper in the extraConfig section under your components.

The documentation in the Claude AI website is clear and detailed so I will focus on the details needed to set up monitoring and clarify usage of additional attributes and labels for filtering.

Claude Usage Monitoring

Claude has a built-in OpenTelemetry exporter to expose usage metrics and logs. You can then use a Grafana dashboard to track cost and token usage per user, department, or any other criteria. It can be enabled and configured using environment variables or settings.json. In a corporate environment, settings.json can be predefined and locked.

AWS Parameter Store to Kubernetes Secrets

You can use AWS Parameters Store to keep your secrets in the safe Cloud storage. Using ESO - External Secrets Operator, you can pull them from AWS and create local Kubernetes Secrets in your namespace.

Install ESO - External Secrets Operator

External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  # --set installCRDs=false

Create IAM Policy with RO access to the store

# Create IAM Policy to enable RO access to Parameter Store
data "aws_iam_policy_document" "ssm_read" {
	statement {
		effect = "Allow"

		actions = [
			"ssm:GetParameter",
			"ssm:GetParameters",
			"ssm:GetParametersByPath"
		]

		resources = ["*"]
	}
}

Create IAM User and attach the policy

# Create IAM user 
resource "aws_iam_user" "eso" {
        name = "tf-parameter-store-ro"
}

resource "aws_iam_user_policy" "ssm_read" {
        name = "ssm-read"
        user = aws_iam_user.eso.name
        policy = data.aws_iam_policy_document.ssm_read.json
}

Create AWS Credentials

# Create Access Key for the RO user used in external K8s ESO
resource "aws_iam_access_key" "eso" {
        user = aws_iam_user.eso.name
}

# Output AWS Credentials
output "access_key_id" {
        value = aws_iam_access_key.eso.id
}

output "secret_access_key" {
        value = aws_iam_access_key.eso.secret
        sensitive = true
}

Configure ESO

Create K8s Secret with AWS credentials

kubectl create secret generic aws-eso-ro \
        --from-literal=access-key-id=YOUR_KEY_ID \
        --from-literal=secret-access-key=YOUR_SECRET_KEY

Create Secret Store

Connect microk8s to AWS Parameter Store using IAM user with RO only access. Use the CRD defined by ESO: