The documentation in the Claude AI website is clear and detailed so I will focus on the details needed to set up monitoring and clarify usage of additional attributes and labels for filtering.

Claude Usage Monitoring

Claude has a built-in OpenTelemetry exporter to expose usage metrics and logs. You can then use a Grafana dashboard to track cost and token usage per user, department, or any other criteria. It can be enabled and configured using environment variables or settings.json. In a corporate environment, settings.json can be predefined and locked.

AWS Parameter Store to Kubernetes Secrets

You can use AWS Parameters Store to keep your secrets in the safe Cloud storage. Using ESO - External Secrets Operator, you can pull them from AWS and create local Kubernetes Secrets in your namespace.

Install ESO - External Secrets Operator

External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  # --set installCRDs=false

Create IAM Policy with RO access to the store

# Create IAM Policy to enable RO access to Parameter Store
data "aws_iam_policy_document" "ssm_read" {
	statement {
		effect = "Allow"

		actions = [
			"ssm:GetParameter",
			"ssm:GetParameters",
			"ssm:GetParametersByPath"
		]

		resources = ["*"]
	}
}

Create IAM User and attach the policy

# Create IAM user 
resource "aws_iam_user" "eso" {
        name = "tf-parameter-store-ro"
}

resource "aws_iam_user_policy" "ssm_read" {
        name = "ssm-read"
        user = aws_iam_user.eso.name
        policy = data.aws_iam_policy_document.ssm_read.json
}

Create AWS Credentials

# Create Access Key for the RO user used in external K8s ESO
resource "aws_iam_access_key" "eso" {
        user = aws_iam_user.eso.name
}

# Output AWS Credentials
output "access_key_id" {
        value = aws_iam_access_key.eso.id
}

output "secret_access_key" {
        value = aws_iam_access_key.eso.secret
        sensitive = true
}

Configure ESO

Create K8s Secret with AWS credentials

kubectl create secret generic aws-eso-ro \
        --from-literal=access-key-id=YOUR_KEY_ID \
        --from-literal=secret-access-key=YOUR_SECRET_KEY

Create Secret Store

Connect microk8s to AWS Parameter Store using IAM user with RO only access. Use the CRD defined by ESO:

Filelog receiver is used to collect logs from the server where otelcol-contrib is running. In otelcol-contrib v111, there are several bugs in transform and some other processors causing log processing to fail. Here is working example to extract and set service_name (as label in Grafana) from the log files following specif pattern. Logs are coming from different apps and in the server they are following the rule:

/some/path/log/services/<service_name/*.log

We will extract <service_name> from directory path and set it as resource.attribute. Loki and Grafana will use this is label and it will show up in Grafana’s drop down for filtering.