Claude Usage Monitoring using built-in Otel Exporter
The documentation in the Claude AI website is clear and detailed so I will focus on the details needed to set up monitoring and clarify usage of additional attributes and labels for filtering.
Claude Usage Monitoring
Claude has a built-in OpenTelemetry exporter to expose usage metrics and logs. You can then use a Grafana dashboard to track cost and token usage per user, department, or any other criteria. It can be enabled and configured using environment variables or settings.json. In a corporate environment, settings.json can be predefined and locked.
External Secrets From Aws Parameter Store
AWS Parameter Store to Kubernetes Secrets
You can use AWS Parameters Store to keep your secrets in the safe Cloud storage. Using ESO - External Secrets Operator, you can pull them from AWS and create local Kubernetes Secrets in your namespace.
Install ESO - External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
# --set installCRDs=false
Create IAM Policy with RO access to the store
# Create IAM Policy to enable RO access to Parameter Store
data "aws_iam_policy_document" "ssm_read" {
statement {
effect = "Allow"
actions = [
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath"
]
resources = ["*"]
}
}
Create IAM User and attach the policy
# Create IAM user
resource "aws_iam_user" "eso" {
name = "tf-parameter-store-ro"
}
resource "aws_iam_user_policy" "ssm_read" {
name = "ssm-read"
user = aws_iam_user.eso.name
policy = data.aws_iam_policy_document.ssm_read.json
}
Create AWS Credentials
# Create Access Key for the RO user used in external K8s ESO
resource "aws_iam_access_key" "eso" {
user = aws_iam_user.eso.name
}
# Output AWS Credentials
output "access_key_id" {
value = aws_iam_access_key.eso.id
}
output "secret_access_key" {
value = aws_iam_access_key.eso.secret
sensitive = true
}
Configure ESO
Create K8s Secret with AWS credentials
kubectl create secret generic aws-eso-ro \
--from-literal=access-key-id=YOUR_KEY_ID \
--from-literal=secret-access-key=YOUR_SECRET_KEY
Create Secret Store
Connect microk8s to AWS Parameter Store using IAM user with RO only access. Use the CRD defined by ESO:
Opentelemetry Filelog Example
Filelog receiver is used to collect logs from the server where otelcol-contrib is running. In otelcol-contrib v111, there are several bugs in transform and some other processors causing log processing to fail. Here is working example to extract and set service_name (as label in Grafana) from the log files following specif pattern. Logs are coming from different apps and in the server they are following the rule:
/some/path/log/services/<service_name/*.log
We will extract <service_name> from directory path and set it as resource.attribute. Loki and Grafana will use this is label and it will show up in Grafana’s drop down for filtering.