External Secrets From Aws Parameter Store

AWS Parameter Store to Kubernetes Secrets

You can use AWS Parameters Store to keep your secrets in the safe Cloud storage. Using ESO - External Secrets Operator, you can pull them from AWS and create local Kubernetes Secrets in your namespace.

Install ESO - External Secrets Operator

External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  # --set installCRDs=false

Create IAM Policy with RO access to the store

# Create IAM Policy to enable RO access to Parameter Store
data "aws_iam_policy_document" "ssm_read" {
	statement {
		effect = "Allow"

		actions = [
			"ssm:GetParameter",
			"ssm:GetParameters",
			"ssm:GetParametersByPath"
		]

		resources = ["*"]
	}
}

Create IAM User and attach the policy

# Create IAM user 
resource "aws_iam_user" "eso" {
        name = "tf-parameter-store-ro"
}

resource "aws_iam_user_policy" "ssm_read" {
        name = "ssm-read"
        user = aws_iam_user.eso.name
        policy = data.aws_iam_policy_document.ssm_read.json
}

Create AWS Credentials

# Create Access Key for the RO user used in external K8s ESO
resource "aws_iam_access_key" "eso" {
        user = aws_iam_user.eso.name
}

# Output AWS Credentials
output "access_key_id" {
        value = aws_iam_access_key.eso.id
}

output "secret_access_key" {
        value = aws_iam_access_key.eso.secret
        sensitive = true
}

Configure ESO

Create K8s Secret with AWS credentials

kubectl create secret generic aws-eso-ro \
        --from-literal=access-key-id=YOUR_KEY_ID \
        --from-literal=secret-access-key=YOUR_SECRET_KEY

Create Secret Store

Connect microk8s to AWS Parameter Store using IAM user with RO only access. Use the CRD defined by ESO:

Opentelemetry Filelog Example

Filelog receiver is used to collect logs from the server where otelcol-contrib is running. In otelcol-contrib v111, there are several bugs in transform and some other processors causing log processing to fail. Here is working example to extract and set service_name (as label in Grafana) from the log files following specif pattern. Logs are coming from different apps and in the server they are following the rule:

/some/path/log/services/<service_name/*.log

We will extract <service_name> from directory path and set it as resource.attribute. Loki and Grafana will use this is label and it will show up in Grafana’s drop down for filtering.

K8s Monitoring Servicemonitor

K8s-monitoring can use prometheus-operator CRDs to discover services for monitoring and scraping. Here are few steps to help troubleshoot your setup.

Enable Prometheus-Operator in Helm


# @section -- Features - Prometheus Operator Objects
prometheusOperatorObjects:
  # -- Enable gathering metrics from Prometheus Operator Objects.
  # @section -- Features - Prometheus Operator Objects
  enabled: true

This feature is using alloy-metrics component, so it must be enabled, too.

Deploy app with exposed metrics

I used bitnami/nginx to test the setup.

O11y | Observability