Expand Pvc for Grafana Mimir on AWS EKS

Mimir is Grafana’s scalable and distributed timeseries database, like Prometheus, for storing metrics. In distributed mode, it has several scalable components and if deployed in AWS it could be configured to be zone aware to provide resilient architecture. Main compnents in distributed mode, like Compactor, could be using PVCs to store temporary data before moving to S3 bucket. When to load is increased, you may need to expand the storage. The goal is to do this without losing any data and without any restart.

Check Storage Class

Storage class must allow PVC to be expanded. In AWS, you can check:

kubectl get storageclass

# allowVolumeExpand: true

Update PVC yaml

Edit or patch PVC for Compactor in the namespace where it is deployed, ie: mimir

kubectl edit pvc mimir-compactor-0 -n <mimir-namespace>

spec:
  resources:
    requests:
      storage: 100Gi

After you save, it will start to expand the volume size.

Delete StatefulSet

In distributed Mimir environment, Compactor PODs are controled by StatefulSet. You can delete it but instruct not delete PODs:

kubectl delete sts <name> -n <namespace> --cascade=orphan

This will delete StatefulSet and make PODs orphans - they will remain running.

Update HELM values

Now you can update values.yaml in your repo for Mimir and upgrade Helm

helm upgrade mimir grafana/mimir-distributed -f values.yaml -n <mimir-namespace>