Gitlab is using 2 different ways to manage and provision resources in your Kubernetes clusters. You can use GitOps way running FluxCD or you can connect your cluster to Gitlab using gitlab-agent and use kubectl commands directly in your .gitlab-ci.yml. If you decide to use gitlab-agent it will install additional POD into your cluster using Helmto keep 2-way communication between cluster and Gitlab. In your Gitlab repo go to section Operate -> Kubernetes clusters and create the new cluster. Save the agentID. ...

The most simple way to run small and cheap Kubernetes cluster in the Cloud. You take the smallest Hetzner server with public IP address and install k3s, for easy management you can use k9s and you are ready to start. Install sudo curl -sfL https://get.k3s.io | sh - This will install and start Kubernetes in your host. Configure cp /etc/rancher/k3s/k3s.yaml ~/.kube/config chown <user>:<group> ~/.kube/config Then you access your k8s from the host using kubectl: ...

The biggest chalenge in building Observability platform for big company is how to make strict policy for matrics labeling and automate configuration of Otel Collectors config files. This could be resolved by using some fleet management solution. Until today, I was not able to find full working open source solution, so you may need to create your own. However you resolve the above issue, it is good practice to have automated check after your mass deployment to see what servers picked up new config and sending signals labeled according to your latest configuration. ...

Microk8s and K3S have RBAC enabled and configured by default. Microk8s configuration has AlwaysAllow policy, so even if you create and assign RBAC role for new user, microk8s will still allow full access. In K3S it is different and RBAC role will be applied. Create User Key openssl genrsa -out <user-name>.key 2048 Creates user’s private key. Create Certificate Signing Request openssl req -new -key <user-name>.key -out <user-name>.csr -subj "/CN=<user-name>/O=<user-group>" Using users private/public key pair and request Certificate Authority to sign and verify the user. ...

After using microk8s for a while, there may be increased disk usage due to a number of docker images saved in /var dir. You can delete these images using crictl tool. It is not part of microk8s installation, so you need to download first. Install crictl Download crictl Connect to containerd sudo ./crictl -r unix:///var/snap/microk8s/common/run/containerd.sock image lis/ Delete unused images sudo ./crictl -r unix:///var/snap/microk8s/common/run/containerd.sock rmi --prune

Application Performance Monitoring is the ultimate level of observability in your systems. This comes on top of infra, network and other types of monitoring, providing info about the health and perfrmance of your applications or services. OpenTelemetry is CNCF project providing standard standard way to collect telemetry data. Supports metrics, traces, and logs with vendor-neutral APIs and SDKs. It goes together with Grafana supported stack to store and visualize signals collected from monitored systems. The stack is made of OpenTelemetry SDK/API, Opentelemetry Collector to collect and transfer signals from applications, DBs, servers and other components of your system. Othe part is made of Grafana stack to store, search and visualize signals: Mimir (Prometheus) to store metrics, Loki to store logs and Tempo to store traces. ...

Kubernetes is orchestrator for apps deployed in containers. Kubernetes Architecture Master Nodes Manage Kubernetes platform and administer worker nodes running the containers (workloads) Kube API Exposes the API so other components can communicate. ETCD Distributed key-value database to track state of the whole system. Controller Manager Take care to maintain desired state of the system. Scheduler Distributes PODs to nodes based on different criterias. ...

Mimir is Grafana’s scalable and distributed timeseries database, like Prometheus, for storing metrics. In distributed mode, it has several scalable components and if deployed in AWS it could be configured to be zone aware to provide resilient architecture. Main compnents in distributed mode, like Compactor, could be using PVCs to store temporary data before moving to S3 bucket. When to load is increased, you may need to expand the storage. The goal is to do this without losing any data and without any restart. ...

Alertmanager is part of Mimir. It will store rules and check them against the received metrics. When the rule is triggered it will send notification to defined notification chanels. It provides API so you can automate alerts provisioning. You could keep alerts under source control and create them from CICD. Alert Alerts are defined in yaml files. Here is sample: # alert.yaml groups: - name: cpu_alerts interval: 30s rules: - alert: HighCPUUsage expr: system_cpu_time_seconds > 100 for: 1m labels: severity: warning annotations: summary: "High CPU usage detected" description: "CPU time exceeded threshold" Curl Mimir API curl -X POST http://<MIMIR_URL>/api/v1/rules/cpu_alerts \ -H "Content-Type: application/yaml" \ --data-binary @alert.yaml