Alertmanager is part of Mimir. It will store rules and check them against the received metrics. When the rule is triggered it will send notification to defined notification chanels. It provides API so you can automate alerts provisioning. You could keep alerts under source control and create them from CICD. Alert Alerts are defined in yaml files. Here is sample: # alert.yaml groups: - name: cpu_alerts interval: 30s rules: - alert: HighCPUUsage expr: system_cpu_time_seconds > 100 for: 1m labels: severity: warning annotations: summary: "High CPU usage detected" description: "CPU time exceeded threshold" Curl Mimir API curl -X POST http://<MIMIR_URL>/api/v1/rules/cpu_alerts \ -H "Content-Type: application/yaml" \ --data-binary @alert.yaml

SNMP is used to monitor many different devices like routers, UPSs, Storages, etc… The ultimate SRE startegy is to inify all monitoring systems into one observability platform and get unique place where you see all info needed in case of emergency. Here is how you can use OpenTelemetry to collect SNMP data from your devices and send it to Prometheus or Mimir. Then you can use Grafana to visualize and create alerts. ...

Opentelemetry collector can help us to get detailed database metrics querying database system tables and views, parsing the results and creating metrics and logs for observability platform, ie. Prometheus or Mimir for metrics and Loki for logs. Basic requirement could be to see top 10 queries consuming the most of your resources. Strategy OpenTelemetry Configuration Strategy MySql and MS SQL assign unique label to each query (digest or query hash) so you can use this field (column) to correlate metrics to logs. This will improve your data ingestion on the observability platform side. Otherwise, you could end up with metrics with huge labels (full query text) and this can affect performanse or indexing in Prometheus or Mimir. ...

In big deployments Grafana may have big number of Orgs, datasources, dashboards. If you need to automate provisioning of Grafana resources, you may try to create your own scripts and tools or try to find some open source solutions to cover your use case. This comes down to 2 options: use Grafana API or use file based provisioning. Typical Use Case The typical use could be: create Azure AD group for users ...

I’m using goreleaser to create binaries for different OS and create release in GitLab. Gorelease will compile the code and based on the config file in your repo create required binaries for different OS and architectures. It will create dist/ dir in your repo, create .gitignore to avoid pushing dist/ into GitLab and calculate checksums for packages. Text file with checksums will be also deployed to GitLab Release page. You need to create Access Token with api access in GitLab and export GITLAB_TOKEN environment variable before you run goreleaser. ...

This is small tool to help you interact with Grafana API. It could be used to create new Organization, add new user and provision basic datasources from templates, ie. Prometheus, Loki, Tempo. You can test it using small demo code but you need to have your grafana instance running and available in some URL. You also need to have local Grafana user/pass with admin privileges. export DEV_GRAFANA_URL=your-url export DEV_GRAFANA_USER=your-user export DEV_GRAFANA_PASS='your-pass-with-special-chars' Grafana Struct Keeps basic data about target Grafana like url, username, password and base64 encoded user:pass needed for Basic Authentication. This will be added into Authorization Header. ...

Useful curl commands to interact with Grafana API. Create ORGs, datasources, etc… # Get all datasources curl -X GET https://GRAFANA_URL/api/datasources -H "Content-Type: application/json" -u 'user:pass' | jq # Get all Organizations curl -X GET https://GRAFANA_URL/api/orgs -H "Content-Type: application/json" -u 'user:pass' | jq # Create new Organization curl -X POST -u 'USER:PASS' -H "Content-Type: application/json" https://GRAFANA_URL/api/orgs -d '{"name": "New Organization"}'

Intro Grafana API allows us to automate admin tasks. We have big number of Orgs mapped to Azure AD groups and we were looking for a way to automate provisioning. Process goes like this: create new Org, provision datasources (with same UIDs), provision dashboards (with same UIDs). In this way all dashboards are working within the Org resusing all URLs, etc… Simple Perl Script to Provision Grafana ORG #! /usr/bin/perl use HTTP::Tiny; use JSON::MaybeXS; use MIME::Base64 qw(encode_base64); my $user = $ENV{'USER'}; my $pass = $ENV{'PASS'}; my %url = ('dev' => 'https://dev-grafana/api/orgs', 'prod' => 'https://prod-grafana/api/orgs'); if(@ARGV != 2) { die "Usage: $0 <org-name> <dev|prod>\n"; } my $org_name = $ARGV[0]; my $environment = $ARGV[1]; die "Environment must be dev or prod\n" if $environment !~ /^(dev|prod)$/; # Create ORG and Get Org ID my $data = { name => $org_name }; my $json = encode_json($data); my $http = HTTP::Tiny->new; my $credentials = encode_base64("$user:$pass", ''); # no newline my $org_id; my $headers = { "Content-Type" => "application/json", "Authorization" => "Basic $credentials" }; my $response = $http->request( 'POST', $url{$environment}, { headers => $headers, content => $json }); if ($response->{success}) { my $resp_data = decode_json($response->{content}); $org_id = $resp_data->{orgId}; } else { die "Failed to create org: $response->{status} $response->{reason}\n"; } print "Created org ID: $org_id\n";

O11y, shorthand for Observability - O + 11 letters + y :-) Observability is the ability to understand the internal state of a system by examining only the data it outputs. Three Pillars Metrics Quantitative measurements collected over time. Useful for dashboards, alerts, and trend analysis. Traces Traces show the journey of a request through a system. Comprised of spans, which represent operations. Useful for understanding latency, bottlenecks, and dependencies. ...

Intro Besides industry stundards using oauth2 Google or Github for authentication of users, you may need to have simple user/pass based authentication. This may be implemented in many different ways, ie using Basic Authentication on the Web Server side. If you would like to implement it using Passport on your backend, here are few point to consider: use HTTPS end to end, keep users and passwords in secure place, use crypto lib. ...