The biggest chalenge in building Observability platform for big company is how to make strict policy for matrics labeling and automate configuration of Otel Collectors config files. This could be resolved by using some fleet management solution. Until today, I was not able to find full working open source solution, so you may need to create your own.
However you resolve the above issue, it is good practice to have automated check after your mass deployment to see what servers picked up new config and sending signals labeled according to your latest configuration.
There are several ways to do it, but basically it comes to using exposed APIs from Prometheus, Mimir, Loki or Tempo to create smart API request and get some query based on your input server list.
Another way could be to use Grafana as proxy to send query to any of the above (or defined) data sources in their format and get the result you need. This may be more suitable in Kubernetes environment because you may avoid exposing all used data sources using ingresses or other methods.
Here is simple tool you can use to perform Grafana Datasource Queries from the CLI:
./grafanactl query --ds-uid mimir --query 'up{host_name="YOUR-HOST"}'
and you will get JSON response from Grafana API, which can be parsed to create any kind of report you need for your automated tests.