SOLR-17798: Integrate SDK OTLP metric exporter

[mlbiscoc]

https://issues.apache.org/jira/browse/SOLR-17798

In SolrMetricManager add a MetricExporter for OTLP metrics to be pushed via GRPC or HTTP.

By default the OTLP exporter is turned off and Solr only exposes metrics as pull through the /admin/metrics endpoint.
The following configuration parameters can be passed as system properties or environment variables:

System property | env variable

solr.otlpMetricExporterEnabled | SOLR_OTLP_METRIC_EXPORTER_ENABLED = true/false
solr.otlpMetricExporterProtocol | SOLR_OTLP_METRIC_EXPORTER_PROTOCOL = grpc/http/none
solr.otlpMetricExporterInterval | SOLR_OTLP_METRIC_EXPORTER_INTERVAL = <ms time>

[mlbiscoc]

I tested this with a OTEL collector using the following collector config:

otel-config.yml
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
  telemetry:
    metrics:
      address: 0.0.0.0:8888
      level: detailed

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: default

Solr pushes the metrics to the collectors receiver with grpc then you just curl the collector endpoint and the exporter gives you prometheus

curl 'localhost:8889/metrics'

I also think this feature branch is at a point where PRs for the migration to OTEL can start happening in parallel across handlers and different classes. Might start making additional PRs for migrations

[mlbiscoc]

We are registering both the prometheus reader for our metrics endpoint and the periodic reader to push metrics to the OTLP exporter.

[mlbiscoc]

It's debatable if this should be here in Core. Should we actually move this into the Open Telemetry module for the OTLP exporter? OTLP tracing exporter is enabled with module only so maybe metrics as well.

[mlbiscoc]

60 seconds is the default according to otel spec so copied it here but users can change the interval as they wish.

[mlbiscoc]

Doing some testing, since we use the getDefault for both grpc and http, it actually reads some default env variables without us having to instrument our own read and configure for the exporter. For example, OTEL_EXPORTER_OTLP_ENDPOINT can change the endpoint where the metrics are pushed but I need to find all the different parameters that this picks up from documentation somewhere.

[mlbiscoc]

Adding in trace based exemplars. If you have Solr distributed trace on and push with OTLP to something like the OTEL collector, you can get exemplars on the metrics honored across the pipeline like so:

solr_metrics_core_requests_times_milliseconds_bucket{collection="demo",core="demo_shard1_replica_n1",handler="/update",job="unknown_service:java",replica="replica_n1",shard="shard1",le="50.0"} 26 1.751574499605e+09 # {trace_id="4438406367ea8ea60f53e3f6e26a409d",span_id="d797ab840ed14e6d"} 30.0 1.75157449662e+09

But this is directly pinned to your traces sampling rate.