feat(api): add an admin only Prometheus metrics endpoint

[matanbaruch]

@DaniilSmirnov you offered to take this on your fork back in DeviceFarmer/stf#860, so here it is, ported to devicehub rather than copied over. The upstream version is DeviceFarmer#893.

Adds GET /api/v1/metrics, returning the device, user and group counters of the platform in the Prometheus text exposition format, so device usage can be graphed in Prometheus and Grafana.

Counters are computed on scrape, not on a timer, so there is no extra unit, no background job and no polling interval to tune. A scrape is three Mongo reads.

What is devicehub specific

This is not the stf patch with the names changed. The parts that had to be rewritten:

1. Device states follow your state machine in ui/src/lib/utils/get-device-state.util.ts, not stf's. That includes the two cases stf does not have: status PREPARING (6) on an Apple device counts as available, and status UNHEALTHY (7) gets its own state.
2. Mongo instead of RethinkDB. getDevicesForMetrics() in the device model reads the collection with a projection limited to present, status, ready, owner, manufacturer and provider.name, so a scrape does not pull whole device documents.
3. paths/metrics.js is generated, not hand written. I ran python3 lib/units/api/gen_routes.py. Heads up that the generator also rewrites the header comment of eight paths/team*.js files that were edited by hand at some point, so their committed form no longer matches the generator output. I reverted those to keep this diff clean, but they will keep showing up for anyone who runs it.
4. Controllers use the specific models (DeviceModel, UserModel, GroupModel) rather than the deprecated concatenated dbapi default export.

Access control

The operation is tagged admin, and the controller checks req.user.privilege itself. The second check is not redundant here. accessTokenAuth() has a branch for req.headers.channel and req.headers.device that returns true without ever setting req.user, so a caller coming through that path would otherwise reach the endpoint with no privilege check and crash on req.user. The controller answers 403 for that and for simple users.

Worth knowing: the counters cover every device, user and group regardless of group membership, which is exactly what a simple user must not see. That is why this is admin only rather than scoped per user. If you want per group metrics later, that is a different endpoint with a group label and a membership filter, and it should not reuse this one.

What it returns

Metric	Labels	Description
devicehub_devices_total		Devices known to DeviceHub, present or not
devicehub_devices_by_state	state	Devices per aggregate device state
devicehub_devices_available		Devices in the available state
devicehub_devices_busy		Devices in the busy state
devicehub_providers_total		Distinct providers serving at least one present device
devicehub_users_total		Users known to DeviceHub
devicehub_users_by_privilege	privilege	Users per privilege
devicehub_groups_total		Groups known to DeviceHub
devicehub_groups_active		Groups currently active
devicehub_groups_by_state	state	Groups per group state
devicehub_groups_by_class	class	Groups per group class

state values are absent, offline, unauthorized, preparing, busy, available, unhealthy and present. The using and automation states of the device table are not exposed, since they are relative to the user looking at the device and mean nothing to a scraper.

Every label value is known up front, so a counter that drops to zero is exported as zero instead of disappearing from the output, which is what you want if you are alerting on it. No metric is labeled with a user email, so there is no PII in the endpoint and no unbounded cardinality.

doc/METRICS.md has the scrape config, a docker compose snippet for Prometheus and Grafana, the curl command and the expected output.

Verification

1. npm run lint: 0 errors, no new warnings in the added files
2. npm run typecheck: clean
3. npm run build, then the metrics module exercised against the built output: 10 device state cases including the Apple PREPARING and UNHEALTHY ones, the aggregation, the exposition format and the label reset behaviour
4. The generated route and the controller loaded from the built output in the same import order the api unit uses, then called directly with the models stubbed: a simple user and a request with no req.user both get 403 JSON, an admin gets 200 with Content-Type: text/plain; version=0.0.4; charset=utf-8 and correct counters

There is no test script in package.json and no JS unit test harness in the repo, so there was nowhere to land the state machine tests as a committed suite. test/api is pytest against a running stack and would need the generated client regenerated for the new operation. If you want either the pytest coverage or a JS unit runner, tell me which and I will send it separately.

I have not run this against a live devicehub with Mongo and real devices. If you have a staging environment, that is the one thing left to confirm.

Unrelated finding

While verifying the above I hit a latent import cycle that is already on master and has nothing to do with this PR, but you probably want to know about it.

Importing any model index as the first module of a process throws:

$ node -e "import('./.build/lib/db/models/device/index.js')"
ReferenceError: Cannot access 'DeviceModel' before initialization
    at .build/lib/db/api.js:11

db/api.js builds its concatenated default export from the five model indexes, and a model index reaches back into db/api.js before that export is initialized. It is harmless in the running app because api/index.js pulls helpers/securityHandlers.js, which imports db/api.js and initializes the graph in a safe order before express-openapi loads any controller. It bites anything that loads a model in isolation, which is most of the ways you would write a unit test. Reproduced on fd1eee58 with no changes applied.