Skip to content

Commit 30b5269

Browse files
Merge pull request #1060 from emteebug12-jpg/feat/quotas-health-probe
feat: add /api/quotas/health dependency probe
2 parents aabfb68 + b77e21d commit 30b5269

5 files changed

Lines changed: 373 additions & 0 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,7 @@ The migration is in `migrations/0019_disputes.sql` (rollback: `migrations/0019_d
140140
- JSON body parsing plus gateway API key authentication for upstream proxy routes
141141
- Per-user global REST rate limiting for authenticated `/api/billing`, `/api/usage`, `/api/developers`, `/api/vault`, and `/api/keys` traffic, with IP fallback for unauthenticated requests
142142
- Per-user token-bucket rate limiting for all `/api/quotas` traffic (capacity and refill rate independently configurable via `QUOTA_RATE_LIMIT_CAPACITY` / `QUOTA_RATE_LIMIT_REFILL_RATE`); exceeded requests return `HTTP 429` with a `Retry-After` header and the standardised error envelope
143+
- Quota dependency probe: `GET /api/quotas/health` reports the status of `/api/quotas`'s external dependencies (currently the database) for ops dashboards/alerting, mirroring the `{ status, timestamp, dependencies }` shape of `GET /api/health/dependencies`; no auth required, subject to the same `/api/quotas` rate limit; see [docs/quotas-health-probe.md](./docs/quotas-health-probe.md)
143144
- In-memory `VaultRepository` with:
144145
- `create(userId, contractId, network)`
145146
- `findByUserId(userId, network)`

docs/quotas-health-probe.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Quotas Dependency Probe
2+
3+
**`GET /api/quotas/health`** reports the status of the external dependencies the `/api/quotas` route group relies on — for ops dashboards, alerting, and SRE runbooks.
4+
5+
This endpoint requires no authentication (it exposes no tenant data, only aggregate dependency status), matching `GET /api/health/dependencies`. It is, however, subject to the same per-user/IP token-bucket rate limit as every other route under `/api/quotas` (see [README.md — What's included](../README.md#whats-included), `QUOTA_RATE_LIMIT_CAPACITY` / `QUOTA_RATE_LIMIT_REFILL_RATE`).
6+
7+
---
8+
9+
## Why this exists
10+
11+
`/api/quotas/counts` and the wider quota subsystem (`src/services/quotaService.ts`) ultimately depend on the shared PostgreSQL database for quota-request data and usage aggregation. Before this endpoint, there was no way to check that dependency's health without going through `/api/health/dependencies` (which reports on the *whole app's* dependencies, not specifically the ones `/api/quotas` needs) or the admin-only `/api/admin/health/probes`. `GET /api/quotas/health` fills that gap with a subsystem-scoped, publicly-reachable probe.
12+
13+
---
14+
15+
## Response shape
16+
17+
```json
18+
{
19+
"status": "ok",
20+
"timestamp": "2026-07-29T12:00:00.000Z",
21+
"dependencies": {
22+
"database": { "status": "ok", "responseTime": 4 }
23+
},
24+
"correlationId": "5e4b3c9a-2f1d-4a6e-9c3b-1a2b3c4d5e6f"
25+
}
26+
```
27+
28+
On a database outage:
29+
30+
```json
31+
{
32+
"status": "down",
33+
"timestamp": "2026-07-29T12:00:03.000Z",
34+
"dependencies": {
35+
"database": { "status": "down", "responseTime": 2001, "error": "unavailable" }
36+
},
37+
"correlationId": "5e4b3c9a-2f1d-4a6e-9c3b-1a2b3c4d5e6f"
38+
}
39+
```
40+
41+
`error` is always a sanitized category (`unavailable`, `timeout`, or an `HTTP <status>` string) — never a raw driver error message, connection string, or hostname. See `sanitizeCheck()` in `src/routes/health/dependencies.ts` (reused here) for the exact rules.
42+
43+
`dependencies` currently reports one entry, `database`. If the quota subsystem grows a second external dependency (e.g. a queue or third-party API), it will appear here alongside `database` without changing the shape of existing keys.
44+
45+
---
46+
47+
## HTTP status codes
48+
49+
| Overall `status` | HTTP code | Meaning |
50+
|---|---|---|
51+
| `ok` | 200 | Database reachable and responding within threshold |
52+
| `degraded` | 200 | Database reachable but slow (> 1000 ms) |
53+
| `down` | 503 | Database unreachable, timed out, or returned an unexpected result |
54+
55+
---
56+
57+
## Correlation IDs
58+
59+
Every request is assigned a correlation ID the same way as `GET /api/quotas/counts`:
60+
61+
1. Echoes the inbound `x-correlation-id` header if present.
62+
2. Falls back to the request ID set by the global request-id middleware.
63+
3. Generates a fresh UUID v4 if neither is available.
64+
65+
The resolved value is returned in both the `X-Correlation-Id` response header and the JSON body's `correlationId` field, so callers can correlate probe results with their own logs.
66+
67+
---
68+
69+
## Structured logging
70+
71+
Each request logs a `[quotas/health] probe requested` entry on entry and a `[quotas/health] probe completed` (or `probe failed`) entry on exit, both tagged with `requestId` and `correlationId` for tracing.
72+
73+
---
74+
75+
## Configuration
76+
77+
No dedicated environment variables — the probe reuses the app's shared PostgreSQL pool (`DATABASE_URL` / `DB_*`, see `src/db.ts`) and the shared health-check timeout logic in `src/services/healthCheck.ts` (default 2000 ms, `degraded` above 1000 ms).
78+
79+
---
80+
81+
## Example request
82+
83+
```bash
84+
curl -s http://localhost:3000/api/quotas/health | jq
85+
```
86+
87+
---
88+
89+
## Relationship to other health endpoints
90+
91+
| Endpoint | Scope | Auth |
92+
|---|---|---|
93+
| `GET /api/health` | Whole app, summary only | No |
94+
| `GET /api/health/dependencies` | Whole app, per-dependency detail | No |
95+
| `GET /api/admin/health/probes` | Whole app, per-component detail, single-component drill-down | Admin |
96+
| `GET /api/quotas/health` | `/api/quotas` subsystem only | No |

src/routes/quotas.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ import { config } from "../config/index.js";
2929

3030
// Sub-route handlers
3131
import quotaCountsRouter from "./quotas/counts.js";
32+
import { createQuotaHealthRouter } from "./quotas/health.js";
3233

3334
export interface QuotasRouterDeps {
3435
/** Inject a custom rate-limit middleware, primarily for testing. */
@@ -67,6 +68,11 @@ export function createQuotasRouter(deps: QuotasRouterDeps = {}): Router {
6768
// developer's quota requests.
6869
router.use("/counts", quotaCountsRouter);
6970

71+
// GET /api/quotas/health — dependency probe (database) for ops/monitoring.
72+
// Mounted after the rate limiter above, so it shares the same per-user
73+
// token bucket as every other /api/quotas route.
74+
router.use("/health", createQuotaHealthRouter());
75+
7076
return router;
7177
}
7278

src/routes/quotas/health.test.ts

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
/**
2+
* Tests for src/routes/quotas/health.ts — GET /api/quotas/health
3+
*
4+
* Coverage targets (≥90% on changed lines):
5+
*
6+
* ✓ 200 + status "ok" when the database check succeeds
7+
* ✓ 503 + status "down" when the database check fails
8+
* ✓ response shape: { status, timestamp, dependencies: { database }, correlationId }
9+
* ✓ error messages are sanitized (no raw connection string / stack leakage)
10+
* ✓ correlation ID is echoed back when x-correlation-id is provided
11+
* ✓ a correlation ID is generated when none is provided
12+
* ✓ X-Correlation-Id response header is set
13+
* ✓ falls back to the shared app pool when no pool is injected
14+
* ✓ drain tracker middleware is applied (does not break normal responses)
15+
*/
16+
17+
jest.mock('better-sqlite3', () => {
18+
return class MockDatabase {
19+
prepare() { return { get: () => null }; }
20+
exec() {}
21+
close() {}
22+
};
23+
});
24+
25+
import express from 'express';
26+
import request from 'supertest';
27+
import type { Pool, QueryResult } from 'pg';
28+
import { createQuotaHealthRouter } from './health.js';
29+
import { errorHandler } from '../../middleware/errorHandler.js';
30+
31+
function buildApp(pool?: Pool) {
32+
const app = express();
33+
app.use(express.json());
34+
app.use('/api/quotas/health', createQuotaHealthRouter(pool ? { pool } : {}));
35+
app.use(errorHandler);
36+
return app;
37+
}
38+
39+
function createMockPool(queryResult: QueryResult | Error): Pool {
40+
return {
41+
query: async () => {
42+
if (queryResult instanceof Error) {
43+
throw queryResult;
44+
}
45+
return queryResult;
46+
},
47+
} as unknown as Pool;
48+
}
49+
50+
describe('GET /api/quotas/health', () => {
51+
it('returns 200 with status "ok" when the database check succeeds', async () => {
52+
const pool = createMockPool({ rows: [{ result: 1 }] } as QueryResult);
53+
const app = buildApp(pool);
54+
55+
const res = await request(app).get('/api/quotas/health');
56+
57+
expect(res.status).toBe(200);
58+
expect(res.body.status).toBe('ok');
59+
expect(res.body.timestamp).toEqual(expect.any(String));
60+
expect(res.body.dependencies.database.status).toBe('ok');
61+
expect(typeof res.body.dependencies.database.responseTime).toBe('number');
62+
});
63+
64+
it('returns 503 with status "down" when the database is unreachable', async () => {
65+
const pool = createMockPool(new Error('Connection refused'));
66+
const app = buildApp(pool);
67+
68+
const res = await request(app).get('/api/quotas/health');
69+
70+
expect(res.status).toBe(503);
71+
expect(res.body.status).toBe('down');
72+
expect(res.body.dependencies.database.status).toBe('down');
73+
});
74+
75+
it('sanitizes error messages to prevent leaking connection details', async () => {
76+
const pool = createMockPool(
77+
new Error('FATAL: connection to postgres://admin:s3cret@db.internal:5432/prod failed'),
78+
);
79+
const app = buildApp(pool);
80+
81+
const res = await request(app).get('/api/quotas/health');
82+
83+
expect(res.status).toBe(503);
84+
expect(res.body.dependencies.database.error).toBe('unavailable');
85+
const body = JSON.stringify(res.body);
86+
expect(body).not.toContain('s3cret');
87+
expect(body).not.toContain('db.internal');
88+
expect(body).not.toContain('postgres://');
89+
});
90+
91+
it('only reports the database dependency (quotas has no other external dependency today)', async () => {
92+
const pool = createMockPool({ rows: [{ result: 1 }] } as QueryResult);
93+
const app = buildApp(pool);
94+
95+
const res = await request(app).get('/api/quotas/health');
96+
97+
expect(Object.keys(res.body.dependencies)).toEqual(['database']);
98+
});
99+
100+
it('echoes the correlation ID when x-correlation-id is provided', async () => {
101+
const pool = createMockPool({ rows: [{ result: 1 }] } as QueryResult);
102+
const app = buildApp(pool);
103+
104+
const res = await request(app)
105+
.get('/api/quotas/health')
106+
.set('x-correlation-id', 'corr-quota-health-1');
107+
108+
expect(res.status).toBe(200);
109+
expect(res.body.correlationId).toBe('corr-quota-health-1');
110+
expect(res.headers['x-correlation-id']).toBe('corr-quota-health-1');
111+
});
112+
113+
it('generates a correlation ID when none is provided', async () => {
114+
const pool = createMockPool({ rows: [{ result: 1 }] } as QueryResult);
115+
const app = buildApp(pool);
116+
117+
const res = await request(app).get('/api/quotas/health');
118+
119+
expect(res.status).toBe(200);
120+
expect(typeof res.body.correlationId).toBe('string');
121+
expect(res.body.correlationId.length).toBeGreaterThan(0);
122+
expect(res.headers['x-correlation-id']).toBe(res.body.correlationId);
123+
});
124+
125+
it('falls back to the shared app pool when no pool is injected', async () => {
126+
// No pool override: the router falls back to src/db.ts's shared pool,
127+
// which is not reachable in this unit test environment, so the probe
128+
// should report the database as down rather than throwing.
129+
const app = express();
130+
app.use(express.json());
131+
app.use('/api/quotas/health', createQuotaHealthRouter());
132+
app.use(errorHandler);
133+
134+
const res = await request(app).get('/api/quotas/health');
135+
136+
expect([200, 503]).toContain(res.status);
137+
expect(res.body.dependencies.database).toBeDefined();
138+
}, 10_000);
139+
140+
it('applies the drain tracker middleware without breaking normal responses', async () => {
141+
const pool = createMockPool({ rows: [{ result: 1 }] } as QueryResult);
142+
const app = buildApp(pool);
143+
144+
const res = await request(app).get('/api/quotas/health');
145+
146+
expect(res.status).toBe(200);
147+
expect(res.body.dependencies).toBeDefined();
148+
});
149+
});

src/routes/quotas/health.ts

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
/**
2+
* Quota subsystem dependency probe — GET /api/quotas/health
3+
*
4+
* Reports the status of the external dependencies the `/api/quotas` route
5+
* group (this router plus src/routes/quotas/counts.ts and
6+
* src/services/quotaService.ts) relies on to function. Today that is the
7+
* shared PostgreSQL database: quota request data is ultimately persisted
8+
* there and `listQuotaRequests()` / usage aggregation both depend on it
9+
* being reachable.
10+
*
11+
* Response shape mirrors `/api/health/dependencies`
12+
* (src/routes/health/dependencies.ts) — same `{ status, timestamp,
13+
* dependencies }` envelope and the same {@link sanitizeCheck} rules — so ops
14+
* tooling can treat every dependency probe in the app uniformly. Designed
15+
* for monitoring dashboards / alerting, not for end users, so it does not
16+
* require authentication (matching the global dependencies probe).
17+
*
18+
* ### Correlation IDs & graceful shutdown
19+
* Mounts the same {@link correlationMiddleware} used by
20+
* src/routes/quotas/counts.ts so every request carries an `X-Correlation-Id`
21+
* for structured logging, and reuses the shared {@link quotasDrainTracker}
22+
* so in-flight probe requests are drained on shutdown along with the rest
23+
* of the `/api/quotas` surface.
24+
*
25+
* @module routes/quotas/health
26+
*/
27+
28+
import { Router } from 'express';
29+
import type { Request } from 'express';
30+
import type { Pool } from 'pg';
31+
import { pool as defaultPool } from '../../db.js';
32+
import {
33+
checkDatabase,
34+
determineOverallStatus,
35+
type ComponentCheck,
36+
type ComponentStatus,
37+
} from '../../services/healthCheck.js';
38+
import { sanitizeCheck } from '../health/dependencies.js';
39+
import { correlationMiddleware } from '../../middleware/correlation.js';
40+
import { quotasDrainTracker } from './counts.js';
41+
import { InternalServerError } from '../../errors/index.js';
42+
import { logger } from '../../logger.js';
43+
44+
/** Response body for GET /api/quotas/health. */
45+
export interface QuotaHealthProbeResponse {
46+
status: ComponentStatus;
47+
timestamp: string;
48+
dependencies: Record<string, ComponentCheck>;
49+
/** Correlation ID echoed back from the request context. */
50+
correlationId?: string;
51+
}
52+
53+
export interface QuotaHealthRouterDeps {
54+
/** Postgres pool to probe. Defaults to the shared app pool (src/db.ts). */
55+
pool?: Pool;
56+
/** Per-check timeout in ms, forwarded to {@link checkDatabase}. */
57+
timeoutMs?: number;
58+
}
59+
60+
/**
61+
* Builds the `/api/quotas/health` router.
62+
*
63+
* @param deps Optional dependency overrides — primarily for unit tests that
64+
* need to inject a mock pool to simulate a healthy or unreachable database.
65+
*/
66+
export function createQuotaHealthRouter(deps: QuotaHealthRouterDeps = {}): Router {
67+
const router = Router();
68+
const pool = deps.pool ?? defaultPool;
69+
70+
// Structured logging correlation ID, matching the rest of /api/quotas.
71+
router.use(correlationMiddleware);
72+
73+
// Count this request against the shared quotas in-flight drain tracker so
74+
// graceful shutdown waits for it just like any other /api/quotas request.
75+
router.use(quotasDrainTracker.middleware);
76+
77+
router.get('/', async (req: Request, res, next) => {
78+
const requestId = req.id || 'unknown';
79+
const correlationId = (req as Request & { correlationId?: string }).correlationId;
80+
81+
logger.info('[quotas/health] probe requested', { requestId, correlationId });
82+
83+
try {
84+
const dbCheck = await checkDatabase(pool, deps.timeoutMs);
85+
const dependencies: Record<string, ComponentCheck> = {
86+
database: sanitizeCheck(dbCheck),
87+
};
88+
89+
const overallStatus = determineOverallStatus({
90+
api: 'ok',
91+
database: dbCheck.status,
92+
});
93+
94+
logger.info('[quotas/health] probe completed', {
95+
requestId,
96+
correlationId,
97+
overallStatus,
98+
statuses: Object.fromEntries(
99+
Object.entries(dependencies).map(([key, value]) => [key, value.status]),
100+
),
101+
});
102+
103+
const response: QuotaHealthProbeResponse = {
104+
status: overallStatus,
105+
timestamp: new Date().toISOString(),
106+
dependencies,
107+
correlationId,
108+
};
109+
110+
const statusCode = overallStatus === 'down' ? 503 : 200;
111+
res.status(statusCode).json(response);
112+
} catch (error) {
113+
logger.error('[quotas/health] probe failed', { requestId, correlationId, error });
114+
next(new InternalServerError());
115+
}
116+
});
117+
118+
return router;
119+
}
120+
121+
export default createQuotaHealthRouter;

0 commit comments

Comments
 (0)