Skip to content

feat: backpressure, search budget and fast ngram comparison#10

Open
loris-av wants to merge 1 commit intolivingdata-co:mainfrom
loris-av:feat/backpressure-and-perf-optimizations
Open

feat: backpressure, search budget and fast ngram comparison#10
loris-av wants to merge 1 commit intolivingdata-co:mainfrom
loris-av:feat/backpressure-and-perf-optimizations

Conversation

@loris-av
Copy link
Copy Markdown

Summary

Three performance improvements for production workloads:

1. Queue backpressure

  • createQueue({maxSize}) rejects new requests when the queue is full (HTTP 503)
  • Purges aborted requests before rejecting to reclaim slots
  • Configurable via ADDOK_MAX_QUEUE_SIZE env var (default: numNodes * 4)

2. Request timeout

  • Requests waiting in queue are rejected after a configurable delay (HTTP 408)
  • Configurable via ADDOK_REQUEST_TIMEOUT env var (default: 5000 ms)
  • Prevents clients from hanging indefinitely under load

3. Python performance optimizations (bridge.py)

Search budget — Caps the collector pipeline time to prevent slow queries from blocking workers:

  • Monkey-patches Search.__call__ and Search.intersect to track elapsed time
  • When budget exceeded and results already found, skips remaining intersections
  • Configurable via ADDOK_SEARCH_BUDGET_MS env var (default: 200 ms, 0 to disable)
  • Logs budget hits to stderr for monitoring

fast_compare_ngrams — Pure Python bigram comparison ~3x faster than ngram.NGram.compare():

  • Same algorithm (bigram intersection / union ratio) but avoids re-creating NGram objects
  • Replaces addok.helpers.results.compare_ngrams at startup
  • Benchmarked on 5K production queries: identical result quality, 3x faster scoring

Test plan

  • Lint passes (xo)
  • All 56 tests pass (including 8 new queue tests)
  • Load test with production-like traffic

Queue backpressure:
- Add maxSize option to createQueue to reject requests when full
- Purge aborted requests before rejecting to reclaim slots
- Add totalSize() and purgeAborted() methods
- Configurable via ADDOK_MAX_QUEUE_SIZE env (default: numNodes * 4)

Request timeout:
- Add requestTimeout option to reject queued requests after delay
- Configurable via ADDOK_REQUEST_TIMEOUT env (default: 5000ms)
- Returns HTTP 408 on timeout, 503 on queue full

Python performance optimizations in bridge.py:
- Search budget: cap collector pipeline time (ADDOK_SEARCH_BUDGET_MS, default 200ms) to prevent slow queries from blocking workers indefinitely
- fast_compare_ngrams: pure Python bigram comparison ~3x faster than ngram.NGram.compare(), avoiding object re-creation on every call

Tests:
- Add queue tests for maxSize, purgeAborted, totalSize
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant