feat: go export microservice drop in with gRPC comms and thin API shim

[PeterBaker0]

Summary

Introduces a Go gRPC export sidecar that performs the heavy lifting of notebook exports while Conductor keeps auth, download-token validation, and HTTP response headers unchanged. The API gains a thin bridge (exportHttpStreamer) that streams gRPC FileChunks straight to the browser and falls back to the existing Node exporters when the sidecar is unavailable.

This is a drop-in rollout: with no export-service configured, behaviour stays on the legacy Node path.

Go export service (export-service/)

• New gRPC service implementing all export formats: CSV, photo ZIP, GeoJSON, KML, full notebook ZIP, and JSON records
• Reads directly from CouchDB; streams output in configurable chunks (32 KiB default)
• Shared protobuf contract at proto/export.proto with committed Go bindings under internal/pb
• Health RPC for readiness checks
• Optional shared-secret auth via x-export-service-secret metadata
• Ports key Node export behaviour: deleted related-record stripping, attachment field detection (TakePhoto + FileUploader), CSV escaping, spatial empty output, structured gRPC error codes
• Optional runtime profiling (EXPORT_PROFILE=true) for memory, goroutines, and active export stats
• Dockerfile for container deployment; CI runs go test ./... and go build on every PR

API shim (api/src/services/)

• exportServiceClient — lazy gRPC client from shared proto; maps export payloads to protobuf requests
• exportHttpStreamer — unified routing for all download routes and GET /:id/records/
• exportTypes — shared TypeScript types for formats, rollout mode, and gRPC payloads
• Rollout controlled by env:
  • EXPORT_SERVICE_GRPC_ADDRESS / EXPORT_SERVICE_GRPC_URL — sidecar address
  • EXPORT_SERVICE_MODE — auto (default), grpc, or legacy
  • EXPORT_SERVICE_SHARED_SECRET — optional mutual auth
  • EXPORT_SERVICE_GRPC_DEADLINE_MS — optional call deadline
• auto mode: pre-stream gRPC failures fall back to legacy Node exporters; mid-stream failures destroy the HTTP response (partial downloads cannot be safely restarted)
• grpc mode: no fallback — sidecar must be available
• Tests default to legacy mode so existing API tests are unaffected

Configuration

Conductor (API)

Variable	Default	Description
EXPORT_SERVICE_GRPC_ADDRESS	(unset)	gRPC address of the sidecar; unset = legacy only
EXPORT_SERVICE_MODE	auto	auto, grpc, or legacy
EXPORT_SERVICE_SHARED_SECRET	(unset)	Optional shared secret sent as gRPC metadata
EXPORT_SERVICE_GRPC_DEADLINE_MS	(unset)	Optional gRPC call deadline in ms

Go export service

Variable	Default	Description
EXPORT_GRPC_ADDR	:9090	Listen address
COUCHDB_INTERNAL_URL	http://localhost:5984	CouchDB base URL
COUCHDB_USER / COUCHDB_PASSWORD	admin / password	CouchDB credentials
EXPORT_PROFILE	false	Print runtime stats every second

See export-service/README.md for local development commands.

Test plan

• cd export-service && go test ./... — unit tests for CSV escaping, filenames, deleted related refs, attachments, spatial output, auth, error mapping
• cd api && pnpm test — includes exportServiceStreamer.test.ts (grpc piping, legacy mode, mid-stream failure, auto fallback)
• CI Build & Lint workflow passes Go export-service step
• Legacy path (no sidecar): leave EXPORT_SERVICE_GRPC_ADDRESS unset; verify CSV, photo ZIP, GeoJSON, KML, full export, and JSON records download as before
• Auto rollout: set EXPORT_SERVICE_GRPC_ADDRESS=localhost:9090, EXPORT_SERVICE_MODE=auto; start sidecar; confirm exports stream via gRPC; stop sidecar and confirm pre-stream fallback to Node
• Forced gRPC: set EXPORT_SERVICE_MODE=grpc; confirm exports fail clearly when sidecar is down (no silent fallback)
• Shared secret: set matching EXPORT_SERVICE_SHARED_SECRET on API and sidecar; confirm unauthenticated gRPC calls are rejected
• Large notebook: full export with attachments — compare output size and filenames against legacy Node export
• Client disconnect: cancel a large download mid-stream; confirm sidecar export is cancelled cleanly