Skip to content

catalog.data.gov

Jump to bottom
David Aguiar edited this page Jul 23, 2026 · 32 revisions

Application Repo

a.k.a Catalog is the public-facing dataset discovery and search application for Data.gov. It serves 515,000+ datasets from 120+ federal, state, municipal, university, and tribal publishing organizations. This is the main app of Data.gov and is generally what folks are thinking about when they refer to Data.gov.

This is a custom Python/Flask application that replaced the legacy CKAN-based catalog in 2025. It reads from the shared harvest database managed by datagov-harvester (it does not write to it -- all dataset metadata is written by the harvester) and uses OpenSearch for full-text search.

Environments

Instance Url
Production catalog.data.gov
Staging catalog-stage.data.gov
Development catalog-dev.data.gov
Legacy catalog (CKAN-based, running through fall 2026) catalog-old.data.gov

All environments run on cloud.gov, cf space names prod, staging, and development.

Architecture

  • Web app: Python/Flask, served via NGINX proxy on cloud.gov
  • Database: Shared Postgres instance managed by datagov-harvester (datagov-harvest-db service) -- read-only. The SQLAlchemy models are duplicated locally in app/models.py for isolation; the app interacts with the shared DB through CatalogDBInterface (app/database/interface.py)
  • Search: OpenSearch (((app_name))-opensearch cloud.gov service)
  • Storage: S3 for sitemaps and static assets
  • Monitoring: New Relic
  • Logging: Logstack (cloud.gov log drain)

Data flow: datagov-harvester pulls agency metadata and writes it to the shared Postgres DB → datagov-catalog reads from Postgres and keeps an OpenSearch index in sync → the Flask app serves search and browse pages from OpenSearch, and dataset/organization detail pages from Postgres.

Local development

Prerequisites: Docker + Docker Compose, Python 3.x with Poetry, Node.js/npm (static assets + accessibility testing).

cp .env.sample .env  # first-time setup, values can be left as defaults for local dev
make install-static  # install static assets (needs npm)
make up              # start local docker env
make load-test-data  # load fixture data

Then visit the app locally and iterate. Useful commands:

  • make test -- full Python test suite
  • make test-pa11y -- accessibility (pa11y-ci) tests, requires the app running
  • make lint-check / make lint-fix -- ruff, isort, black
  • make poetry-update -- keep local Poetry in sync with what CI uses

Install the git pre-commit hooks once per clone (pip install pre-commit && pre-commit install) to run the same formatters on staged files automatically before each commit; CI still runs make lint-check across the whole tree.

See the datagov-catalog README for the full environment variable reference (DB connection, OpenSearch host, New Relic, S3/sitemap credentials, etc).

Search

Search is powered by OpenSearch, kept in sync with Postgres via a daily batch job (flask search sync clears the index and re-indexes every dataset). See docs/opensearch.md for how indexing and cursor-based pagination (search_after/after) work.

Search filters

Dataset search supports filters (keyword, organization, organization type, publisher, geography, spatial data, collection). Each filter is a single FilterDefinition registered in app/search/filters/ that owns its own request parsing, URL serialization, OpenSearch clause building, sidebar rendering, and OpenAPI docs -- adding a new filter means writing one new module and registering it, not touching four different files.

Performance

See docs/load-test.md for ApacheBench comparisons against the legacy catalog. Summary: search response times are comparable except for very broad queries with many matches (an open area for improvement); page load times are dramatically better across the board (e.g. dataset detail pages saturate at ~120 req/s vs. ~5 req/s on the legacy catalog).

Collections

Some datasets are grouped into collections, where a parent dataset represents a set of related child datasets (e.g. a series of related files from the same agency program). Collections work differently depending on how the child datasets are harvested:

  • For WAF sources, datagov-harvester has a dedicated waf-collection source type: each source has a collection_parent_url, and the harvester creates a parent dataset that the child records point back to (see harvester/harvest.py).
  • More generally, a dataset's isPartOf field marks it as a member of a collection. The catalog indexes dcat.isPartOf in OpenSearch and exposes a collection search filter (app/search/filters/collection.py) so members of a collection can be queried together.

This replaced the CKAN-era mechanism (collection_package_id, described on the Collections wiki page), which only applies to inventory.data.gov now.

Duplicate datasets

The old CKAN-based system could harvest the same dataset twice under different identifiers, requiring manual cleanup via SQL queries and a standalone datagov-dedupe tool. That failure mode no longer exists in the current pipeline: datagov-harvester's HarvestSource.filter_duplicate_identifiers() (in harvester/harvest.py) checks every incoming record's identifier against the others in the same job before processing. Duplicates are recorded as a harvest_record_error (rather than becoming a dataset) and dropped from the job.

If you're debugging why a dataset didn't show up after a harvest, check for duplicate-identifier errors on the job via the Harvest API rather than looking for duplicate rows in dataset -- see Harvest O&M and Debugging for query examples.

Logs

See New Relic and/or the cloud.gov wiki page for how to review logs.

Automated tasks

GitHub Actions (.github/workflows) run scheduled and on-push jobs against the current catalog:

  • Restart -- rolling-restarts datagov-catalog and datagov-catalog-proxy on staging and prod every 15 minutes
  • Create Sitemaps -- runs flask sitemap generate and flask sitemap verify as cloud.gov tasks daily (and on demand per environment)
  • Snyk Scan -- weekly dependency vulnerability scan across all projects
  • Check DB Models -- on every push/PR, diffs app/models.py and shared/constants.py against the copies in datagov-harvester to catch drift between the two repos' duplicated schema

See Automated O&M Tasks for the equivalent automation on the legacy catalog (catalog-old.data.gov) and inventory.data.gov, which still use CKAN-era GitHub Actions.

Deployment

Deployments run automatically via GitHub Actions (deploy.yml) on push to main:

  1. Lint -- ruff
  2. Deploy to staging -- deploys to the staging cloud.gov space, runs a smoke test
  3. Deploy to prod -- deploys to the prod cloud.gov space after staging succeeds, runs a smoke test
  4. Lighthouse CI audit -- runs against prod after deploy

For emergency deployments outside of the normal CI/CD pipeline, see Break Glass deployment.

Related resources

This page previously described the legacy CKAN-based catalog (SOLR, ckan-php-manager, apache2/Ansible ops). That system is now catalog-old.data.gov; this page covers the current Flask/OpenSearch app. Updated 2026-07-23.

Clone this wiki locally