Home

Welcome to the Data.gov wiki. Pages are grouped below by topic — see the sidebar for the full alphabetical list.

System components

The applications and sites that make up Data.gov, plus how to manage them.

• Site components — Map of everything that makes up Data.gov
• www.data.gov — Public homepage (WordPress, cloud.gov Pages)
• catalog.data.gov — Dataset search/discovery app (Python/Flask + OpenSearch)
• harvest.data.gov — Automated metadata ingestion service
• inventory.data.gov — Agency-facing metadata management (CKAN)
• Static sites — resources.data.gov, strategy.data.gov
• Updating organization logos on Catalog — How to add/update an org's logo

Harvesting

How agency metadata gets pulled in, and how to debug it when it doesn't.

• harvest.data.gov — Current harvest pipeline: architecture, terminology, job types
• Harvest O&M and Debugging — Operational procedures, Harvest API query examples
• Broken Harvest Sources — Diagnosing agency-reported harvest source issues
• Examples of Harvest Job Errors — Catalog of known error types and what they mean

Operations & monitoring

Runbooks, on-call responsibilities, and where to look when something's wrong.

• Runbook — Primary incident/on-call runbook
• Operation and Maintenance Responsibilities — O&M role and duties
• Operation and Maintenance Event Notes — Log of past O&M events
• Automated O&M Tasks — Scheduled jobs/actions that self-report failures
• Log Review — What's expected vs. what needs investigation, New Relic
• Metrics — Jobs producing usage metrics
• Dataset metrics — GA4 + New Relic dataset usage tracking
• Google Analytics — GA account structure and access
• Break Glass deployment — Emergency deploys outside normal CI/CD
• New Relic — Account structure, NerdGraph API access, alerting/dashboard reference (includes license key rotation steps)
• Cloud.gov Apps Spaces Memory Usage — Memory usage tracking spreadsheet

Infrastructure & security

Platform, networking, auth, and the security processes we follow around them.

• cloud.gov — Platform reference and helpful links
• DNS — DNS change process (via 18F, Terraform)
• TLS SSL Certificates — Certificate management
• Firewall Change Requests — Ingress/egress firewall process
• GSA VPN — VPN access to BSP environments
• WAF and rate limiting — AWS WAF behavior and rate-limit handling
• Egress & ingress proxies — Why/how we proxy traffic
• SAML2 authentication — SAML2 setup (inventory.data.gov only)
• Login.gov SAML certificate rotation steps — Annual cert rotation process
• Add Login.gov as Authentication — Integrating Login.gov (Flask apps)
• Dependency scanning — Snyk/Dependabot triage
• Significant Change rubric — Security Impact Analysis rubric
• Checklist for new repositories — Pre-production repo checklist

Team & process

Onboarding, collaboration norms, and day-to-day team conventions.

• Onboarding Offboarding — New team member setup and access
• Pull Request Template — Current per-repo PR templates
• Slack — Channel structure and usage
• Team Collaboration and Pair Programming — Meeting/pairing conventions
• Docker Best Practices — Docker/Compose conventions
• Managing Forks — Fork ownership guidelines
• Data.gov Mission and Strategy — Team history and mission
• Coding Best Practices — ⚠ Flagged outdated, needs review
• Email — ⚠ TODO, tracked in issue #3798

Legacy CKAN reference (inventory.data.gov only)

catalog.data.gov migrated off CKAN to a custom Python/Flask app in 2025. These pages now apply only to inventory.data.gov, which remains CKAN-based, unless noted otherwise.

• CKAN commands — Common CKAN commands
• CKAN API tasks — Recurring CKAN API tasks
• Catalog CKAN 2.8 upgrade — Historical CKAN 2.8 upgrade notes
• Solr on ECS - EFS — Solr ops (inventory only; catalog uses OpenSearch)
• Duplicate Harvest Source Cleanup and Other Nuclear Harvest Fixes — Manual harvest-source cleanup (legacy H1.0)
• Stuck Harvest Fix — Stuck-harvest procedures for legacy H1.0
• Geospatial metadata — Geospatial metadata standards (mostly still current)
• Collections — Collections handling

Deprecated / superseded

Kept for history — not needed day-to-day.

• Alerts Event - Standard Operating Procedure → moved to Runbook
• Web Application Security Scans → moved to Operation and Maintenance Responsibilities
• Keypair Rotation — deprecated
• Nessus Agent Upgrade — deprecated
• BSP or FCS requests — being phased out