[DBA-140] Fix BigQuery credentials_json auth in DuckDB ATTACH#271
[DBA-140] Fix BigQuery credentials_json auth in DuckDB ATTACH#271gasparian wants to merge 6 commits into
Conversation
DuckDB's BigQuery extension only recognizes credentials_file (a file path), not credentials_json. When BigQueryServiceAccountJsonAuth is used, write the JSON credentials to a temporary file and substitute credentials_file in the connection string. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ion string DuckDB's BigQuery extension does not accept credentials in the ATTACH connection string. Use CREATE SECRET with SERVICE_ACCOUNT_PATH or SERVICE_ACCOUNT_JSON to pass credentials via DuckDB's secret manager, which is the mechanism the extension actually supports. Verified manually against real BigQuery with both credentials_json and credentials_file auth types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Fixes BigQuery authentication with DuckDB by moving service-account credentials out of the ATTACH connection string (which DuckDB’s BigQuery extension rejects) and into DuckDB’s secret manager.
Changes:
- Create a DuckDB secret for BigQuery auth (service account file or inline JSON) before running
ATTACH. - Remove credentials from the BigQuery
ATTACHconnection string generation. - Update unit tests to assert
CREATE SECRETbehavior and ordering.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
databao/agent/databases/bigquery_adapter.py |
Adds _create_secret_if_needed() and updates DuckDB registration flow to use secrets for BigQuery auth. |
tests/test_bigquery_adapter.py |
Updates tests to validate secret creation, exclude creds from ATTACH, and assert correct call ordering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot review feedback: - Use DuckDB parameter binding (?) instead of string interpolation to prevent SQL injection and handle special characters safely - Use CREATE OR REPLACE SECRET for idempotency - Escape double quotes in secret name Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use TYPE bigquery (lowercase) in CREATE SECRET for consistency with ATTACH and other adapters - Escape double quotes in name for ATTACH SQL identifier Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Prevent invalid SQL / injection if additional_properties values contain single quotes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…tion string Prevent credentials_json/credentials_file from leaking into the ATTACH connection string if they end up in additional_properties. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
Unknown key in connection string: credentials_jsonerror when using BigQuery with service account credentialsCREATE SECRET_create_secret_if_needed()that usesSERVICE_ACCOUNT_PATHorSERVICE_ACCOUNT_JSONvia DuckDB's secret managerChanges
Change 1: Use CREATE SECRET for BigQuery auth
Credentials are no longer passed in the ATTACH connection string. Instead,
register_in_duckdbcreates a DuckDB secret with the appropriate auth parameter before attaching.Affected files
databao/agent/databases/bigquery_adapter.pyChange 2: Update tests
Tests now verify that
CREATE SECRETis called with the correct parameters and that credentials are excluded from the ATTACH SQL.Affected files
tests/test_bigquery_adapter.pyTest Plan
make checkpasses (ruff, mypy)datalore-internalproject) with bothcredentials_jsonandcredentials_fileauth types🤖 Generated with Claude Code