[Stream Firestore to BigQuery] Runtime Status: Processing Description Configuring BigQuery Sync

[james-wang-careerduck]

I am trying to use the extension to Stream Firestore to BigQuery. The initial setup triggered a number of errors which were all permission-related. I then granted the roles that seemed to be required to the service account. It has been a few hours now stuck in runtime status: processing. How much longer should I wait? Please advise.

[james-wang-careerduck]

@cabljac I would very much appreciate it if you can advise here.

[cabljac]

Hi @james-wang-careerduck - thanks for raising this, hope i can help.

The extension should grant permissions automatically. Can you provide logs of the errors at all?

If you add a document to the collection specified in params, do you see it get reflected in BigQuery at all?

[james-wang-careerduck]

No, not reflecting in BigQuery at all. The extension installation process never reached completion.

I have working collections and am able to export collections to BigQuery manually.

This morning I saw an update to the extension is available. I proceeded to install the update, then got an error updating

Error Log #1

ERROR 2025-04-22T19:16:55.389903356Z
Validation failed for trigger projects/decisive-fold-436618-h2/locations/us-central1/triggers/ext-firestore-bigquery-export-fsexportbigquery-070677: The request was invalid: generic::invalid_argument: Database '(default)' does not exist in region 'us-central1'. Did you mean region 'nam5'?. See 'gcloud firestore https://cloud.google.com/sdk/gcloud/reference/firestore/databases/list to list all Firestore databases under the project.

Error Log #2

ERROR: failed to create image cache: accessing cache image "us-central1-docker.pkg.dev/decisive-fold-436618-h2/gcf-artifacts/ext--firestore--bigquery--export--setup_big_query_sync/cache:latest": connect to repo store "us-central1-docker.pkg.dev/decisive-fold-436618-h2/gcf-artifacts/ext--firestore--bigquery--export--setup_big_query_sync/cache:latest": GET https://us-central1-docker.pkg.dev/v2/token?scope=repository%3Adecisive-fold-436618-h2%2Fgcf-artifacts%2Fext--firestore--bigquery--export--setup_big_query_sync%2Fcache%3Apull&service=: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/decisive-fold-436618-h2/locations/us-central1/repositories/gcf-artifacts" (or it may not exist)

[cabljac]

Hi, the first error seems like a misconfiguration, have you made sure you're pointing the extension to the correct database region for your firestore? It looks like your firestore instance is on nam5 (multi region US)

ERROR 2025-04-22T19:16:55.389903356Z
Validation failed for trigger projects/decisive-fold-436618-h2/locations/us-central1/triggers/ext-firestore-bigquery-export-fsexportbigquery-070677: The request was invalid: generic::invalid_argument: Database '(default)' does not exist in region 'us-central1'. Did you mean region 'nam5'?. See 'gcloud firestore https://cloud.google.com/sdk/gcloud/reference/firestore/databases/list to list all Firestore databases under the project.