🐛 [firestore-bigquery-export] Error when mirroring data to BigQuery FirebaseFunctionsError: Queue does not exist

[Fasust]

Extension Setup

• Extension name: firestore-bigquery-export
• Extension version: 0.1.51
• Configuration values (redact info where appropriate):

    COLLECTION_PATH = ***
    TABLE_ID        = ***
    DATABASE                      =  ***
    DATABASE_REGION               = "eur3"
    DATASET_LOCATION              = "europe-west1"
    BIGQUERY_PROJECT_ID           = ***
    DATASET_ID                    =  ***
    MAX_DISPATCHES_PER_SECOND     = 255
    USE_NEW_SNAPSHOT_QUERY_SYNTAX = "yes"
    EXCLUDE_OLD_DATA              = "no"
    WILDCARD_IDS                  = "false"
    LOG_LEVEL                     = "info"
    MAX_ENQUEUE_ATTEMPTS          = 3
    VIEW_TYPE                     = "view"
    TABLE_PARTITIONING           = "DAY"
    TIME_PARTITIONING_FIELD      = "timestamp"
    TIME_PARTITIONING_FIELD_TYPE = "omit"

The Problem

The following error was thrown by the fsexportbigquery cloud function, that was generated by the firestore-bigquery-export extension.

Error when mirroring data to BigQuery FirebaseFunctionsError: Queue does not exist. If you just created the queue, wait at least a minute for the queue to initialize.
    at FunctionsApiClient.toFirebaseError (/workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:305:16)
    at FunctionsApiClient.enqueue (/workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:146:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /workspace/lib/index.js:108:9 {
  errorInfo: {
    code: 'functions/not-found',
    message: 'Queue does not exist. If you just created the queue, wait at least a minute for the queue to initialize.'
  },
  codePrefix: 'functions'
}

This issue only occurred once so far in our setup. Previous and subsequent runs of fsexportbigquery did not throw this error. This issue happened at 2025-07-21T00:55:03.861724322Z. We did not recently make changes to the extension configuration. The extension has been running fine for months before this error was logged.

Steps to reproduce:

Unknown

Expected result

fsexportbigquery runs without throwing exceptions

Actual result

fsexportbigquery runs, and sometimes throws this exception

[CorieW]

Hi @Fasust

Thanks for reporting this issue! We’ve received it and are reviewing it. We’ll provide updates as soon as possible.

[HassanBahati]

Hi @Fasust,

Thanks again for reporting this issue.

We’ve been actively investigating over the past few days and wanted to check in to see if you have any additional logs or context that could help us. In particular, did you happen to update the extension around the time the issue occurred?

During our investigation, we’ve identified a potential race condition in how the extension accesses certain resources. We're currently looking into this further and will keep you updated as we discover more.

Appreciate your patience and help!

[Fasust]

Hi @Fasust,

Thanks again for reporting this issue.

We’ve been actively investigating over the past few days and wanted to check in to see if you have any additional logs or context that could help us. In particular, did you happen to update the extension around the time the issue occurred?

During our investigation, we’ve identified a potential race condition in how the extension accesses certain resources. We're currently looking into this further and will keep you updated as we discover more.

Appreciate your patience and help!

Hey @HassanBahati,

We did not make any changes to the extension version around the time of the issue. The version has not changed since Jul 8, 2024, which was 378 days before the issue occurred. Although I do need to correct the version I reported the issue for. We are actually running the extension on 0.1.51. I originally reported it for 0.2.4, my mistake.
I sadly do not have any additional logs to help the investigation.

[cabljac]

@Fasust Hi! can you confirm, there was one instance of this bug you saw and no other?

Or has it occurred at all since the first occurrence?

The potential race condition we found is unlikely the cause of this issue, as it should only occur during reconfiguration or reinstallation.

If this has only occurred once then my gut feeling is that there may have been some brief downtime with the Tasks API which coincided with other APIs the BigQuery extension uses.

I have several improvements we could make to ensure the extension is more robust, but it will be a matter of obtaining priority for these tasks.

[Fasust]

@cabljac Hey,

Yes, we have only observed this issue once.
More resilience against such outages sounds like a valuable improvement.