🐛 [firestore-bigquery-export] Custom event handler for onSuccess not called

[myd-mannheim]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

• For general technical questions, post a question on StackOverflow
  with the firebase tag.
• For general Firebase discussion, use the firebase-talk
  google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
  Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: [firestore-bigquery-export]
• Extension version: firebase/firestore-bigquery-export@0.1.55
• Configuration values (redact info where appropriate):
  ALLOWED_EVENT_TYPES=firebase.extensions.firestore-counter.v1.onSuccess
  BIGQUERY_PROJECT_ID=${param:PROJECT_ID}
  COLLECTION_PATH=recipes
  DATASET_ID=recipes_dataset_export_raw
  DATASET_LOCATION=europe-west3
  EVENTARC_CHANNEL=projects/my-xxx/locations/us-central1/channels/firebase
  EXCLUDE_OLD_DATA=no
  LOG_FAILED_EXPORTS=no
  MAX_DISPATCHES_PER_SECOND=100
  TABLE_ID=recipes_table_export_raw
  TABLE_PARTITIONING=NONE
  TIME_PARTITIONING_FIELD_TYPE=omit
  USE_COLLECTION_GROUP_QUERY=no
  USE_NEW_SNAPSHOT_QUERY_SYNTAX=yes
  WILDCARD_IDS=false

[REQUIRED] Step 3: Describe the problem

I've setup firestore-bigquery-export to raise the event firebase.extensions.firestore-counter.v1.onSuccess. Whats quite odd is that in the 'Extension configuration' view of the extension instance the description of this event is 'Occurs when image resizing completes successfully. The event will contain further details about specific formats and sizes.'. Seems to be a copy/paste from the other extension ;)
I implemented and deployed a custom event handler as firebase v2 function and deployed it in the same firebase project (region of the function us-central1). The actual sync of the firebase data into BigQuery works fine. However, my event handler for onSuccess is never executed and I also cannot confirm that the event is even raised (because I can't find any logs about it).
In GC console unter Eventarc I see that the channel and trigger have been created with the correct with the correct destination. Trigger and channel are both in region us-central1.

Steps to reproduce:

Setup a Firebase project with a firestore collection in europe-west3 and install the firestore-bigquery-export extension with mentioned config.
Deploy the following firebase function as event handler.

import eventarc = require('firebase-functions/v2/eventarc')
import logger = require('firebase-functions/logger')

exports.onBigQuerySyncSuccess = eventarc.onCustomEventPublished(
{
eventType: 'firebase.extensions.firestore-counter.v1.onSuccess',
channel: projects/${firebaseFunctionsParams.projectID}/locations/us-central1/channels/firebase,
region: 'us-central1',
},
(event) => {
logger.info('handled firebase.extensions.firestore-counter.v1.onSuccess; received data', event)
return Promise.resolve()
})

Expected result

When changing a document in the collection it is syned into BigQuery and afterward the event handler onBigQuerySyncSuccess is executed.

Actual result

Sync into BigQuery works, event handler is not executed. Somehow I have the impression that the event isn't even raised.

In the GC logs there is a log entry for ext-firestore-bigquery-export-fsexportbigquery with description "Initializing extension with configuration". The "jsonPayload" does not include eventarc channel like in the extensions config data:
jsonPayload: {
bqProjectId: "my-xxx"
clustering: null
collectionPath: "recipes"
databaseId: "(default)"
datasetId: "recipes_dataset_export_raw"
datasetLocation: "europe-west3"
doBackfill: false
docsPerBackfill: 200
excludeOldData: false
initialized: false
instanceId: "firestore-bigquery-export"
location: "us-central1"
logFailedExportData: false
maxDispatchesPerSecond: 100
message: "Initializing extension with configuration"
tableId: "recipes_table_export_raw"
timePartitioning: null
useCollectionGroupQuery: false
useNewSnapshotQuerySyntax: true
wildcardIds: false
}

Thanks,
Max

[myd-mannheim]

Another weird observation is that in Firebase console in the config manage screen of the extension the event are prefixed with "firebase.extensions.firestore-counter.v1"

However, in the source code of the big query export extension the event type have prefix "firebase.extensions.firestore-bigquery-export.v1" (see

extensions/firestore-bigquery-export/functions/src/events.ts

Line 8 in ecc213e

`firebase.extensions.${EXTENSION_NAME}.v1.${eventName}`;

)
In the extension.yaml (https://github.com/firebase/extensions/blob/ecc213e1b384d4a15be629f163d22c431c85d5b2/firestore-bigquery-export/extension.yaml) there is one event listed where the prefix sounds more like it actually belongs to this extension: firebase.extensions.big-query-export.v1.sync.start but it is still different from whats used according to the source code.
It seems the config and whats actually happening is out of sync.

Still, even when I deploy an event handler for firebase.extensions.firestore-bigquery-export.v1.onSuccess it is not called.
However, when I raise the same event on the same channel from my own firebase function then my event handler is called. I raise the event like this:
adminEventarc.getEventarc().channel(
'projects/my-xxx/locations/us-central1/channels/firebase',
{
allowedEventTypes: 'firebase.extensions.firestore-bigquery-export.v1.onSuccess',
}
).publish({
type: 'firebase.extensions.firestore-bigquery-export.v1.onSuccess',
subject: 'test',
data: {
message: 'test-data',
},
})

[myd-mannheim]

Also here

extensions/firestore-bigquery-export/functions/src/events.ts

Line 17 in ecc213e

allowedEventTypes: process.env.EXT_SELECTED_EVENTS,

I have the impression it should not be allowedEventTypes: process.env.EXT_SELECTED_EVENTS but allowedEventTypes: process.env.ALLOWED_EVENT_TYPES because ALLOWED_EVENT_TYPES is the name of the variable in the config env for the extension.

[shubhamkislay]

I'm facing the same issue. Did you get it to work?

[junaidxabd]

Having the same issue here

[cabljac]

Hi all,

Thanks for reporting this issue! We’ve received it and are reviewing it. We’ll provide updates as soon as possible.

[beenerdy]

Same issue here

[beenerdy]

Summary of events.recordSuccessEvent behavior in Firestore BigQuery Export:

• recordSuccessEvent is only called in syncBigQuery when a failed direct write to BigQuery is successfully processed via the task queue.
• For successful direct writes in fsexportbigquery, no success event is published to Eventarc, only logs.complete() is called.
• This means Eventarc listeners on the firebase channel will only see success events (e.g., firebase.extensions.big-query-export.v1.sync.success) for retried operations, not for immediate successful writes.