🐛 [firestore-export-bigquery] updates to the (default) Firestore Database has stopped since Sep. 16th

[chsyip]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-export-bigquery (storage-resize-images, firestore-send-email, etc)
• Extension version: 0.1.5
• Configuration values (redact info where appropriate):
  • Display Name: Real Time Customer (1-Prod)
  • Dataset ID: real_time_customer_1

[REQUIRED] Step 3: Describe the problem

Description:

This is a continuation of Case 63380535 from Google Support.

Data in Firestore from the Data Transfer in BQ (Real Time Customer (1-Prod)) does not appear correctly - it is still showing test data mid-September.

Backfills have been running to test the data since the 16th, and they are displayed as successful in BigQuery.

However, in Firestore, the latest run is from the 16th on a scheduled job, and not any of the newer backfilled nor scheduled jobs from the same data transfer.

The logs associated with the Data Transfer also look successful and I'm having difficulty deciphering next steps from the logs in our GCP project.

The data may be available in the associated BQ tables in real_time_customer_1 but it is not visible in the Firestore interface in order to verify.

[CorieW]

Hi @chsyip . We've received your issue and will notify you as soon as we have updates.

[cabljac]

Hi @chsyip, thanks for raising this! There are actually two extensions that involve firestore and bigquery, one for each direction.

• Stream Firestore to BigQuery
• Export BigQuery to Firestore

Please correct me if i'm wrong but it sounds like you're having issue with the Export BigQuery to Firestore extension?

That extension actually lives in this repo but no need to move the issue, we can investigate and track this here.

To help us reproduce and resolve the issue - I see you've provided:

• Display Name
• Dataset Id

Could you provide the configuration values for the remaining parameters? Please redact any sensitive data as needed.

• Location
• BigQuery Dataset Location
• Destination Table Name
• Query String
• Schedule
• Pub Sub Topic
• Firestore Collection

(and the optional param, whether or not it is set)

• Partitioning Field

[cabljac]

Additionally, to help us understand the timeline:

• Did this extension ever work successfully for you?
• If yes, when did it start working, and when did you first notice it stopped working?
• Was September 16th the last time you saw data successfully exported to Firestore?

In particular, did you reconfigure or reinstall the extension at all?

You mentioned logs with respect to data transfer looked successful, so just to confirm: you didn't see any logs along the lines of:

"Skipping handling pubsub message because
  transferConfig '...' is not associated with extension instance '...'"

or anything like that?

[chsyip]

Hi @cabljac! Thanks for jumping in.

Yes, the extension is indeed Export BigQuery to Firestore

Location: us-central1
BigQuery Dataset Location: US
Destination Table Name: real_time_customer_1
Query String: [REDACTED]
Schedule: every 12 hours
Pub Sub Topic: real_time_customer_1
Firestore Collection: transferConfigs_1
And there are no partitioning fields

I believe the extension worked with some test data but at the time it was a much shorter schedule to ensure it was working over a joint debugging session with Google Support. Sept 16th would have been when I last re-configured the transfer configs to reflect what I wanted it to do in production.

I don't believe I'd seen any logs like that - I'll go through logs around that time to verify

[cabljac]

Thanks for the info! We'll investigate and reply with updates as soon as we have any

[CorieW]

Hi @chsyip.

We've been unable to reproduce the issue so far.

So we can continue to investigate, do you mind sharing the following:

• Testing transfer configuration details.
• The changes between your testing config and production config.

[chsyip]

This is the ID of the transfer: 68d8cfe5-0000-25a3-b5a5-582429bfdcdc
It selects fields from a staging table, which is populated by a different data transfer.

The differences in testing and production configs were:

• We changed the frequency that the transfer runs (in testing, it was every 5 min to ensure it was working)
• The SQL statement has changed a few times while we were testing the data flow - it started from a SELECT * LIMIT 5 statement on an existing table in the same project, and after un-pausing it is now SELECT everything from a staging table