Add document transformation capability for indexing #123
Conversation
|
|
||
| const url = `https://${region}-${projectId}.cloudfunctions.net/${transformFunctionName}`; | ||
|
|
||
| const response = await fetch(url, { |
There was a problem hiding this comment.
@tharropoulos Can we add automatic retries here with exponential backoff?
There was a problem hiding this comment.
@tharropoulos Looks like this is not addressed yet
extension.yaml
Outdated
| example: transformDoc | ||
| default: "" | ||
| required: false | ||
| - param: TRANSFORM_FUNCTION_PROJECT_ID |
There was a problem hiding this comment.
@tharropoulos Given the number of new parameters this will add to the UI, what do you think about adding a constraint so that the function has to be in the same project and region as the one the extension is installed in?
I know it will limit some functionality, but it will also help keep the config simple.
Unless, is there a way to move this under an advanced collapsible section in the installation UI?
- add `transformDocument` function to call external transformation functions - implement error handling for failed transformations with fallback to original doc - add comprehensive tests for the transformation functionality - update package.json with new test command
- update indexOnWrite to conditionally use transformation function - check for transformFunctionName in config before transforming - apply transformation before converting to typesense document
- add TRANSFORM_FUNCTION_NAME parameter for specifying transform function - add TRANSFORM_FUNCTION_PROJECT_ID for cross-project function support - add TRANSFORM_FUNCTION_REGION for region specification - add TRANSFORM_FUNCTION_SECRET for authorization to protected functions
- Remove `TRANSFORM_FUNCTION_PROJECT_ID` and `TRANSFORM_FUNCTION_REGION` parameters from `extension.yaml` - Update `config.js` to use `GCLOUD_PROJECT` and `LOCATION` environment
tharropoulos
force-pushed
the
transform-func
branch
from
4579313 to
f5c6b8e
Compare
|
|
||
| const url = `https://${region}-${projectId}.cloudfunctions.net/${transformFunctionName}`; | ||
|
|
||
| const response = await fetch(url, { |
There was a problem hiding this comment.
@tharropoulos Looks like this is not addressed yet
| transformFunctionName: process.env.TRANSFORM_FUNCTION_NAME, | ||
| transformFunctionSecret: process.env.TRANSFORM_FUNCTION_SECRET, | ||
| transformFunctionProjectId: process.env.GCLOUD_PROJECT, | ||
| transformFunctionRegion: process.env.LOCATION, |
There was a problem hiding this comment.
Are the above two lines still needed?
There was a problem hiding this comment.
We have to call out to eh region-project id. Both of those are coming from env variables. The gcloud project is automatic and the location is being set from extension.yaml.
TLDR
New options to transform Firestore documents before indexing to Typesense.
Change Summary
Added Configuration:
extension.yaml:TRANSFORM_FUNCTION_NAME: Name of Cloud Function for document transformationTRANSFORM_FUNCTION_PROJECT_ID: Project where transform function is deployedTRANSFORM_FUNCTION_REGION: Region of transform functionTRANSFORM_FUNCTION_SECRET: Auth secret for transform functionAdded Functionality:
In
utils.js:transformDocument(): Calls external transform function with error handlingIn
indexOnWrite.js:Added Tests:
New file
test/utilsTransform.spec.js:In
package.json:test:utilsPR Checklist