fix(db-mongodb): improve compatability with Firestore database #12763

elliott-w · 2025-06-11T11:24:30Z

What?

Adds four more arguments to the mongooseAdapter:

  useJoinAggregations?: boolean  /* The big one */
  useAlternativeDropDatabase?: boolean
  useBigIntForNumberIDs?: boolean
  usePipelineInSortLookup?: boolean

Also export a new compatabilityOptions object from @payloadcms/db-mongodb where each key is a mongo-compatible database and the value is the recommended mongooseAdapter settings for compatability.

Why?

When using firestore and visiting /admin/collections/media/payload-folders, we get:

MongoServerError: invalid field(s) in lookup: [let, pipeline], only lookup(from, localField, foreignField, as) is supported

Firestore doesn't support the full MongoDB aggregation API used by Payload which gets used when building aggregations for populating join fields.

There are several other compatability issues with Firestore:

The invalid pipeline property is used in the $lookup aggregation in buildSortParams
Firestore only supports number IDs of type Long, but Mongoose converts custom ID fields of type number to Double
Firestore does not support the dropDatabase command
Firestore does not support the createIndex command (not addressed in this PR)

How?

useJoinAggregations?: boolean  /* The big one */

When this is false we skip the buildJoinAggregation() pipeline and resolve the join fields through multiple queries. This can potentially be used with AWS DocumentDB and Azure Cosmos DB to support join fields, but I have not tested with either of these databases.

useAlternativeDropDatabase?: boolean

When true, monkey-patch (replace) the dropDatabase function so that it calls collection.deleteMany({}) on every collection instead of sending a single dropDatabase command to the database

useBigIntForNumberIDs?: boolean

When true, use mongoose.Schema.Types.BigInt for custom ID fields of type number which converts to a firestore Long behind the scenes

  usePipelineInSortLookup?: boolean

When false, modify the sortAggregation pipeline in buildSortParams() so that we don't use the pipeline property in the $lookup aggregation. Results in slightly worse performance when sorting by relationship properties.

Limitations

This PR does not add support for transactions or creating indexes in firestore

Fixes

Fixed a bug (and added a test) where you weren't able to sort by multiple properties on a relationship field.

Future work

Firestore supports simple $lookup aggregations but other databases might not. Could add a useSortAggregations property which can be used to disable aggregations in sorting.
Currently the resolveJoins() function does sorting client-side but it could use simple aggregations lookup(from, localField, foreignField, as) for sorting instead

…NS === 'true'

…s with polymorphic relationships"

r1tsuu

Really hugee PR! Thank you! Here's some feedback / questions

r1tsuu · 2025-06-13T11:35:10Z

packages/db-mongodb/src/queries/buildSortParam.ts

+      const as = `__${relationshipPath.replace(/\./g, '__')}`
+
+      // If we have not already sorted on this relationship yet, we need to add a lookup stage
+      if (!sortAggregation.some((each) => '$lookup' in each && each.$lookup.as === as)) {
+        let localField = versions ? `version.${relationshipPath}` : relationshipPath
+
+        if (adapter.compatabilityMode === 'firestore') {
+          const flattenedField = `__${localField.replace(/\./g, '__')}_lookup`
+          sortAggregation.push({
+            $addFields: {
+              [flattenedField]: `$${localField}`,
+            },
+          })
+          localField = flattenedField
+        }
+
        sortAggregation.push({
          $lookup: {
-            as: `__${path}`,
+            as,
            foreignField: '_id',
            from: foreignCollection.Model.collection.name,
-            localField: versions ? `version.${relationshipPath}` : relationshipPath,
-            pipeline: [
-              {
-                $project: {
-                  [sortFieldPath]: true,
+            localField,
+            ...(adapter.compatabilityMode !== 'firestore' && {
+              pipeline: [
+                {
+                  $project: {
+                    [sortFieldPath]: true,
+                  },
                },
-              },
-            ],
+              ],
+            }),
          },
        })

-        sort[`__${path}.${sortFieldPath}`] = sortDirection
+        if (adapter.compatabilityMode === 'firestore') {
+          sortAggregation.push({
+            $unset: localField,
+          })
+        }
+      }

-        return true
+      if (adapter.compatabilityMode !== 'firestore') {
+        const lookup = sortAggregation.find(
+          (each) => '$lookup' in each && each.$lookup.as === as,
+        ) as PipelineStage.Lookup
+        const pipeline = lookup.$lookup.pipeline![0] as PipelineStage.Project
+        pipeline.$project[sortFieldPath] = true
      }
+
+      sort[`${as}.${sortFieldPath}`] = sortDirection
+      return true


You need only 1 condition here with firestore, you should just ignore relationshipsSort:

payload/packages/db-mongodb/src/queries/buildSortParam.ts

Lines 180 to 194 in 53835f2

if (

sortAggregation &&

relationshipSort({

adapter,

fields,

locale,

path: sortProperty,

sort: acc,

sortAggregation,

sortDirection,

versions,

})

) {

return acc

}

since aggregations are not supported there.

@r1tsuu Actually firestore does support aggregations, just not the full aggregation API. So when sorting on relationship properties, even though we can't project on relationship fields I think it's more efficient to $lookup the entire relationship document and use the $sort aggregation in mongodb, rather than doing the sorting client-side.

For the test 'should sort by multiple properties of a relationship', this is what the native aggregation looks like:

[ { "$match": {} }, { "$lookup": { "as": "__director", "foreignField": "_id", "from": "directors", "localField": "director", "pipeline": [ { "$project": { "name": true, "createdAt": true } } ] } }, { "$sort": { "__director.name": 1, "__director.createdAt": 1, "createdAt": -1 } }, { "$skip": 0 }, { "$limit": 10 } ]

And for firestore:

[ { "$match": {} }, { "$addFields": { "__director_lookup": "$director" } }, { "$lookup": { "as": "__director", "foreignField": "_id", "from": "directors", "localField": "__director_lookup" } }, { "$unset": "__director_lookup" }, { "$sort": { "__director.name": 1, "__director.createdAt": 1, "createdAt": -1 } }, { "$skip": 0 }, { "$limit": 10 } ]

Could potentially unset the relationship field (e.g. __director) after sorting to reduce the amount of unecessary data sent back to the client. What do you think?

r1tsuu · 2025-06-13T11:36:27Z

test/generateDatabaseAdapter.ts

+  firestore: `
+  import { mongooseAdapter } from '@payloadcms/db-mongodb'
+
+  if (!process.env.DATABASE_URI) {
+    throw new Error('DATABASE_URI must be set when using firestore')
+  }
+
+  export const databaseAdapter = mongooseAdapter({
+    ensureIndexes: false,
+    disableIndexHints: true,
+    useJoinAggregations: false,
+    url: process.env.DATABASE_URI,
+    collation: {
+      strength: 1,
+    },
+    compatabilityMode: 'firestore'
+  })`,


Add this to the workflow so the tests with it will run on CI

payload/.github/workflows/main.yml

Lines 166 to 176 in 53835f2

strategy:

fail-fast: false

matrix:

database:

- mongodb

- postgres

- postgres-custom-schema

- postgres-uuid

- supabase

- sqlite

- sqlite-uuid

@r1tsuu Would you guys setup a firestore database for testing? Or do you want me to just set it up so it tests against in-memory mongodb with the feature flags specifically for firestore?

Either way it will fail some tests (mostly those related to indexes and transactions) so would need to add some way to skip those tests for firestore.

r1tsuu · 2025-06-13T11:41:46Z

packages/db-mongodb/src/index.ts

@@ -110,6 +110,8 @@ export interface Args {
  collation?: Omit<CollationOptions, 'locale'>

  collectionsSchemaOptions?: Partial<Record<CollectionSlug, SchemaOptions>>
+  /** Solves some common issues related to the specified database. Full compatability is not guaranteed. */
+  compatabilityMode?: 'firestore'


I agree with @DanRibbens here on compatiblityMode. This property shouldn't be used for the adapter's logic, but only to set some default properties, like useJoinAggregations: false. Another approach, instead of having this "useless" property would be something like:

import { mongooseAdapter, firestoreCompatibillity } from '@payloadcms/db-mognodb' mongooseAdapter({ ...firestoreCompatibillity, url: process.env.DATABASE_URI })

what do you think?

I will implement this!

r1tsuu · 2025-06-13T11:43:57Z

packages/db-mongodb/src/models/buildSchema.ts

+        _id:
+          idField.type === 'number'
+            ? payload.db.compatabilityMode === 'firestore'
+              ? mongoose.Schema.Types.BigInt
+              : Number


should be a flag, like useBigIntForNumberIDs: true

r1tsuu · 2025-06-13T11:44:10Z

packages/db-mongodb/src/models/buildSchema.ts

+      if (payload.db.compatabilityMode === 'firestore') {
+        return mongoose.Schema.Types.BigInt
+      } else {
+        return mongoose.Schema.Types.Number
+      }


here as well

r1tsuu · 2025-06-13T11:49:41Z

test/relationships/int.spec.ts

+        it('should sort by multiple properties of a relationship', async () => {
+          await payload.delete({ collection: 'directors', where: {} })
+          await payload.delete({ collection: 'movies', where: {} })
+
+          const createDirector = {
+            collection: 'directors',
+            data: {
+              name: 'Dan',
+            },
+          } as const
+
+          const director_1 = await payload.create(createDirector)


I suppose this and your changes to the logic in buildSortParam are not related to firestore compatibility? I wonder if it'd be better to open a separate, fix PR for this, which should be easier for review and have more high priority for merge. But if we can merge this one quickly it's fine too.

@r1tsuu buildSortParam changes are required for compatability with firestore, see comment above

r1tsuu · 2025-06-13T11:56:12Z

packages/db-mongodb/src/utilities/resolveJoins.ts

+  // Add polymorphic joins
+  for (const join of collectionConfig.polymorphicJoins || []) {
+    // For polymorphic joins, we use the collections array as the target
+    joinMap[join.joinPath] = { ...join, targetCollection: join.field.collection as string }
+  }


I have a question here. One of the nice things about polymorphic joins is that like with SQL' UNION ALL, it can sort and so paginate documents from multiple collections at once, so your result isn't always just like:
[...Documents from collection 1, ...Documents from collection 2], rather you can have:
[Document from collection 1, Document from collection 2, Document from collection 1]

I suppose without aggregations this is not possible, right?

Yeah you could do this with the $unionWith stage operator, but firestore doesn't support that either. My intuition is that most mongo-compatible nosql databases are unlikely to support the stage operators ($unionWith or pipeline in $lookup) required for polymorphic joins.

r1tsuu · 2025-06-13T12:01:59Z

packages/db-mongodb/src/utilities/resolveJoins.ts

+      // Sort the grouped results
+      const sortParam = joinQuery.sort || joinDef.field.defaultSort
+      for (const parentKey in grouped) {
+        if (sortParam) {
+          // Parse the sort parameter
+          let sortField: string
+          let sortDirection: 'asc' | 'desc' = 'asc'
+
+          if (typeof sortParam === 'string') {
+            if (sortParam.startsWith('-')) {
+              sortField = sortParam.substring(1)
+              sortDirection = 'desc'
+            } else {
+              sortField = sortParam
+            }
+          } else {
+            // For non-string sort params, fall back to default
+            sortField = 'createdAt'
+            sortDirection = 'desc'
+          }
+
+          grouped[parentKey]!.sort((a, b) => {


OK, this is the answer to my question above. You did this at the app level, what about performance since you need to retrieve all the documents that match the query from the DB for each collection, and there's no any projection set either?

r1tsuu · 2025-06-13T12:06:45Z

packages/db-mongodb/src/utilities/resolveJoins.ts

+    // Execute the query to get all related documents
+    const results = await JoinModel.find(whereQuery, null).sort(mongooseSort).lean()


I think this can use projection, at least for non polymorphic you surely need only _id, otherwise you need fields that are specified in sort I suppose.

Took me a while to figure out why you'd need a projection here, because I couldn't find anyway to project on the join documents in the Local API.

It seems that the initial find only queries join document ids (and sort fields) then there's a secondary set of finds (one for each collection in the join) that gets executed in the afterRead hook to populate those documents with the full data.

Is that correct? If so, why not just use a single query with aggregations? Is it because it's easier to support other databases that way?

I will add a projection for the initial find.

The test was incorrectly expecting only 1 document when it should expect 2. The documentsAndFolders join correctly returns documents from both folderPoly1 and folderPoly2 collections when querying with a relationTo filter that includes both collection types. Also cleaned up the test setup to remove redundant WHERE filters that were causing the test to only return one collection type. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Add commonTitle field to FolderPoly1 and FolderPoly2 collections to support the polymorphic join test case requirements. This field provides a shared property across both collection types for testing purposes. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

This reverts commit 853e0c6.

elliott-w · 2025-06-16T13:08:00Z

@DanRibbens

I removed the compatabilityMode arg and now export a compatabilityOptions object instead.

@r1tsuu

I've simplified the resolveJoins code quite a bit and added a projection.

I've added firestore to the testing matrix, but it doesn't use all of compatabilityOptions.firestore because some of them cause tests to fail (related to transactions and indexes). I don't think firestore provides github actions for their database since it's quite new, but you can always create a firestore database for testing purposes and just reference it with DATABASE_URI.

Also updated the original comment to represent the current state of this PR.

…oins

elliott-w changed the title ~~fix(db-mongodb): Improve compatability with Firestore database~~ fix(db-mongodb): improve compatability with Firestore database Jun 11, 2025

elliott-w force-pushed the firestore branch from 9d2c331 to b67270c Compare June 12, 2025 00:32

elliott-w added 28 commits June 12, 2025 10:42

Add manual join option to Mongo adapter

c182997

Disable memory server if DATABASE_URI set

42d4ca5

Prevent createIndex and dropDatabase operations if PAYLOAD_MANUAL_JOI…

620d053

…NS === 'true'

Add support for PAYLOAD_DATABASE to be 'firestore'

7b9b256

Fix dropDatabase monkey patch

824ab15

Fix customIDType 'number'

a78ba19

More work

eebaa46

Fix buildSortParam

5f7a8c7

Fix buildSortParam

6004bf7

Delete manual joins folder

75ec812

Polymorphic joins

e2230e0

better polymorphic joins

2d09272

More progress

ef3fe35

Fix queryDrafts

320050d

remove __resolveJoins flag

e533186

Revert resolveJoins

67113e4

Fix returning 0 documents rather than all when limit is 0

e8ec31d

Fix queryDrafts

6db79b7

Use similar versions/drafts detection to buildJoinAggregation

912af71

Fix test "should join across multiple collections"

99f3586

Sinplify polymorphic joins

dae7a80

Fix test "should not throw a path validation error when querying join…

c40062c

…s with polymorphic relationships"

Fix type and lint issues

b9b0542

Fix failing test "should rollback operations on failure"

d348921

Remove transactions

09a2996

Optimize resolveJoins performance

dad811c

Update mongodb documentation

a5e9178

Fix mongoose adapter args

717b311

r1tsuu requested changes Jun 13, 2025

View reviewed changes

elliott-w and others added 24 commits June 14, 2025 13:31

Move utility functions to bottom of file

485a51c

Improve resolveJoins type comments

b23c2b2

Fix transform not being applied for version documents

acdb24b

Step

75546af

Fix typescript and lint errors

dfcedfe

Fix test 'should join across multiple collections'

d8be930

Add ability to sort by multiple properties

34d4d20

Add projections

3437f4b

Extract buildJoinProjection

7aa4476

Simplify by treating all joins as polymorphic

d9c87ee

Revert joins int.spec.ts

f8e8322

Fix bugs

a10a552

Apply limit and page client-side

0253b27

Fix joinPath on arrays

0dae7f2

Fix polymorphic relationship fields

0662b95

Fix versions

30334c5

Add compatabilityOptions utility and remove compatabilityMode arg

7420449

Revert "Add commonTitle field to polymorphic test collections"

d49213a

This reverts commit 853e0c6.

Fix lint errors resolveJoins

a8482fd

Improve generateDatabaseAdapter firestore settings

833c504

Add firestore to database testing matrix

f092e8d

Tidy up comments

235af62

elliott-w requested review from DanRibbens and r1tsuu June 16, 2025 13:02

elliott-w added 2 commits June 18, 2025 09:37

Update documentation

5d14cdc

Apply sort and pagination at the database level for non-polymorphic j…

90ee5a7

…oins

	if (
	sortAggregation &&
	relationshipSort({
	adapter,
	fields,
	locale,
	path: sortProperty,
	sort: acc,
	sortAggregation,
	sortDirection,
	versions,
	})
	) {
	return acc
	}

	strategy:
	fail-fast: false
	matrix:
	database:
	- mongodb
	- postgres
	- postgres-custom-schema
	- postgres-uuid
	- supabase
	- sqlite
	- sqlite-uuid

		// Execute the query to get all related documents
		const results = await JoinModel.find(whereQuery, null).sort(mongooseSort).lean()

fix(db-mongodb): improve compatability with Firestore database #12763

Are you sure you want to change the base?

fix(db-mongodb): improve compatability with Firestore database #12763

Conversation

elliott-w commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Limitations

Fixes

Future work

Uh oh!

r1tsuu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elliott-w Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elliott-w commented Jun 16, 2025

Uh oh!

Uh oh!

elliott-w commented Jun 11, 2025 •

edited

Loading

elliott-w Jun 16, 2025 •

edited

Loading