feat: query with empty filter

[zulkhair]

Add Pagination and Query All Entities Support to Cardinal ECS

Summary

This PR adds pagination support (limit and offset) to the Cardinal ECS query system and enables querying all entities without specifying component filters. The changes include refactoring the search implementation for better maintainability and adding comprehensive test coverage.

Features

1. Pagination Support

• Limit: Maximum number of results to return (default: 50, max: 10,000)
• Offset: Number of results to skip before returning (default: 0)
• Early termination when limit is reached for better performance
• Consistent pagination within query sessions

2. Query All Entities

• Empty Find list now queries all entities regardless of components
• Match parameter is ignored when Find is empty (backward compatible)
• Enables use cases like "get all entities" or "count total entities"

3. Code Refactoring

• Extracted helper functions for better code organization:
  • buildEntityResult(): Creates result map from entity and components
  • matchesFilter(): Checks if entity matches filter expression
  • normalizeLimit(): Applies default and enforces maximum limit
  • processEntity(): Processes single entity with filter, offset, and limit logic

Changes

ECS Layer (pkg/cardinal/ecs/search.go)

• Added Limit and Offset fields to SearchParam struct
• Added constants: DefaultQueryLimit (50), MaxQueryLimit (10000), MinQueryLimit (0), MinQueryOffset (0)
• Modified validateAndGetFilter() to allow empty Find list
• Modified findMatchingArchetypes() to return all archetype IDs when Find is empty
• Refactored NewSearch() to implement pagination with early termination
• Added helper functions for better code organization

Service Layer (pkg/cardinal/service/query.go)

• Added Limit and Offset fields to Query struct
• Updated parseQuery() to parse limit and offset from protobuf payload
• Removed duplicate default limit handling (now handled in ECS layer)

Protobuf Schema (proto/worldengine/isc/v1/query.proto)

• Removed min_len: 1 validation from find field to allow empty arrays
• Removed not_in: [0] validation from match field to allow MATCH_UNSPECIFIED when find is empty
• Added limit field (int32, field 4) with validation: gte: 0, lte: 10000
• Added offset field (int32, field 5) with validation: gte: 0

Testing

• Updated existing tests to reflect new behavior (empty Find no longer errors)
• Added comprehensive test coverage for:
  • Empty Find queries (querying all entities)
  • Pagination (Limit and Offset)
  • Edge cases (negative values, limit beyond total results)
  • Pagination combined with Where filters
  • Pagination across multiple archetypes

API Changes

Request Format

{
  "address": {
    "realm": "REALM_WORLD",
    "region": "us-west-2",
    "organization": "organization",
    "project": "query-demo",
    "serviceId": "game"
  },
  "query": {
    "find": ["charactertag", "health"],  // Empty array = all entities
    "match": "MATCH_CONTAINS",           // Ignored when find is empty
    "where": "health.hp > 50",            // Optional filter
    "limit": 10,                          // New: max results (default: 50)
    "offset": 0                           // New: skip N results (default: 0)
  }
}

Response Format

Unchanged - returns QueryResult with entities array.

Backward Compatibility

✅ Fully backward compatible

• Empty Find list: Previously returned error, now returns all entities
• Missing limit/offset: Defaults applied automatically (limit: 50, offset: 0)
• Existing queries without pagination continue to work

Performance Considerations

• Early termination: Query stops as soon as limit is reached
• Default limit: Prevents unbounded queries (default: 50, max: 10,000)
• Memory efficient: Results slice pre-allocated with capacity = limit

Known Issues / Notes

⚠️ Gateway Compatibility: The gateway service needs to be updated to properly parse the limit and offset fields from the JSON request. Currently, these fields may not be extracted from the nested query object during JSON-to-protobuf conversion. This is a gateway issue, not a Cardinal issue.

Workaround: Once the gateway is updated to parse these fields correctly, pagination will work as expected.

Testing

All existing tests pass. New tests added:

• TestSearch_EmptyFind - Query all entities
• TestSearch_Pagination - Comprehensive pagination tests
  • Default limit (50)
  • Specified limit
  • Offset skipping
  • Offset beyond total results
  • Pagination with Where filter
  • Pagination across multiple archetypes

Example Usage

Query All Entities

{
  "query": {
    "find": [],
    "limit": 10
  }
}

Pagination - Page 1

{
  "query": {
    "find": ["charactertag", "health"],
    "match": "MATCH_CONTAINS",
    "limit": 5,
    "offset": 0
  }
}

Pagination - Page 2

{
  "query": {
    "find": ["charactertag", "health"],
    "match": "MATCH_CONTAINS",
    "limit": 5,
    "offset": 5
  }
}

[zulkhair]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: query with empty filter #863 👈 (View in Graphite)
• refactor: cardinal #865 : 2 other dependent PRs (#867 , #868 )
• main

---

How to use the Graphite Merge Queue

Add the label graphite/merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[rmrt1n]

I'd rather we create a new, explicit match type, e.g. SearchAll instead doing it implicitly if find is empty. In this case, find can be empty only when match type is all.

[zulkhair]

That's a good idea, updated the code

[rmrt1n]

We have a hard limit of 2^32 entities, so limit should also be type uint32. Limit and offset can never be negative, we should take advantage of the type system so we don't have to write extra code like check int boundaries, e.g. the validations and normalizeLimit can be removed.

[zulkhair]

what a great feedback, updated the code

[rmrt1n]

The fact that this function need all these pointers as args very often means it's not meant to be factored out. I'd put this back into NewSearch so you can see the whole filtering logic in one place.

[zulkhair]

moved to NewSearch for readability

[rmrt1n]

You can omit the size and limit, make for maps doesn't accept the capacity field. Limit is huge if not specified, so we'll just use Go's map default size. A downside is this can result in more allocations than if we specified the size, but we'll worry about this later if somehow the allocations affects the tickrate.

[zulkhair]

updated the code

[rmrt1n]

I'd prefer to make the default limit unlimited (max u32), which seems more intuitive as having a non-unlimited default limit means we have more work to educate the users. We can worry about memory usage much later.

[zulkhair]

updated the code

[rmrt1n]

nit: Move the public APIs to the top of the file since that's the natural reading order. Helper functions go after since they're not too important.

[zulkhair]

updated the code

[codecov]

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.