Setup `CollectionSearchExtension` and `FreeTextAdvancedExtension` #28

fmigneault · 2025-05-01T03:26:57Z

Based on #27

These extensions can be imported with updated python dependencies, but the underlying SQL collection_search function requires a more recent pgstac docker image. However, that upgrade (pgstac 0.6 to 7.x, 8.x, 9.x) requires many PG 13→15→17 migrations, so it will be done in a follow-up version to keep this fix minimal.

i.e.: must do a few iterative call of pypgstac migrate --debug from a ghcr.io/stac-utils/pgstac:v0.7.0, ghcr.io/stac-utils/pgstac:v0.8.0, ghcr.io/stac-utils/pgstac:v0.9.0 and maybe others in between... in https://github.com/bird-house/birdhouse-deploy, probably setup something that detects the current schema, and desired minor version, and do them in a loop. Otherwise, just provide manual operations to be applied.

To avoid the above complicated procedure evaluated in bird-house/birdhouse-deploy#534, instead we will wait for bird-house/birdhouse-deploy#532 to simply export the data, and then force upgrade the STAC app from scratch using latest https://github.com/stac-utils/pgstac/pkgs/container/pgstac and STAC-FastAPI. After that, this PR can be safely included since the STAC-FastAPI will be able to include the relevant collection_search SQL function.

mishaschwartz

Can you also please add an entry in CHANGES.md and run the pre-commit checks?

The pre-commit checks should run whenever a commit is pushed but maybe that got disabled at some point? Can you please check if the pre-commit.ci plugin is enabled for this repo? (I don't have permission to view that).

For now can you either run the pre-commit hooks locally or run:

ruff check
ruff format

src/stac_app.py

fmigneault · 2025-06-09T14:31:27Z

@mishaschwartz

The pre-commit checks should run [...]

It wasn't enabled. I've activated it.

mishaschwartz · 2025-06-09T14:57:51Z

It wasn't enabled. I've activated it.

Thanks!

fmigneault · 2025-07-07T19:53:12Z

Got something working, but FreeTextAdvancedExtension is blocked from invalid parsing.
Waiting on feedback from stac-utils/stac-fastapi#849

for more information, see https://pre-commit.ci

…to more-stac-extensions

fmigneault · 2025-07-17T01:51:25Z

I'm able to make FreeTextAdvancedExtension work. Using following patches:

add missing q param for item search and collection-items search stac-utils/stac-fastapi-pgstac#267

Pending fix for some edge case:

Free-Text Search fails if include/exclude are used by themselves stac-utils/pgstac#376

…stac#269)

for more information, see https://pre-commit.ci

…to more-stac-extensions

fmigneault · 2025-08-15T02:01:03Z

Evaluated search variants

Search items across collections

> curl -X GET 'http://localhost:8000/stac/search?q=SeaLake%20OR%20River' | jq '.features[]|.id,.collection' | paste - -
"EuroSAT-full-train-sample-14830-class-SeaLake"	"EuroSAT-full-train"
"EuroSAT-full-train-sample-13483-class-River"	"EuroSAT-full-train"

> curl -X POST http://localhost:8000/stac/search -d '{"q": ["SeaLake", "River"]}' -H 'Content-Type: application/json' | jq '.features[]|.id,.collection' | paste - -
"EuroSAT-full-train-sample-14830-class-SeaLake"	"EuroSAT-full-train"
"EuroSAT-full-train-sample-13483-class-River"	"EuroSAT-full-train"

> curl -X POST http://localhost:8000/stac/search -d '{"q": "SeaLake OR River"}' -H 'Content-Type: application/json' | jq '.features[]|.id,.collection' | paste - -
"EuroSAT-full-train-sample-14830-class-SeaLake"	"EuroSAT-full-train"
"EuroSAT-full-train-sample-13483-class-River"	"EuroSAT-full-train"

Search items in collection

> curl -X GET 'http://localhost:8000/stac/collections/EuroSAT-full-train/items?q=SeaLake%20OR%20River' | jq '.features[]|.id,.collection' | paste - -
"EuroSAT-full-train-sample-14830-class-SeaLake"	"EuroSAT-full-train"
"EuroSAT-full-train-sample-13483-class-River"	"EuroSAT-full-train"

Search collections metadata

> curl -X GET 'http://localhost:8000/stac/collections?q=SeaLake%20OR%20River'  | jq '.collections[].id'
**no match - OK**

> curl -X GET 'http://localhost:8000/stac/collections?q=train'  | jq '.collections[].id'
"EuroSAT-full-train"

> curl -X GET 'http://localhost:8000/stac/collections?q=train,subset'  | jq '.collections[].id'
"EuroSAT-full-train"

> curl -X GET 'http://localhost:8000/stac/collections?q=train%20OR%20subset'  | jq '.collections[].id'
"EuroSAT-full-train"

> curl -X GET 'http://localhost:8000/stac/collections?q=train%20AND%20subset'  | jq '.collections[].id'
**no match - OK**

for more information, see https://pre-commit.ci

mishaschwartz

Nice! This looks great and I'm glad it's working. Thanks for adding the usage examples in the comment as well.

Just one question below about commented code but otherwise it looks good.

mishaschwartz · 2025-08-15T13:41:15Z

src/stac_app.py

+        CollectionSearchFilterExtension(client=FiltersClient()),
+    ],
+)
+# collection_search_extension = CollectionSearchPostExtension.from_extensions(  # GET + POST


is this left on purpose or should it be removed?

Yes, it is left on purpose to highlight the conflicting endpoints, since the extension must be only partially enabled.
(✅ POST /search, ✅ GET /search, ❌ POST /collections, ✅ GET /collections?q=...)

## Overview Update all STAC components and extra functionalities related to them. ## Changes **Non-breaking changes** >[!WARNING] > Assumes that STAC API `5.2.0-crim-1.1.0` (default) is already applied. If migrating from older versions, all database migration issues when moving from 3.x still apply. - STAC: Update STAC Browser to [`crim-ca/stac-browser:3.4.0-dev`](https://github.com/crim-ca/stac-browser/releases/tag/3.4.0-dev). - Dockers are now built directly with the GitHub CI releases (see https://github.com/crim-ca/stac-browser/pkgs/container/stac-browser). - Synchronize with latest changes (as of 2025-08-16). - Beside the necessary `prefixPath` override for Nginx Proxy redirect and a minor HTML file resolution fix, the image is entirely up-to-date with the official upstream code. - Supports STAC 1.1.0. - Supports language locales. - Greatly improves parsing of STAC metadata and their visual rendering. - Allows dynamic runtime [config.js](https://github.com/radiantearth/stac-browser/blob/main/config.js) overrides. For the time being, only the required `catalogUrl` is overridden, but further settings could be added later on. - STAC: Update STAC API to `crim-ca/stac-app:2.0.1`. - Changes in [`crim-ca/stac-app:2.0.0`](https://github.com/crim-ca/stac-app/releases/tag/2.0.0) includes: - migration to `stac-fastapi==6.0.0` and corresponding fixes to support it - add `q` parameter free-text search on `/search`, `/collections` and `/collections/{collectionId}/items` endpoints - enforce JSON schema validation of all `stac_extensions` referenced by published STAC Items and Collections - multiple dependency updates - Minor package dependency fix in [`crim-ca/stac-app:2.0.1`](https://github.com/crim-ca/stac-app/releases/tag/2.0.1). - STAC: Update `stac-db` and `stac-migration` to version `0.9.8`. - STAC: Add `optional/stac-db-persist` and `STAC_DB_PERSIST_DIR` to allow custom STAC DB metadata storage location. **Breaking changes** >[!WARNING] > If a STAC Item or Collection is POST'd to the STAC API and that its JSON definition does not respect all of its `stac_extensions`'s schemas, the operation will now be refused. While this is potentially "breaking", it is the right expectation, as invalid contents would otherwise be allowed and inserted into the DB. - n/a ## Related Issue / Discussion - #561 - crim-ca/stac-app#37 - crim-ca/stac-app#28 - crim-ca/stac-app#40 - crim-ca/stac-browser#6 - fixes #562 - fixes #498, fixes #346 - technically, still a fork, but it's a 1-liner diff until radiantearth/stac-browser#653 is merged - diff: radiantearth/stac-browser@main...crim-ca:stac-browser:master - closes crim-ca/stac-populator#100 ## CI Operations  birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

prepare dependencies for CollectionSearchExtension and FreeTextExtension

177a48a

fmigneault self-assigned this May 1, 2025

fmigneault added 2 commits April 30, 2025 23:31

update enabled experimental extensions

9a2ffb9

Merge remote-tracking branch 'origin/main' into more-stac-extensions

6616793

fmigneault changed the base branch from fix-token-paging to main June 5, 2025 23:33

fmigneault marked this pull request as ready for review June 5, 2025 23:40

fmigneault requested a review from mishaschwartz June 5, 2025 23:40

mishaschwartz requested changes Jun 9, 2025

View reviewed changes

src/stac_app.py Outdated Show resolved Hide resolved

src/stac_app.py Outdated Show resolved Hide resolved

fmigneault added 3 commits June 9, 2025 10:59

add changes

55e72f8

fix lint

1cab6ea

resolve extension conflicts

1a155bb

fmigneault and others added 7 commits July 7, 2025 15:58

remove unused imports

7cf889f

Merge remote-tracking branch 'origin/main' into more-stac-extensions

0973afb

[pre-commit.ci] auto fixes from pre-commit.com hooks

19b6941

for more information, see https://pre-commit.ci

Update CHANGES.md

fa3a777

Update CHANGES.md

7328534

fix lint

1887752

Merge branch 'more-stac-extensions' of github.com:crim-ca/stac-app in…

38fd3f4

…to more-stac-extensions

fmigneault mentioned this pull request Jul 10, 2025

Endless loop in export crim-ca/stac-populator#100

Closed

Merge branch 'stac-fastapi-6' into more-stac-extensions

f89b277

fmigneault and others added 4 commits July 16, 2025 23:15

use stac-extensions validation (depends on stac-utils/stac-fastapi-pg…

21aee0e

…stac#269)

[pre-commit.ci] auto fixes from pre-commit.com hooks

5ce01b7

for more information, see https://pre-commit.ci

fix free text search extensions (basic and advanced simultaneously)

586ee85

Merge branch 'more-stac-extensions' of github.com:crim-ca/stac-app in…

a58158f

…to more-stac-extensions

update changelog

4c4dbc5

fmigneault and others added 2 commits August 14, 2025 22:14

Merge branch 'main' into more-stac-extensions

6da300b

[pre-commit.ci] auto fixes from pre-commit.com hooks

d0ef01d

for more information, see https://pre-commit.ci

fmigneault requested a review from mishaschwartz August 15, 2025 02:15

fmigneault and others added 3 commits August 14, 2025 22:18

add changelog entry

fb9d835

fix pre-commit linting

f97347e

[pre-commit.ci] auto fixes from pre-commit.com hooks

fa15226

for more information, see https://pre-commit.ci

mishaschwartz approved these changes Aug 15, 2025

View reviewed changes

fmigneault merged commit fe077b3 into main Aug 15, 2025
1 check passed

fmigneault mentioned this pull request Aug 16, 2025

Update all STAC components + extra functionalities bird-house/birdhouse-deploy#575

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Setup `CollectionSearchExtension` and `FreeTextAdvancedExtension` #28

Setup `CollectionSearchExtension` and `FreeTextAdvancedExtension` #28

Uh oh!

fmigneault commented May 1, 2025 •

edited

Loading

Uh oh!

mishaschwartz left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

fmigneault commented Jun 9, 2025

Uh oh!

mishaschwartz commented Jun 9, 2025 •

edited

Loading

Uh oh!

fmigneault commented Jul 7, 2025

Uh oh!

fmigneault commented Jul 17, 2025

Uh oh!

fmigneault commented Aug 15, 2025

Uh oh!

mishaschwartz left a comment

Uh oh!

mishaschwartz Aug 15, 2025

Uh oh!

fmigneault Aug 15, 2025

Uh oh!

Uh oh!

Uh oh!

Setup CollectionSearchExtension and FreeTextAdvancedExtension #28

Setup CollectionSearchExtension and FreeTextAdvancedExtension #28

Uh oh!

Conversation

fmigneault commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mishaschwartz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fmigneault commented Jun 9, 2025

Uh oh!

mishaschwartz commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fmigneault commented Jul 7, 2025

Uh oh!

fmigneault commented Jul 17, 2025

Uh oh!

fmigneault commented Aug 15, 2025

Evaluated search variants

Search items across collections

Search items in collection

Search collections metadata

Uh oh!

mishaschwartz left a comment

Choose a reason for hiding this comment

Uh oh!

mishaschwartz Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

fmigneault Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Setup `CollectionSearchExtension` and `FreeTextAdvancedExtension` #28

Setup `CollectionSearchExtension` and `FreeTextAdvancedExtension` #28

fmigneault commented May 1, 2025 •

edited

Loading

mishaschwartz left a comment •

edited

Loading

mishaschwartz commented Jun 9, 2025 •

edited

Loading