feat(map): Resolve STAC inconsistencies across the codebase. BM-1578 by Wentao-Kuang · Pull Request #87 · linz/topographic-system

Wentao-Kuang · 2026-03-18T01:43:17Z

Motivation

Our current qgis project deployment structure is not correct to put the tag at the top level

./githash/nztopo50map/nz-topo50-map.qgs

We should revert the category as followed, so that we could include more map series in future

qgis:
  - catalog.json
  - nztopo50map:
    - catalog.json

    - latest:
        - collection.json
        - project.qgs # links to release=v0.2.3

    - next:
        - collection.json
        - project.qgs # links to commit=4aba34b5accb0002867af66f6a92a35e0a4be7ca

    - release=v0.0.1:
        - collection.json
        - project.qgs 

    - release=v0.2.3:
        - collection.json
        - project.qgs 

    - pull_request=1: # Mutable
        collection.json
        project.qgs # links to commit=36125a7fbbae012f2809c5f8c3af6a2363c0a305

    - pull_request=2: # Mutable
        collection.json
        project.qgs

    - commit_prefix=3: # Immutable
        - catalog.json
        - commit=36125a7fbbae012f2809c5f8c3af6a2363c0a305:
            collection.json
            project.qgs
        - commit=3dfee564ea82f3da4ae16b84f5fe4a833e4c89d1:
            collection.json
            project.qgs

    - commit_prefix=4: # Immutable
        - catalog.json
        - commit=4aba34b5accb0002867af66f6a92a35e0a4be7ca:
            collection.json
            project.qgs

Modifications

Correct all the paths.
Add addition catalog under the mapseries level as followed. ./nztopo50map/catalog.json

Verification

https://github.com/linz/topographic-qgis/actions/runs/23275314183/job/67676963416?pr=16

github-actions · 2026-03-18T23:59:48Z

Changes Summary

Total Features Changed: 0
Datasets Affected: 0
Git Diff Lines: 1

Kart Diff

Kart Diff (0 features)

Git Diff

Git Diff (1 lines)

packages/kart/src/cli/__tests__/action.flow.test.container.ts

packages/kart/src/cli/action.to.parquet.ts

packages/kart/src/cli/action.validate.ts

schmidtnz · 2026-03-31T00:13:25Z

packages/kart/src/cli/action.validate.ts

+    // const rootCatalog = new URL('catalog.json', args.output);



Suggested change

// const rootCatalog = new URL('catalog.json', args.output);

schmidtnz · 2026-03-31T00:19:28Z

packages/map/src/limit.ts

+export function qFromArgs(args: {} | { concurrency?: number }): LimitFunction {
+  if ('concurrency' in args && typeof args.concurrency === 'number') return pLimit(args.concurrency);
+  return pLimit(qLimitDefault);
+}


does the limiter want to live in shared?

schmidtnz · 2026-03-31T01:15:50Z

packages/shared/src/download.ts

+}
+
+/**
+ * Recursively find the target data collection.json from the root catalog,


Suggested change

* Recursively find the target data collection.json from the root catalog,

* Recursively find the target data latest/collection.json from the root catalog,

schmidtnz · 2026-03-31T01:16:13Z

packages/shared/src/download.ts

+ * @param stacUrl The URL of the root STAC catalog.
+ * @param layerName The name of the vector layer to find.
+ *
+ * @returns Target data collection.json URL if found, otherwise throws an error.


Suggested change

* @returns Target data collection.json URL if found, otherwise throws an error.

* @returns Target data latest/collection.json URL if found, otherwise throws an error.

schmidtnz · 2026-03-31T01:16:48Z

packages/shared/src/download.ts

+ *
+ * @returns Target data collection.json URL if found, otherwise throws an error.
+ */
+export async function getDataFromCatalog(stacUrl: URL, layerName: string): Promise<URL> {


should this accept an optional version that defaults to latest?

schmidtnz · 2026-03-31T01:25:44Z

packages/shared/src/download.ts

+  }
+
+  const dataLink = catalog.links.find((link) => link.href.endsWith('/data/catalog.json'));
+  if (dataLink) return getDataFromCatalog(new URL(dataLink.href, stacUrl), layerName);


Am I misreading this or does this only really recurse into nested layers of data/data/data/.../data/layer/ directories?
Can we just return a link to new URL('/data/layer/latest/collection.json', stacUrl) if that exists, instead of all this magic?

schmidtnz · 2026-03-31T02:32:39Z

packages/shared/src/parquet.metadata.ts

+      const name = col.meta_data.path_in_schema.join('.');
+      const current = (tableStats[name] ??= { name, type: col.meta_data.type.toLowerCase() });
+      aggregateStats(current, col);
+      if (createDateKey !== 'create_date' && name.endsWith('_date')) createDateKey = name;


createDateKey will end up with the value create_date (if present) or the last column ending in _date.
It might make more sense to

use any known date field
OR

rely on only one known field (in which case, updated would make more sense than created?)

for the temporal extents.

schmidtnz · 2026-03-31T03:13:48Z

packages/stac/src/__test__/paths.test.ts

+    const sourcefile = new URL('https://foo.com/foo/bar/collection.json');
+    const sourceDir = new URL('.', sourcefile);


Suggested change

const sourcefile = new URL('https://foo.com/foo/bar/collection.json');

const sourceDir = new URL('.', sourcefile);

const sourceFile = new URL('https://foo.com/foo/bar/collection.json');

const sourceDir = new URL('https://foo.com/foo/bar/');

NIT - I'd prefer this to be set explicitly

schmidtnz · 2026-03-31T03:16:09Z

packages/stac/src/__test__/paths.test.ts

+  });
+
+  it('should not make relative if protocol hosts or other parts differ', () => {
+    assert.equal(getRelativePath(new URL('s3://foo/a.txt'), new URL('s3://foo/b.txt')), './a.txt');


This isn't testing what it says.

Suggested change

assert.equal(getRelativePath(new URL('s3://foo/a.txt'), new URL('s3://foo/b.txt')), './a.txt');

assert.equal(getRelativePath(new URL('s3://foo/a.txt'), new URL('s3://bar/b.txt')), 's3://foo/a.txt');

schmidtnz · 2026-03-31T04:07:10Z

packages/stac/src/geo.ts

+
+function extendItemFromCollection(base: StacItem, extension: StacCollection) {
+  const bounds = [];
+  if (base.geometry) bounds.push(normalizePolygonToMultiPolygon(base));


extract bit that's shared between this and extendCollectionFromItem into a separate function?

Update the order of qgis project deployment path

257da54

Wentao-Kuang requested a review from a team as a code owner March 18, 2026 01:43

Wentao-Kuang had a problem deploying to nonprod March 18, 2026 01:43 — with GitHub Actions Failure

Wentao-Kuang mentioned this pull request Mar 18, 2026

ci: Visual diff for the qgis project. BM-1554 linz/topographic-qgis#16

Open

Restructure the qgis deployment into hive structure.

72ed8b5

Wentao-Kuang temporarily deployed to nonprod March 18, 2026 23:55 — with GitHub Actions Inactive

Wentao-Kuang changed the title ~~feat(map): Update the order of qgis project deployment path~~ feat(map): Update the order of qgis project deployment path into hive structure. BM-1577 Mar 18, 2026

Wentao-Kuang added the container build pull request container for testing label Mar 18, 2026

Wentao-Kuang temporarily deployed to nonprod March 18, 2026 23:58 — with GitHub Actions Inactive

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 00:01 — with GitHub Actions Inactive

Refactoring the code

1b2298a

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 00:58 — with GitHub Actions Inactive

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Mar 19, 2026

Remove the back up file

e59a84a

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 00:59 — with GitHub Actions Inactive

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 01:03 — with GitHub Actions Inactive

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 01:06 — with GitHub Actions Inactive

Add missing types

9b2b36e

Wentao-Kuang temporarily deployed to nonprod March 19, 2026 01:13 — with GitHub Actions Inactive

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Mar 19, 2026

blacha temporarily deployed to nonprod March 27, 2026 03:38 — with GitHub Actions Inactive

blacha had a problem deploying to nonprod March 27, 2026 03:41 — with GitHub Actions Failure

blacha temporarily deployed to nonprod March 27, 2026 03:41 — with GitHub Actions Inactive

blacha added 9 commits March 27, 2026 17:15

fix: e2e needs npm ci

09a2c6c

refactor: invaliad e2e command

5f812e7

feat: ensure stac is valid when items are updated

52bc376

refactor: fixup build failure

5a53ebd

refactor: lint

852edb5

refactor: show logs

78ab8aa

refactor: ensure files are downloaded before map production

705bcab

refactor: ignore e2e test folder

1462f7c

refactor: use a cmd-ts parser

edcdc7d