Skip to content

Managed checkpoint 0.4.3#568

Open
mjohns-databricks wants to merge 60 commits intomainfrom
managed_checkpoint_0.4.3
Open

Managed checkpoint 0.4.3#568
mjohns-databricks wants to merge 60 commits intomainfrom
managed_checkpoint_0.4.3

Conversation

@mjohns-databricks
Copy link
Contributor

In preparation for upcoming transition to dblabs spatial-utils (follow-on to dblabs mosaic but for DBR 14.3+, adjusted for various enhancements relating to our productized geospatial APIs):

  • Significant streamlining of internal GDAL Dataset handling to include "hydrating" (loading the object) more lazily
  • Dropped "Mosaic" from the serialized internal objects: MosaicRasterTile, MosaicRasterGDAL, and MosaicRasterBandGDAL.
  • All newly generated RasterTile objects can store the tile payload (BinaryType | StringType | GDAL Dataset) to the configured fuse checkpoint dir (see below); raster tiles generated in 0.4.1 and 0.4.2 can be loaded as-is (structure was different prior to that).

Additionally, there are various maintenance commits, see [DRAFT] CHANGELOG for more.

…cleanup based on configured policies, e.g. file age of 30 minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant