Normalization (and automated conversion R/Python) of data types ?

[Artur-man]

What would be the ideal input for the dtype normalization below ? say that the input arrays are composed of characters should we pass S8 or U8 or is it possible to pass chararcter and let Dtype automatically figure out the correct numpy data type ?

https://github.com/keller-mark/pizzarr/blob/f84355d2708c22dc6e703f3cdd83d218221b352a/R/normalize.R#L106-L130

The typical scenario would be that one inserts a full character array, then type is provided to the dtype appropriately.

zarr.array <- pizzarr::zarr_open(store = "data/string_test.zarr")
z1 <- zarr.array$create_dataset(name = "assay", data = array(rep("a", 10), dim = 10), shape = 10)
zarr.array$get_item("assay")$get_item("...")$data

[1] "Buffer has ${numDataElements} of dtype ${dtype}, shape is too large or small"
Error in private$chunk_getitem_part2(part1_result, proj$chunk_coords,  : 
  Different type of error - rethrow

Looks like now the type is given as "<f8" (float32 ?) if not provided by the user.

[dblodgett-usgs]

Just leaving a note on what this is about...

This issue is about automatic dtype inference for character/string data in pizzarr.

The problem: when a user creates a dataset with data = array(rep("a", 10), dim = 10) without specifying dtype, pizzarr defaults to (float64), which then fails because the data is character, not numeric.

The ask: pizzarr should detect that the input data is character and automatically select an appropriate string dtype (like |S8 or |U8) instead of requiring the user to manually specify it.

This is essentially a dtype inference gap — the zarr_create / create_dataset path doesn't inspect the R data type to pick a sensible default dtype for non-numeric data.