Preserve nanosecond precision in Python dataset construction

[sophianggan]

What it is

Preserves exact nanosecond precision for pandas timestamps and timedeltas when users declare Value("timestamp[ns]") or Value("duration[ns]").

Previously, Dataset.from_dict, Dataset.from_list, and Dataset.map reported a nanosecond feature while silently storing values truncated to microseconds. For example, 2024-01-01 00:00:00.123456789 became 2024-01-01 00:00:00.123456.

How it works

During Python dataset construction, pandas temporal scalars were first converted to microsecond-resolution Python objects. The writer then created an untyped Arrow array, so the discarded digits could not be recovered by the later cast to timestamp[ns] or duration[ns].

• Feature encoding. Pandas temporal scalars remain intact when the declared feature contains a nanosecond timestamp or duration.
• Arrow construction. The writer passes the known target type to pa.array(...), allowing PyArrow to preserve the requested resolution from the start.
• Compatibility. Unsupported inputs fall back to the existing conversion and casting path.
• Nested features. Detection covers lists, large lists, tuples, and structs. A cached column set avoids repeatedly traversing schemas on the write path.

Scope

The new path applies only to explicitly declared nanosecond timestamp and duration features. Default Python-object conversion, other temporal units, and undeclared type inference remain unchanged.

The fix supports naive and timezone-aware timestamps, timedeltas, mixed pandas and Python values, nulls, try_type, nested lists, and nested structs.

Verification

• Unit and integration suite: exact Arrow int64 values for timestamps, durations, timezone-aware values, mixed temporal inputs, nulls, try_type, and nested features.
• Public API coverage: Dataset.from_dict, Dataset.from_list, and Dataset.map, checking both decoded rows and underlying Arrow storage.
• Feature mutation coverage: cached nanosecond columns remain synchronized across all supported Features dictionary mutations.
• E2E wheel top-hat: built a wheel, installed it in a clean temporary environment, and exercised all three public construction paths outside the source checkout. Each preserved timestamp 1704067200123456789 and duration 86400123456789 exactly.
• Apple Silicon: validated on native ARM64 macOS with Python 3.13 and PyArrow 25.0.0.

Checks

.venv/bin/pytest -q tests/features/test_features.py tests/test_arrow_writer.py tests/test_arrow_dataset.py   659 passed, 58 skipped, 6 subtests passed
.venv/bin/ruff check src/datasets/arrow_writer.py src/datasets/features/features.py tests/features/test_features.py tests/test_arrow_writer.py tests/test_arrow_dataset.py   passed
.venv/bin/ruff format --check src/datasets/arrow_writer.py src/datasets/features/features.py tests/features/test_features.py tests/test_arrow_writer.py tests/test_arrow_dataset.py   passed
git diff --check   passed
.venv/bin/python -m build --wheel   passed
clean-wheel E2E top-hat for from_dict, from_list, and map   passed

---

Closes #8391.