to_json cannot round-trip temporal columns: all timestamps written as epoch milliseconds

[LeSingh1]

Describe the bug

Dataset.to_json() serializes every temporal column as a bare epoch integer at a single fixed resolution (pandas' date_unit="ms" default), with no unit recorded in the output. Timestamps therefore do not survive a to_json → from_json round trip: they either come back silently wrong or raise.

Steps to reproduce the bug

import datetime, tempfile
from datasets import Dataset, Features, Value

for unit, val in [("s",  datetime.datetime(2024,1,1,12,30,45)),
                  ("ms", datetime.datetime(2024,1,1,12,30,45,123000)),
                  ("us", datetime.datetime(2024,1,1,12,30,45,123456))]:
    f = Features({"t": Value(f"timestamp[{unit}]")})
    ds = Dataset.from_dict({"t": [val]}, features=f)
    p = tempfile.mktemp(suffix=".jsonl")
    ds.to_json(p)
    print(unit, open(p).read().strip(), end="  ")
    try:
        print(Dataset.from_json(p, features=f)[0]["t"])
    except Exception as e:
        print(type(e).__name__, e)

s   {"t":1704112245000}  OverflowError date value out of range
ms  {"t":1704112245123}  2024-01-01 12:30:45.123000        # only this one works
us  {"t":1704112245123}  1970-01-20 17:21:52.245123        # silently wrong

The timestamp[us] case is the most dangerous: no error, and the value comes back 54 years off. timestamp[us] and [ns] also silently lose sub-millisecond precision on write. Reading back without explicit features yields Value('int64') for all three.

Cause

src/datasets/io/json.py calls batch.to_json(...) without setting date_format / date_unit, so pandas' date_unit="ms" default is applied uniformly to every temporal column regardless of its Arrow unit.

Notes for whoever picks this up

There is no single date_unit value that fixes this, because pandas applies one global setting to all temporal columns:

• date_unit="us" fixes [us] but breaks [s] (OverflowError: Python int too large to convert to C int) and [ms] (OverflowError: date value out of range) — both of which behave acceptably today.
• date_format="iso" makes the value self-describing and fixes [us], but [s] and [ms] then fail to load with DatasetGenerationError, and inference reads the column back as Value('string').

So it likely needs the writer to derive the unit/format per column from its Arrow type, and possibly the JSON loader to recognise ISO timestamps. Worth noting that changing the on-disk format affects every existing JSON export, so this may warrant an opt-in or a version-gated default.

Happy to open a PR if you can point me at the direction you'd prefer.

Environment info

• datasets 5.0.2.dev0 (main @ b7cb10b)
• pyarrow 25.0.0, pandas 2.x, Python 3.12