Allow for the pyarrow type system for _pandas_like backends

[tswast]

Context: I'd like to make a "bigframes" (BigQuery DataFrames) Narwhals backend (work-in-progress here: https://github.com/tswast/narwhals/commits/bigframes/)

While implementing this, the biggest roadblock I hit is with regards to dtypes. In general, BigFrames defaults to pyarrow as the default dtypes for multiple reasons.

1. It avoids data loss. BigQuery treats NULL and NaN as separate values, whereas this isn't possible with the numpy-based dtypes.
2. It seems to be the direction of pandas (though this is hotly debated: Moving to PyArrow dtypes by default pandas-dev/pandas#61618)

I'm curious if it would be possible to make _pandas_like backends that use the pyarrow type system? Alternatively, maybe there's a way for a _pandas_like backend to supply its own mapping from narwhals dtypes to pandas-y dtypes?

[dangotbanned]

Thanks for opening this discussion, @tswast (apologies nobody has seen it yet)

It seems like you're very clued in on the woes of pandas' various type systems!

1. It avoids data loss. BigQuery treats NULL and NaN as separate values, whereas this isn't possible with the numpy-based dtypes.

Yep, we're kinda dealing with this at the moment - but it is somewhat of a moving target (#3160 (comment))

I'm curious if it would be possible to make _pandas_like backends that use the pyarrow type system?

These are all the pandas APIs I know of that may be helpful:

import pandas as pd

# NOTE: `pandas>=3` only
pd.options.mode.nan_is_na = False

# NOTE: Not in `pandas-stubs`
pd.options.future.infer_string = True
pd.options.future.no_silent_downcasting = True

pd.options.mode.string_storage = "pyarrow"

pd.DataFrame().convert_dtypes(dtype_backend="pyarrow")
pd.Series().convert_dtypes(dtype_backend="pyarrow")

Publicly we have narwhals.maybe_convert_dtypes, which wraps the last two methods.

Internally, this search will point you to where you can specify dtype_backend="pyarrow".

One last thing, while testing bigframes you might find it helpful to compare against these two 🙂

pytest --constructors='pandas[pyarrow],modin[pyarrow]'

[MarcoGorelli]

hi @tswast - i don't understand the question, could you elaborate please?

[dangotbanned]

I think the main ask was:

I'm curious if it would be possible to make _pandas_like backends that use the pyarrow type system?

Which I interpreted as:

Could we have the option to configure narwhals to ensure dtype_backend="pyarrow" is always used?

I think this would be a very desirable feature to have - but since pandas themselves are still struggling with preserving pyarrow dtypes between operations - I'm not sure what we could do to improve things 😢

[tswast]

@dangotbanned That's a great summary, yeah.

For some additional context, the pyarrow type system is desirable for us mainly to avoid the implicit cast to float64 that happens with the numpy types whenever null values are involved. This implicit cast is a lossy one that causes data loss for the more extreme integer values.

[MarcoGorelli]

It should just use what the user passes in, and preserve the pyarrow dtypes where possible

Is that not what you're seeing? Do you have an example?

[tswast]

I haven't tried too much yet. I'm wondering what to do about the test suite, as it tests a lot of numpy types that bigframes doesn't support.

…

On Tue, Dec 16, 2025 at 9:54 AM Marco Edward Gorelli < ***@***.***> wrote: It should just use what the user passes in, and preserve the pyarrow dtypes where possible Is that not what you're seeing? Do you have an example? — Reply to this email directly, view it on GitHub <#3180 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4OAZGVOS3UX4NTE6XSVL4CATKZAVCNFSM6AAAAACINQGYLWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMRXGA4DKNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>