enh: Make Narwhals pluggable

[MarcoGorelli]

In theory, Narwhals is extensible. In practice, it's not clear what the rules are around that, nor how to test extensions

#2978 starts the plugin system, and https://github.com/MarcoGorelli/narwhals-daft will be meant to serve as a reference implementation

[MarcoGorelli]

@dangotbanned regarding narwhals-dev/narwhals-daft#9 (comment) , could spell out your suggestion in terms of how it would change #2978 please?

[MarcoGorelli]

i don't think this actually defers importing daft. or rather, it imports daft if I have narwhals-daft installed even if native_object isn't a daft dataframe

[dangotbanned]

i don't think this actually defers importing daft. or rather, it imports daft if I have narwhals-daft installed even if native_object isn't a daft dataframe

I feel like I'm missing something here 🤔

If someone has installed a plugin, why would it matter whether or not is has been imported?

If someone authors a plugin, are they expected to write it in such a way to avoid imports at all costs?

I don't think these are questions that generally make much sense to answer this early

[MarcoGorelli]

i think we can expect plugins to expose a is_native_object function which checks if it's a native object without importing it, just like we have is_pandas_dataframe which checks if it's a pandas dataframe without importing pandas

[MarcoGorelli]

msg = "eager_only and series_only options are not supported as daft is lazy-only."

not sure i get this either, eager_only=False should be allowed

[dangotbanned]

msg = "eager_only and series_only options are not supported as daft is lazy-only."

not sure i get this either, eager_only=False should be allowed

It is allowed, that validation happens after using the plugin doesn't it?

narwhals/narwhals/translate.py

Lines 357 to 398 in 0aeb59d

	if sys.version_info >= (3, 10):
	discovered_plugins = discover_plugins(group="narwhals.plugins")
	for plugin in discovered_plugins: # pragma: no cover
	obj = plugin.load()
	with contextlib.suppress(TypeError):
	native_object = obj.from_native(native_object)
	break
	# Extensions
	if is_compliant_dataframe(native_object):
	if series_only:
	if not pass_through:
	msg = "Cannot only use `series_only` with dataframe"
	raise TypeError(msg)
	return native_object
	return version.dataframe(
	native_object.__narwhals_dataframe__()._with_version(version), level="full"
	)
	if is_compliant_lazyframe(native_object):
	if series_only:
	if not pass_through:
	msg = "Cannot only use `series_only` with lazyframe"
	raise TypeError(msg)
	return native_object
	if eager_only or eager_or_interchange_only:
	if not pass_through:
	msg = "Cannot only use `eager_only` or `eager_or_interchange_only` with lazyframe"
	raise TypeError(msg)
	return native_object
	return version.lazyframe(
	native_object.__narwhals_lazyframe__()._with_version(version), level="full"
	)
	if is_compliant_series(native_object):
	if not allow_series:
	if not pass_through:
	msg = "Please set `allow_series=True` or `series_only=True`"
	raise TypeError(msg)
	return native_object
	return version.series(
	native_object.__narwhals_series__()._with_version(version), level="full"
	)

Tbh that extra error handling + signature extension can probably just be removed from DaftNamespace
I think I only half realised we didn't need it, and only removed it from the narwhals-side 😅

[ym-pett]

thank you for the ideas & discussion @dangotbanned! To understand better I need to see it implemented & play with things, so that's what I'll do next. I might branch off with Dan's suggestions and continue on the create_fromnative_daft with your suggestion @MarcoGorelli.

I'll drop questions/comments as I go here.

[dangotbanned]

How do plugins change backwards-compatibility?

This is a very broad and open-ended question 😄

Feel free to explore however seems appropriate, but to help I'll give two examples of the kind of issues I'm thinking about

Internal Example

We have a PR in progress which will add an optional keep argument to Expr.mode

• feat: Allow mode(..., keep="any") #3019
• https://github.com/narwhals-dev/narwhals/pull/3019/files#diff-fcc0aa435a8b4a82b4906d293391578edbffb75873d992f6f8b32704f07bedac

Adding a new optional argument with a default allows us to maintain backwards compatibility in nw.Expr.
However, it becomes a breaking change for a plugin's *Expr.mode to suddenly receive an argument to a method that did not accept it before.

External Example

scikit-learn is an incredibly popular project, which has been around for a very long time.

Despite this, there still seems to be regular breaking changes for downstream extenders like sktime

• Stable extender contract via fit / _fit resp predict / _predict separation scikit-learn/scikit-learn#31912

Is there anything we can learn from this case?

[FBruzzesi]

@dangotbanned I think that one misconception is that a plugin must implement the entire set of methods. But from a user point of view, it doesn't.

Say plotly uses 10 methods from narwhals, and now wants to support dict[str, list[Any]] as if it was a dataframe. Then they will just need to implement those methods that they need. Everything else can be not_implemented (or not even that explicitly not implemented).

I understand that from the protocol point of view it won't be an actual compliant dataframe, but why should they care about methods they don't need?. As long as it implements __narwhals_dataframe__ and the methods they need, then it is usable within the narwhals logic and framework.

---

Regarding this #3018 (reply in thread), it definitely feels too strict for me in the current moment:

• there are no plugins that we are aware of
• if there were, we don't know if this change would actually break
• one such change for mode enable to use it across 4 more backends and in group by context, which was not possible before - The effort to update a plugin can be done by us, similarly to how we often take care of downstream projects if we introduce a breaking change.

The behavior of the existing signature MUST not change

That's already the case for stable namespaces. We change the behavior if and only if polars does it or something is unstable (This brings me to the following question: what if polars changes behavior? What should we do in such a case?)

We already make a lot of promises for downstream users, I have the feeling we want to achieve too much for a use case we don't really have just yet.

---

Maybe this is something worth discussing in a (emergency) community call since everyone has a different idea in a way or another 😇

[MarcoGorelli]

😆

I understand that from the protocol point of view it won't be an actual compliant dataframe

it's also fine to, as we're doing in narwhals-daft, just put not_implemented in all the non-implemented places to appease the type-checker. Or, as Dan suggested, use __getattr__ and raise a standardised error message for non-implemented methods

I have the feeling we want to achieve too much for a use case we don't really have just yet.

Agree, I think we can strive to keep the compliant protocols backwards compatible, but "MUST not change" feels like too strong of a statement

Plugins address the 1% use-case that we can't or don't want to support directly in Narwhals, I don't think we need to make as strong compatibility promises there.

We'll try our best, and the rules set out in #3018 (reply in thread) look like very good guidelines to me, so long as don't follow the religiously and allow ourselves to break them if necessary

---

It's certainly good to be thinking about backcompat for plugins anyway, so thanks for bringing all this up!

[dangotbanned]

Thanks both (#3018 (reply in thread)), (#3018 (reply in thread))

Sorry to be boring, but I don't find myself disagreeing with much of what either of you said 😂

Regarding the strictness of #3018 (reply in thread) - I'm in no way locked in on those rules specifically.
The main goal I would want - is to provide honest guidance on what a extension author can and cannot expect.

If we want to keep things flexible, then an admonition in /backcompat/ to directly state that this doesn't apply to extensions is also fine 🙂

[MarcoGorelli]

yup that's a good idea!

[ym-pett]

apologies if I'm not contributing to the discussion - I'm currently in learning mode :). I'm trying to summarise the different thoughts in a document for my understanding and may post a link when done.

[tokoko]

Have you thought about packaging tests as a separate library? narwhals-tests for example. plugins would be able to run them w/o adding narwhals as a submodule.

[MarcoGorelli]

if possible, i like the idea, submodules are finnicky to work with