diff --git a/doc/development/api_vs_testing.rst b/doc/development/api_vs_testing.rst new file mode 100644 index 000000000..a567b5b4e --- /dev/null +++ b/doc/development/api_vs_testing.rst @@ -0,0 +1,208 @@ +.. _dev_api_vs_testing: + +Two layers, one purpose: the dispatch API and ``df_module`` +============================================================ + +skrub has two distinct mechanisms for handling multiple dataframe backends: +the dispatched *dataframe API* (``skrub/_dataframe``, ``skrub/_dispatch.py``) and +the *``df_module`` fixture* (``skrub/conftest.py``). They look superficially +similar — both abstract over pandas and polars — but they exist at different +levels and serve different roles. This guide explains the design rationale, +where the boundary between them lies, and how to decide which to use. + +.. contents:: Contents + :local: + :depth: 2 + + +The two layers at a glance +--------------------------- + ++---------------------------+--------------------------------------+------------------------------------+ +| | Dispatch API | ``df_module`` fixture | ++===========================+======================================+====================================+ +| **Where** | ``skrub/_dataframe/_common.py``, | ``skrub/conftest.py`` | +| | ``skrub/_dispatch.py`` | | ++---------------------------+--------------------------------------+------------------------------------+ +| **When it runs** | Production — at import time and | Test time — under pytest only | +| | when skrub functions are called | | ++---------------------------+--------------------------------------+------------------------------------+ +| **What it abstracts** | *How* to perform an operation | *How* to construct inputs and | +| | (fill nulls, get shape, cast, …) | assert outputs in a test | ++---------------------------+--------------------------------------+------------------------------------+ +| **Who uses it** | skrub transformers, encoders, | Test functions | +| | utility functions | | ++---------------------------+--------------------------------------+------------------------------------+ +| **Mechanism** | ``functools.singledispatch`` + | pytest ``@fixture(params=…)`` | +| | ``specialize`` decorator | | ++---------------------------+--------------------------------------+------------------------------------+ +| **Result of abstraction** | A single call site (``sbd.fill_nulls``) | A single test body runs 3 times | +| | works for any backend | (one per configuration) | ++---------------------------+--------------------------------------+------------------------------------+ + + +The dataframe API: writing library-agnostic production code +----------------------------------------------------------- + +The dataframe API solves a problem that arises *at runtime*: a transformer +receives a DataFrame or a Series whose backend is not known at the time the +code was written. The ``@dispatch`` / ``specialize`` mechanism routes the +call to the correct implementation based on the actual type of the object. + +Call sites in production code are completely library-agnostic: + +.. code-block:: python + + import skrub._dataframe as sbd + + def _process(col): + if sbd.has_nulls(col): + col = sbd.fill_nulls(col, 0) + return sbd.to_float32(col) + +Neither ``pandas`` nor ``polars`` is imported here. The correct +implementation — ``col.fillna(0)`` for pandas or ``col.fill_null(0)`` for +polars — is selected automatically. + +The key properties of the dataframe API: + +* It is **production code** that ships as part of skrub's package. +* It is imported and executed whenever a user calls a skrub estimator. +* It says nothing about *how to build* a DataFrame or *how to assert equality*; + it only defines *operations* on existing objects. +* It must handle real user data — arbitrary DataFrames that arrive from outside + skrub. + + +``df_module``: testing library-agnostic code +--------------------------------------------- + +``df_module`` solves a different problem: when a test needs to construct +inputs, call a function, and check the output, it must do so in a way that +works for all three configurations. ``df_module`` provides a uniform +interface for these test-time concerns. + +A test using ``df_module`` is collected once and run three times by pytest: + +.. code-block:: python + + def test_fill_nulls(df_module): + col = df_module.make_column("x", [1.0, None, 3.0]) + result = sbd.fill_nulls(col, 0.0) + expected = df_module.make_column("x", [1.0, 0.0, 3.0]) + df_module.assert_column_equal(result, expected) + +``df_module.make_column`` builds a ``pd.Series`` (numpy dtypes), a +``pd.Series`` (nullable dtypes), or a ``pl.Series`` depending on the +parameter. ``df_module.assert_column_equal`` delegates to the right +``*.testing`` module. No ``if`` branches, no duplication. + +The key properties of ``df_module``: + +* It is **test infrastructure** that lives in ``conftest.py`` and is never + imported in production code. +* It is only active when pytest runs. +* It knows how to *construct* DataFrames and *assert equality*, not how to + perform arbitrary operations on them. +* It works with controlled, synthetic data, not real user data. + + +Why not collapse them? +---------------------- + +The two layers could theoretically be merged — for example, ``df_module`` +could use ``sbd.*`` to build its example objects. There are deliberate +reasons not to do this. + +**``df_module`` does not build on the dataframe API.** + ``df_module`` is part of the test bootstrap. If it relied on ``sbd.*`` + internally, a bug in the dispatch layer would corrupt the test inputs + themselves, making it impossible to distinguish "the function under test is + broken" from "the test fixture is broken". Using the backends' own + constructors (``pd.DataFrame.from_dict``, ``pl.from_dict``, …) keeps the + fixture independent. + +**The dataframe API does not know about test concerns.** + The dispatch API is a production abstraction over dataframe *operations*. + Concepts like "an example DataFrame with four rows and mixed dtypes" or + "assert that two Series are equal up to dtype" are test concerns, not + operation concerns. Mixing them would blur the boundary between production + and test code. + +**Three configurations, not two libraries.** + The dataframe API dispatches on ``pandas.DataFrame`` vs ``polars.DataFrame`` + — two cases. The test suite has three configurations: pandas-numpy-dtypes, + pandas-nullable-dtypes, and polars. The extra pandas configuration catches + dtype-specific bugs that would be invisible if tests only ran against one + pandas variant. The dataframe layer has no notion of "which pandas", + because from a runtime perspective both are ``pd.DataFrame``; only the test + layer needs to distinguish them. + + +Decision guide +-------------- + +Use this table to decide where new code or infrastructure belongs. + ++-----------------------------------------------+-----------------------------------+ +| Situation | What to do | ++===============================================+===================================+ +| I need to perform a dataframe operation in | Use ``sbd.*``. If the function | +| a transformer or utility function. | does not exist yet, add it to | +| | ``_common.py`` (see | +| | :ref:`dev_dataframe_api`). | ++-----------------------------------------------+-----------------------------------+ +| I need to perform an operation that is | Define a local ``@dispatch`` | +| specific to one module (e.g. a helper in | function in that module. Do not | +| ``_datetime_encoder.py``) and has no reuse | add it to ``_common.py``. | +| outside it. | | ++-----------------------------------------------+-----------------------------------+ +| I need to write a test for code that touches | Use ``df_module``. | +| a DataFrame or column. | | ++-----------------------------------------------+-----------------------------------+ +| I need a test that must run only for pandas. | Use ``pd_module`` instead of | +| | ``df_module``. | ++-----------------------------------------------+-----------------------------------+ +| I need a test that must run only for polars. | Use ``pl_module`` (auto-skips if | +| | polars is not installed). | ++-----------------------------------------------+-----------------------------------+ +| I need to construct a backend-appropriate | Use ``df_module.make_dataframe`` | +| DataFrame in a test. | or ``df_module.make_column``. | ++-----------------------------------------------+-----------------------------------+ +| I need a dtype value in a test that works | Use ``df_module.dtypes["float64"]``| +| across configurations. | (or whichever key you need). | ++-----------------------------------------------+-----------------------------------+ +| I need to assert equality in a test. | Use ``df_module.assert_frame_equal``| +| | or ``df_module.assert_column_equal``.| ++-----------------------------------------------+-----------------------------------+ +| I am unsure whether a new operation belongs | If other transformers would | +| in ``_common.py`` or should be local. | benefit from it: ``_common.py``. | +| | If it is specific to one class: | +| | local ``@dispatch``. | ++-----------------------------------------------+-----------------------------------+ + +A concrete heuristic: if you are writing code that will run when a user calls +``TableVectorizer().fit_transform(df)``, use the dataframe API. If you are +writing code that only runs under ``pytest``, use ``df_module``. + + +Parallel structure +------------------ + +Despite their different purposes, the two layers do mirror each other in one +respect: both have a "all backends" path and a "specific backend" escape hatch. + +In the **dataframe API**: + +* ``sbd.fill_nulls(col, 0)`` — generic, works for any backend. +* ``@fill_nulls.specialize("pandas", argument_type="Column")`` — pandas-specific + implementation, invoked automatically. + +In the **test fixture**: + +* ``df_module`` — generic, runs for every configuration. +* ``pd_module`` / ``pl_module`` — fixed to one backend, used when the test + itself is backend-specific. + +The design principle is the same in both cases: write the general case once +and isolate backend differences to dedicated, clearly labelled places. diff --git a/doc/development/dataframe_api.rst b/doc/development/dataframe_api.rst new file mode 100644 index 000000000..4640602fb --- /dev/null +++ b/doc/development/dataframe_api.rst @@ -0,0 +1,358 @@ +.. _dev_dataframe_api: + +The dispatch-based dataframe API +================================= + +skrub targets both pandas and polars as first-class backends. Rather than +scattering ``if pandas … else polars …`` branches throughout the codebase, all +dataframe and column operations are funnelled through a thin dispatch layer that +selects the right implementation at call time. This guide explains how that +layer works and how to extend it. + +.. contents:: Contents + :local: + :depth: 2 + + +Motivation +---------- + +The naive approach to multi-backend support is to branch on the library name +wherever an operation is performed: + +.. code-block:: python + + # don't do this + if isinstance(col, pd.Series): + result = col.fillna(0) + else: + result = col.fill_null(0) + +This pattern is fragile: the same check must be repeated everywhere, adding a +third backend means touching every branch, and tests must cover every path +explicitly. The dispatch layer inverts the design: each operation is defined +once as a *generic function*, and the concrete implementation is registered +separately per library. Call sites stay library-agnostic. + +.. code-block:: python + + import skrub._dataframe as sbd + + result = sbd.fill_nulls(col, 0) # works for pandas Series or polars Series + + +How dispatching works +--------------------- + +The mechanism lives in ``skrub/_dispatch.py`` and is built on top of the +standard library's :func:`functools.singledispatch`. + +``functools.singledispatch`` selects an implementation based on the *type* of +the first argument. The wrinkle is that some backends (currently polars) are +optional dependencies: you cannot import ``polars.DataFrame`` to register a +specialisation if polars is not installed. The ``dispatch`` decorator works +around this by accepting the library name as a string and resolving types only +when the library is actually importable. + +The type registry +~~~~~~~~~~~~~~~~~ + +Internally, ``_dispatch.py`` maintains a registry mapping library names to +their concrete types: + +.. code-block:: text + + "pandas" → { + "DataFrame": (pandas.DataFrame,), + "Column": (pandas.Series,), + } + + "polars" → { + "DataFrame": (polars.DataFrame,), + "LazyFrame": (polars.LazyFrame,), + "EagerFrame": (polars.DataFrame,), + "Column": (polars.Series,), + } + +These string names (``"DataFrame"``, ``"Column"``, ``"LazyFrame"``) are what +you pass to ``specialize``'s ``argument_type`` keyword. + +The ``@dispatch`` decorator +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Applying ``@dispatch`` to a function converts it into a generic function and +adds a ``specialize`` attribute: + +.. code-block:: python + + from skrub._dispatch import dispatch, raise_dispatch_unregistered_type + + @dispatch + def fill_nulls(col, value): + raise_dispatch_unregistered_type(col, kind="Series") + +The default body is the fallback that runs when no specialisation has been +registered for the argument's type. The idiomatic choice is to raise a +descriptive error with ``raise_dispatch_unregistered_type``, though some +functions use a safe no-op default (e.g. ``reset_index`` which is a pandas +concept and simply returns ``obj`` unchanged for everything else). + +Registering specialisations with ``specialize`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + @fill_nulls.specialize("pandas", argument_type="Column") + def _fill_nulls_pandas(col, value): + return col.fillna(value) + + @fill_nulls.specialize("polars", argument_type="Column") + def _fill_nulls_polars(col, value): + return col.fill_null(value) + +``specialize`` takes two arguments: + +* **Library name** (``"pandas"`` or ``"polars"``): the concrete implementations + are looked up only if that library is importable; otherwise the decorator + is a no-op and the function is never registered. +* **``argument_type``** (optional): one of the string keys in the type registry, + or a tuple of them. Omitting it registers the specialisation for *all* types + in that library (DataFrame, Column, and LazyFrame for polars). + ++----------------------------------+------------------------------------------------+ +| ``argument_type`` | Registers for | ++==================================+================================================+ +| ``None`` (default) | All types in the library | ++----------------------------------+------------------------------------------------+ +| ``"DataFrame"`` | DataFrame class only | ++----------------------------------+------------------------------------------------+ +| ``"Column"`` | Series class only | ++----------------------------------+------------------------------------------------+ +| ``"LazyFrame"`` | polars LazyFrame only | ++----------------------------------+------------------------------------------------+ +| ``("DataFrame", "Column")`` | Both DataFrame and Series | ++----------------------------------+------------------------------------------------+ + +The **last** registered specialisation wins for a given type; there is no +priority ordering based on specificity. This means that if you register a +specialisation for all pandas types and then later register one for +``"Column"`` only, the second one will override the first for Series objects. + +Naming convention +~~~~~~~~~~~~~~~~~ + +Specialised implementations follow the pattern +``__`` for library-wide specialisations, or +``___`` when the ``argument_type`` is scoped: + +.. code-block:: python + + _fill_nulls_pandas # pandas-wide + _fill_nulls_polars # polars-wide + _to_numpy_pandas_column # pandas, Column only + _to_numpy_pandas_table # pandas, DataFrame only + +The names are arbitrary and have no effect on dispatch; they are a +documentation and searchability convention. + +Error messages +~~~~~~~~~~~~~~ + +``raise_dispatch_unregistered_type`` produces three distinct error messages: + +* **Unknown type**: "Expecting a Pandas or Polars , but got …" +* **DataOp**: tells the caller to use ``.skb.eval()`` or ``.skb.apply_func()`` + instead. +* **LazyFrame**: tells the caller to call ``.collect()`` first. + + +Using the API +------------- + +Import convention +~~~~~~~~~~~~~~~~~ + +Throughout skrub, the module is imported under the alias ``sbd`` (or +occasionally ``ns`` in older code and docstrings): + +.. code-block:: python + + import skrub._dataframe as sbd + +This is a private module; it is not part of the public skrub API. + +Available functions +~~~~~~~~~~~~~~~~~~~ + +All public functions are re-exported from ``skrub/_dataframe/__init__.py`` +via ``from ._common import *``. They are grouped conceptually in +``_common.__all__``: + +**Type inspection** + ``dataframe_module_name``, ``is_pandas``, ``is_polars``, + ``is_dataframe``, ``is_lazyframe``, ``is_column`` + +**Conversions** + ``to_list``, ``to_numpy``, ``to_pandas``, + ``make_dataframe_like``, ``make_column_like``, ``null_value_for``, + ``all_null_like``, ``concat``, ``is_column_list``, ``to_column_list``, + ``col``, ``col_by_idx``, ``collect`` + +**Shape and metadata** + ``shape``, ``to_frame``, ``name``, ``column_names``, ``rename``, + ``set_column_names``, ``reset_index``, ``copy_index``, ``index``, ``drop`` + +**Dtype inspection and casting** + ``dtype``, ``dtypes``, ``cast``, ``is_bool``, ``is_numeric``, + ``is_integer``, ``is_float``, ``to_float32``, ``is_string``, ``to_string``, + ``is_object``, ``is_any_date``, ``to_datetime``, ``is_duration``, + ``is_categorical``, ``to_categorical``, ``is_all_null``, ``is_empty_frame``, + ``is_pandas_extension_dtype``, ``pandas_convert_dtypes``, ``is_pandas_object`` + +**Values** + ``all``, ``any``, ``sum``, ``min``, ``max``, ``std``, ``mean``, + ``pearson_corr``, ``sort``, ``value_counts``, ``quantile``, + ``is_null``, ``has_nulls``, ``drop_nulls``, ``fill_nulls``, + ``n_unique``, ``unique``, ``filter``, ``where``, ``where_row``, + ``sample``, ``head``, ``slice``, ``select_rows``, ``replace``, + ``with_columns``, ``abs``, ``total_seconds``, ``is_sorted`` + +Example usage: + +.. code-block:: python + + import skrub._dataframe as sbd + + # Works for a pandas Series or a polars Series + col = ... + if sbd.has_nulls(col): + col = sbd.fill_nulls(col, 0) + + # Works for a pandas DataFrame or a polars DataFrame + df = ... + n_rows, n_cols = sbd.shape(df) + names = sbd.column_names(df) + + +Adding a function to ``_common.py`` +------------------------------------ + +Step 1 — write the generic function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add the function near related ones in ``skrub/_dataframe/_common.py``. +The first argument must be the dataframe or column that will drive dispatch. + +.. code-block:: python + + @dispatch + def clip(col, lower, upper): + """Clip values in a column to [lower, upper].""" + raise_dispatch_unregistered_type(col, kind="Series") + +If a sensible no-op default exists (e.g. the operation is pandas-specific), +you can return ``obj`` or another safe value instead of raising. + +Step 2 — add specialisations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + @clip.specialize("pandas", argument_type="Column") + def _clip_pandas(col, lower, upper): + return col.clip(lower=lower, upper=upper) + + @clip.specialize("polars", argument_type="Column") + def _clip_polars(col, lower, upper): + return col.clip(lower_bound=lower, upper_bound=upper) + +Step 3 — add to ``__all__`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add the function name to the ``__all__`` list at the top of ``_common.py``, +in the appropriate section. + +Step 4 — write tests +~~~~~~~~~~~~~~~~~~~~~ + +Add a test in ``skrub/_dataframe/tests/test_common.py`` using the +``df_module`` fixture (see :ref:`dev_testing`). + +.. code-block:: python + + def test_clip(df_module): + col = df_module.make_column("x", [1, 5, 10, -3]) + result = sbd.clip(col, lower=0, upper=7) + expected = df_module.make_column("x", [1, 5, 7, 0]) + df_module.assert_column_equal(result, expected) + + +Defining dispatched functions outside ``_common.py`` +------------------------------------------------------ + +Not all dispatched functions belong in ``_common.py``. If the operation is +tightly coupled to a specific transformer or sub-module and has no use +elsewhere, define it locally in that module. Examples include +``_is_date`` and ``_get_dt_feature`` in ``skrub/_datetime_encoder.py``, and +``_str_replace`` in ``skrub/_to_float.py``. + +The pattern is identical to ``_common.py``, using the same ``dispatch`` and +``raise_dispatch_unregistered_type`` imported from ``skrub._dispatch``: + +.. code-block:: python + + # skrub/_my_transformer.py + + from ._dispatch import dispatch, raise_dispatch_unregistered_type + import skrub._dataframe as sbd + + @dispatch + def _extract_something(col): + raise_dispatch_unregistered_type(col, kind="Series") + + @_extract_something.specialize("pandas", argument_type="Column") + def _extract_something_pandas(col): + # pandas-specific implementation + return col.str.extract(r"(\d+)") + + @_extract_something.specialize("polars", argument_type="Column") + def _extract_something_polars(col): + # polars-specific implementation + return col.str.extract(r"(\d+)", group_index=0) + +A real example from ``skrub/_datetime_encoder.py``: + +.. code-block:: python + + @dispatch + def _is_date(col): + from ._dispatch import raise_dispatch_unregistered_type + raise_dispatch_unregistered_type(col, kind="Series") + + @_is_date.specialize("pandas", argument_type="Column") + def _is_date_pandas(col): + col = sbd.drop_nulls(col) + return (col.dt.normalize() == col).all() + + @_is_date.specialize("polars", argument_type="Column") + def _is_date_polars(col): + return (col.dt.date() == col).all() + +Functions defined this way are **not** exported from ``skrub._dataframe``; they +are module-private helpers. Only add a function to ``_common.py`` and its +``__all__`` when it is genuinely reusable across multiple parts of skrub. + + +Rules for production code +-------------------------- + +* **Always use** ``sbd.*`` for dataframe or column operations; never call + ``df.method()`` directly outside a ``specialize`` block. +* Keep the **first argument** as the dispatching argument (the dataframe or + column). A function ``sample(n, df)`` would dispatch on ``n``, which is + wrong; it must be ``sample(df, n)``. +* Inside a ``specialize`` block, it is safe to import the backend module and + call any of its methods — you are guaranteed the first argument is that + backend's type. +* Outside a ``specialize`` block, never import polars unconditionally; polars + is an optional dependency. diff --git a/doc/development/testing.rst b/doc/development/testing.rst new file mode 100644 index 000000000..81bcd735a --- /dev/null +++ b/doc/development/testing.rst @@ -0,0 +1,329 @@ +.. _dev_testing: + +Testing with the ``df_module`` fixture +======================================= + +skrub's test suite must verify that every dataframe-aware feature works +correctly for all supported backends and dtype configurations. Writing the +same test three times would be tedious and error-prone, so the suite provides a +parametrised fixture, ``df_module``, that multiplies a single test across all +configurations automatically. + +This guide explains how ``df_module`` works, what attributes it provides, and +how to write effective tests using it. + +.. contents:: Contents + :local: + :depth: 2 + + +Why a parametrised fixture +--------------------------- + +skrub supports three distinct configurations in practice: + +* **pandas with NumPy dtypes** (``pandas-numpy-dtypes``): the classical pandas + dtypes backed by NumPy arrays — e.g. ``np.float64``, ``np.int64``. Integer + columns that contain ``None`` will be cast to ``float64`` because NumPy + integers cannot represent missing values. + +* **pandas with nullable extension dtypes** (``pandas-nullable-dtypes``): + pandas' own nullable types — e.g. ``pd.Float64Dtype()``, ``pd.Int64Dtype()``. + These represent missing values without promoting integer columns to float and + behave somewhat differently from NumPy-backed dtypes. + +* **polars** (``polars``): polars DataFrames and Series, which have their own + type system, naming conventions, and API. + +A test that only runs under one configuration may pass while silently failing +under the other two. The ``df_module`` fixture ensures that a single test +function covers all three automatically. + +How pytest sees it: a test that requests ``df_module`` is collected once and +run three times, once per parameter, producing independent pass/fail results. +If polars is not installed, the polars parameter is absent and the test runs +twice. + + +Anatomy of ``df_module`` +------------------------- + +``df_module`` is defined in ``skrub/conftest.py`` and returns a +:class:`types.SimpleNamespace` with a consistent set of attributes. The +attributes are designed to normalise the differences between libraries so test +bodies need no ``if pandas / if polars`` branches (with few exceptions). + +The fixture signature: + +.. code-block:: python + + @pytest.fixture(params=["pandas-numpy-dtypes", "pandas-nullable-dtypes", "polars"]) + def df_module(request): + return _DATAFRAME_MODULES_INFO[request.param] + +Attributes +~~~~~~~~~~ + +``name`` — ``str`` + The library name: ``"pandas"`` or ``"polars"``. Useful when a test must + assert on the name or skip/branch based on the library. + + .. code-block:: python + + assert sbd.dataframe_module_name(df) == df_module.name + +``description`` — ``str`` + The full configuration key: ``"pandas-numpy-dtypes"``, + ``"pandas-nullable-dtypes"``, or ``"polars"``. Use this when you need to + distinguish between the two pandas configurations. + +``module`` — module object + The backend module itself (``pandas`` or ``polars``). Useful if you need + to access constants or secondary helpers directly. + +``DataFrame`` — class + The DataFrame class for this configuration: ``pd.DataFrame`` or + ``pl.DataFrame``. + +``Column`` — class + The column/series class: ``pd.Series`` or ``pl.Series``. + +``make_dataframe(data: dict) → DataFrame`` + Build a DataFrame from a column-name → values dictionary. Under + ``pandas-nullable-dtypes`` it additionally calls ``.convert_dtypes()``, so + the resulting dtypes are the nullable extension types. + + .. code-block:: python + + df = df_module.make_dataframe({"a": [1, 2, 3], "b": ["x", "y", "z"]}) + +``make_column(name: str, values: list) → Column`` + Build a single column. + + .. code-block:: python + + col = df_module.make_column("score", [1.0, 2.5, None]) + +``assert_frame_equal(left, right, **kwargs)`` + Assert that two DataFrames are equal, using the backend's own testing + helper (``pandas.testing.assert_frame_equal`` or + ``polars.testing.assert_frame_equal``). + +``assert_column_equal(left, right, **kwargs)`` + Assert that two columns are equal. + +``empty_dataframe`` — DataFrame + A DataFrame with zero rows and zero columns. Useful as a trivial input to + check that functions handle empty frames gracefully. + +``empty_column`` — Column + A column of length zero. + +``empty_lazyframe`` — polars LazyFrame + A lazy DataFrame with zero rows and zero columns. **Only present for the + polars configuration**; accessing it on a pandas ``df_module`` will raise + ``AttributeError``. + +``example_dataframe`` — DataFrame + A ready-made DataFrame containing one column of each common dtype: integer + (with nulls), integer (without nulls), float, string, boolean (with nulls), + boolean (without nulls), datetime, and date. The exact values are defined + by ``_example_data_dict`` in ``conftest.py``. Use this when you want a + realistic multi-type frame without constructing one manually. + +``example_column`` — Column + The ``"float-col"`` column from ``example_dataframe`` (floats with one + ``None``). + +``dtypes`` — ``dict`` + A mapping from dtype name (string) to the appropriate dtype value for this + configuration. The keys are ``"float32"``, ``"float64"``, ``"int32"``, + ``"int64"``, and ``"category"``. + + +----------+------------------+-------------------+-------------+ + | Key | numpy-dtypes | nullable-dtypes | polars | + +==========+==================+===================+=============+ + | float32 | ``np.float32`` | ``Float32Dtype`` | ``pl.Float32`` | + +----------+------------------+-------------------+-------------+ + | float64 | ``np.float64`` | ``Float64Dtype`` | ``pl.Float64`` | + +----------+------------------+-------------------+-------------+ + | int32 | ``np.int32`` | ``Int32Dtype`` | ``pl.Int32`` | + +----------+------------------+-------------------+-------------+ + | int64 | ``np.int64`` | ``Int64Dtype`` | ``pl.Int64`` | + +----------+------------------+-------------------+-------------+ + | category | ``CategoricalDtype`` | ``CategoricalDtype`` | ``pl.Categorical`` | + +----------+------------------+-------------------+-------------+ + + +Writing a basic test +--------------------- + +Here is a minimal test of a hypothetical ``my_transform`` function: + +.. code-block:: python + + import skrub._dataframe as sbd + + def test_my_transform(df_module): + # Build backend-appropriate inputs + col = df_module.make_column("x", [1.0, 2.0, None, 4.0]) + + result = my_transform(col) + + expected = df_module.make_column("x", [1.0, 4.0, None, 16.0]) + df_module.assert_column_equal(result, expected) + +A few rules of thumb: + +* Build inputs with ``df_module.make_dataframe`` / ``df_module.make_column`` + so that dtypes are correct for the current configuration. +* Assert with ``df_module.assert_frame_equal`` / ``df_module.assert_column_equal`` + rather than hand-rolling equality checks. These helpers understand + backend-specific equality semantics (e.g. null handling). +* When you need to check a dtype, use ``df_module.dtypes["float64"]`` rather + than hard-coding ``np.float64``; the correct value depends on the + configuration. + +Using the ``dtypes`` dict +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Suppose you are testing a function that should return a float32 column: + +.. code-block:: python + + def test_returns_float32(df_module): + col = df_module.make_column("x", [1, 2, 3]) + result = sbd.to_float32(col) + assert sbd.dtype(result) == df_module.dtypes["float32"] + + +Defining example dataframes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Depending on the test, it is normally better to define dataframes that are tailored +for the desired test result. ``df_module.make_dataframe`` generates a dataframe for +each module starting from a python dictionary: + +.. code-block:: python + + def test_column_names(df_module): + df = df_module.make_dataframe({"a": [1, 2, 3], "b": [4, 5, 6]}) + names = sbd.column_names(df) + assert "a" in names + assert "b" in names + +If multiple types are required for a given test, then ``example_dataframe`` can +be used to avoid boilerplate: + +.. code-block:: python + + def test_column_names(df_module): + df = df_module.example_dataframe + names = sbd.column_names(df) + assert "float-col" in names + assert "datetime-col" in names + + +Related fixtures +----------------- + +Several narrower fixtures complement ``df_module``. + +``pd_module`` +~~~~~~~~~~~~~ + +Always the ``"pandas-numpy-dtypes"`` configuration. Use when you need to test +pandas-specific behaviour that is not part of the cross-backend API, or when +the test only makes sense for pandas. + +.. code-block:: python + + def test_pandas_index_is_reset(pd_module): + df = pd_module.make_dataframe({"a": [1, 2, 3]}) + df.index = [10, 20, 30] + result = sbd.reset_index(df) + assert list(result.index) == [0, 1, 2] + +``pl_module`` +~~~~~~~~~~~~~ + +The polars configuration. If polars is not installed, the test is +automatically skipped with ``pytest.skip``. Use for polars-specific +behaviour. + +.. code-block:: python + + def test_lazyframe_is_rejected(pl_module): + lazy = pl_module.empty_lazyframe + with pytest.raises(TypeError, match="LazyFrames are not yet supported"): + sbd.shape(lazy) + +``all_dataframe_modules`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Returns the full ``dict`` mapping configuration name to namespace. Use when +you need to iterate over all configurations programmatically inside a single +test body rather than through pytest parametrisation. + +``use_fit_transform`` +~~~~~~~~~~~~~~~~~~~~~ + +A boolean fixture parametrised as ``[False, True]``. Use it to run the same +test through both ``fit`` + ``transform`` and ``fit_transform`` without +duplicating the test body: + +.. code-block:: python + + def test_encoder(df_module, use_fit_transform): + enc = MyEncoder() + if use_fit_transform: + result = enc.fit_transform(df_module.example_dataframe) + else: + result = enc.fit(df_module.example_dataframe).transform( + df_module.example_dataframe + ) + ... + + +Polars-specific considerations +------------------------------- + +LazyFrames +~~~~~~~~~~ + +The ``df_module`` fixture provides an ``empty_lazyframe`` attribute only for +the polars configuration. Most skrub functions expect an *eager* DataFrame; +passing a LazyFrame raises a ``TypeError`` with a message telling the caller +to call ``.collect()``. Test this behaviour explicitly if your function could +receive a LazyFrame. + +The ``skip_polars_installed_without_pyarrow`` mark +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some polars operations (some date/time conversions, functions that involve the +computation of column associations) require pyarrow. +A mark is available to skip those tests when polars is installed but pyarrow +is not: + +.. code-block:: python + + from skrub.conftest import skip_polars_installed_without_pyarrow + + @skip_polars_installed_without_pyarrow + def test_datetime_conversion(df_module): + ... + +Apply this mark to tests that call polars functionality backed by pyarrow. + +Where tests live +----------------- + +Tests for transformers and their functions live in their respective test file: +the code of the ``DatetimeEncoder`` is in ``skrub/_datetime_encoder.py``, while +its tests are in ``skrub/tests/test_datetime_encoder.py``. + +Each submodule contains both the code and its tests. For example, the code for the +dataframe API is in ``skrub/_dataframe``, while the relative tests are in +``skrub/_dataframe/tests``. All tests can request ``df_module``: the fixture is +visible to the entire ``skrub/`` test tree because it is defined in +``skrub/conftest.py``.