Explore use of Python DataFrame API instead of pandas

[Pennycook]

Feature/behavior summary

There are efforts underway to define a standard API for DataFrames, here: https://data-apis.org/dataframe-api/draft/index.html

Rewriting the P3 Analysis Library API to accept standard DataFrames instead of pandas.DataFrame objects would potentially enable us to ingest data from more sources.

Request attributes

• Would this be a refactor of existing code?
• Does this proposal require new package dependencies?
• Would this change break backwards compatibility?

Related issues

None

Solution description

• Add dataframe_api_compat as a dependency
• Rewrite user-facing P3 Analysis Library functions using the standard DataFrame API, as demonstrated by https://data-apis.org/dataframe-api-compat/basics/complete_example/
• Rewrite tests to use DataFrame objects from multiple sources (e.g., pandas, polars)

Additional notes

An initial prototype could just focus on one or two functions (say, p3.metrics.application_efficiency and p3.metrics.pp).

[Pennycook]

I had a chance to look at this today, and migrated two functions to the DataFrame API. It was fairly straightforward, and I think in some ways the API is actually cleaner than direct usage of Pandas. For example:

Before

def _cast_to_numeric(df: DataFrame, columns: list[str]) -> DataFrame:
    _require_columns(df, columns)
    result = df.copy(deep=True)
    for column in columns:
        try:
            result[column] = pd.to_numeric(df[column])
        except Exception:
            msg = "Column '%s' must contain only numeric values."
            raise TypeError(msg % (column))
    return result

After

def _cast_to_numeric(df: SupportsDataFrameAPI, columns: list[str]) -> Any:
    _require_columns(df, columns)
    df = df.__dataframe_consortium_standard__(api_version="2023.11-beta")
    ns = df.__dataframe_namespace__()
    for column in columns:
        try:
            if not ns.is_dtype(df.col(column).dtype, kind="numeric"):
                df = df.assign(df.col(column).cast(ns.Float64()))
        except Exception:
            msg = "Column '%s' must contain only numeric values."
            raise TypeError(msg % (column))
    return df.dataframe

Unfortunately, while Googling around for information and API usage examples, I also discovered that efforts to develop this standard have stalled (see data-apis/dataframe-api#363). I'm going to close this issue, now, and we can revisit this idea if/when there is a widely adopted standard for dataframes.

[MarcoGorelli]

I'm going to close this issue, now, and we can revisit this idea if/when there is a widely adopted standard for dataframes.

Just in case it's useful to you, Narwhals has become quite widely adopted, and is used by several major libraries. I started it out of frustration with the dataframe api going nowhere and staying too tied to pandas syntax

In this case, you could write

import narwhals as nw
from narwhals.typing import IntoDataFrameT

def _cast_to_numeric(df: IntoDataFrameT, columns: list[str]) -> IntoDataFrameT:
    _require_columns(df, columns)
    return (
        nw.from_native(df_native)
        .with_columns(nw.selectors.numeric().cast(nw.Float64))
        .to_native()
    )

[Pennycook]

Thanks, @MarcoGorelli. This definitely looks interesting.