Skip to content

BUG: Register SQLite adapters only once per process (GH#64337)#66296

Open
q121212 wants to merge 2 commits into
pandas-dev:mainfrom
q121212:fix/64337-sqlite-adapters-once
Open

BUG: Register SQLite adapters only once per process (GH#64337)#66296
q121212 wants to merge 2 commits into
pandas-dev:mainfrom
q121212:fix/64337-sqlite-adapters-once

Conversation

@q121212

@q121212 q121212 commented Jul 11, 2026

Copy link
Copy Markdown

Every call to DataFrame.to_sql() with a SQLite connection creates a new SQLiteTable
instance, whose __init__ unconditionally calls sqlite3.register_adapter() and
sqlite3.register_converter()global mutations. After Python 3.12 removed the
default adapters for datetime types, pandas added its own (commit c9a8f95f).
But re-registering them on every to_sql() call overwrites any custom adapters or
converters the user may have registered.

Side-effect analysis

Without pandas' adapters, to_sql() crashes on Python 3.12+ when inserting
datetime.time, datetime.date, or datetime.datetime columns. The built-in
sqlite3 module no longer provides default adapters for these types.
Therefore, we cannot simply stop registering — we must keep the registration
but make it non-destructive.

Fix

Use a module-level _SQLITE_ADAPTERS_REGISTERED flag to ensure adapters and
converters are registered only once per process lifetime. After the first
to_sql() call, subsequent calls are no-ops and no longer overwrite user
handlers.

File: pandas/io/sql.py

# Module-level state
_SQLITE_ADAPTERS_REGISTERED: bool = False

class SQLiteTable(SQLTable):
    def _register_date_adapters(self) -> None:
        import sqlite3

        global _SQLITE_ADAPTERS_REGISTERED
        if _SQLITE_ADAPTERS_REGISTERED:
            return

        # ... register adapters and converters as before ...

        _SQLITE_ADAPTERS_REGISTERED = True

Added test

File: pandas/tests/io/test_sql.py

test_to_sql_preserves_user_sqlite_adapters — verifies that a custom
adapter registered after the first to_sql() call is NOT overwritten
by subsequent to_sql() calls.

Limitations (documented)

If the user registers custom adapters before their first to_sql() call,
pandas will still overwrite them on that first call. This is the same as
current behavior. A follow-up PR can implement save/restore for full
protection.

Checklist

  • Bug fix
  • Tests pass: pytest pandas/tests/io/test_sql.py -k test_to_sql_preserves
  • No new dependencies
  • Backward compatible — all existing to_sql behavior preserved
  • No performance regression (single boolean check)

q121212 added 2 commits July 12, 2026 02:06
Every call to DataFrame.to_sql() with a SQLite connection created a new
SQLiteTable instance whose __init__ unconditionally called
sqlite3.register_adapter() and sqlite3.register_converter(). These are
global mutations, overwriting any custom adapters/converters the user
may have registered.

Fix: use a module-level _SQLITE_ADAPTERS_REGISTERED flag so adapters
and converters are registered only once per process. After the first
to_sql() call, subsequent calls are no-ops.

This preserves pandas' required adapters (needed because Python 3.12+
removed default datetime adapters) while not stomping on user handlers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant