DuckDB read/write integration #220
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomasmarwitz
force-pushed
the
duckdb-backend
branch
from
422bc49 to
7378065
Compare
xhochy
reviewed
Apr 14, 2025
Member
There was a problem hiding this comment.
These functions all look familiar. Are these arrow rewrites of ones that exist as pandas versions?
Contributor
Author
There was a problem hiding this comment.
align_categories is:
-
helper/align_categories is rewrite of:
plateau/plateau/io_components/utils.py
Line 295 in 7190d1b
-
cast_categoricals_to_dictionary is a take on
Line 269 in 7190d1b
-
Just noticed that empty_table_from_schema can be a one-liner conversion
Co-authored-by: Uwe L. Korn <[email protected]>
Remove prints.
thomasmarwitz
force-pushed
the
duckdb-backend
branch
from
aaa2305 to
1cdae26
Compare
Contributor
Author
|
@xhochy do you think we can get this merged today? |
xhochy
reviewed
Jul 18, 2025
Comment on lines
1112
to
-997
| def build_indices(self, columns: Iterable[str]): | ||
| """This builds the indices for this metapartition for the given |
Member
There was a problem hiding this comment.
We should restore the docstring.
| return sorted(secondary_indices) | ||
|
|
||
|
|
||
| def group_table_by_partition_keys(table: pa.Table, partition_on: list[str]): |
Member
There was a problem hiding this comment.
This is the only place where polars is used. Let's figure out how to do it without it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add initial support for duckdb (and pyarrow) s.t. no conversion to
pandas.DataFrameis performed.