Avoid clones in make_array for StructArray and GenericByteViewArray
#9114
Conversation
| fn from(value: ArrayData) -> Self { | ||
| let views = value.buffers()[0].clone(); | ||
| let views = ScalarBuffer::new(views, value.offset(), value.len()); | ||
| let buffers = value.buffers()[1..].to_vec(); |
There was a problem hiding this comment.
this call to_vec() allocates a new Vec which is unecessary
|
|
||
| impl<T: ByteViewType + ?Sized> From<ArrayData> for GenericByteViewArray<T> { | ||
| fn from(value: ArrayData) -> Self { | ||
| let views = value.buffers()[0].clone(); |
There was a problem hiding this comment.
cloneing the buffers is relatively cheap (they are Arcd internally) so avoiding this just makes the code easier to follow, I don't think it will be any significant savings
| make_array(cd.slice(parent_offset, parent_len)) | ||
| } else { | ||
| make_array(cd.clone()) | ||
| make_array(cd) |
There was a problem hiding this comment.
cd.clone() clones the ArrayData (and allocates a new Vec) for each child, recursively, which is unecessary
| .into_iter() | ||
| .map(|cd| { | ||
| if parent_offset != 0 || parent_len != cd.len() { | ||
| make_array(cd.slice(parent_offset, parent_len)) |
There was a problem hiding this comment.
we can probably avoid an additional allocation for sliced arrays by making a version of slice() that consumes self -- like sliced() perhaps 🤔
|
Thanks @mhilton |
| fn from(data: ArrayData) -> Self { | ||
| let (_data_type, len, nulls, offset, mut buffers, _child_data) = data.into_parts(); | ||
| let views = buffers.remove(0); // need to maintain order of remaining buffers | ||
| let buffers = buffers.into(); // convert to Arc |
There was a problem hiding this comment.
| let buffers = buffers.into(); // convert to Arc | |
| let buffers = Arc::from(buffers); |
Is this equivalent?
There was a problem hiding this comment.
I was going to ask the same thing... except for Arc::new. Longer than .into() but shorter than having to comment it.
| fn from(data: ArrayData) -> Self { | ||
| let (_data_type, len, nulls, offset, mut buffers, _child_data) = data.into_parts(); | ||
| let views = buffers.remove(0); // need to maintain order of remaining buffers | ||
| let buffers = buffers.into(); // convert to Arc |
There was a problem hiding this comment.
I was going to ask the same thing... except for Arc::new. Longer than .into() but shorter than having to comment it.
4 participants
Which issue does this PR close?
make_array) #9061ArrayData(speed up reading from Parquet reader) #9058Rationale for this change
The current implementation of
make_arrayfor StructArray and GenericByteViewArray clonesArrayDatawhich allocates a new Vec. This is unnecessary given thatmake_arrayis passed an owned ArrayDataWhat changes are included in this PR?
into_parts)Are these changes tested?
Yes by CI
Are there any user-facing changes?
A few fewer allocations when creating arrays