Skip to content

[Bug] QDP cannot read default LZ4-compressed Feather files #1472

Description

@viiccwen

What

QDP documents .feather as a supported CUDA file format. PyArrow writes Feather V2 files with LZ4 compression by default, but the QDP Arrow dependency is built without IPC compression support. Files produced by the standard PyArrow API therefore fail in both direct encoding and the Rust-backed loader.

Steps to reproduce:

from pathlib import Path

import numpy as np
import pyarrow as pa
import pyarrow.feather as feather
from qumat_qdp import QdpEngine, QuantumDataLoader

path = Path("vectors.feather")
values = np.eye(4, dtype=np.float64)
vectors = pa.FixedSizeListArray.from_arrays(
    pa.array(values.reshape(-1)),
    list_size=4,
)
feather.write_feather(pa.table({"data": vectors}), path)

QdpEngine().encode(path, num_qubits=2, encoding_method="amplitude")

list(
    QuantumDataLoader()
    .backend("rust")
    .qubits(2)
    .batches(1, size=2)
    .source_file(str(path))
)

Both paths fail with an error containing:

Invalid argument error: lz4 IPC decompression requires the lz4 feature

Actual result: Both documented CUDA file APIs reject a Feather V2 file written with PyArrow's default compression.

Expected result: QDP reads Feather files written with PyArrow's default settings through direct encoding and the Rust-backed loader.

This issue is limited to compressed Arrow IPC/Feather input. PyTorch .pt and .pth input is tracked separately because it fails through a different reader path.

Environment

  • OS: Linux
  • Python version: 3.12.12
  • Qumat version: 0.7.0.dev0 (qumat-qdp 0.3.0.dev0)
  • PyArrow version: 25.0.1
  • GPU: NVIDIA GeForce RTX 4090

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions