OpenDAL's current raw read path models Access::read(path, OpRead { range, .. }) as opening one concrete range stream.
This works naturally for HTTP object stores because each range read is a separate request. However, handle based services such as fs, hdfs, sftp, and monoiofs can reuse an opened handle and execute multiple positioned reads against it. The current core reader planner still calls Access::read for every planned range, so those services repeatedly open and close handles for workloads like Reader::read, Reader::fetch, chunked reads, and seek based async readers.
We should change the raw read contract so Access::read(path, OpRead) returns a reusable raw reader for the path and read condition set, and move byte ranges to methods on that returned reader.
OpenDAL's current raw read path models
Access::read(path, OpRead { range, .. })as opening one concrete range stream.This works naturally for HTTP object stores because each range read is a separate request. However, handle based services such as
fs,hdfs,sftp, andmonoiofscan reuse an opened handle and execute multiple positioned reads against it. The current core reader planner still callsAccess::readfor every planned range, so those services repeatedly open and close handles for workloads likeReader::read,Reader::fetch, chunked reads, and seek based async readers.We should change the raw read contract so
Access::read(path, OpRead)returns a reusable raw reader for the path and read condition set, and move byte ranges to methods on that returned reader.