Add FGDS (an alternative solution of GDS) copier - #101
Conversation
|
@kk47 This is great! First, I want you to resolve the lint and test issues. The test seems to fail due to unconditional includes of Linux header files in windows backend. Do you want to enable this for vllm? In that case, current entry point is now ParallelLoader. So please update it if you want to influence much broad users. |
Hi @takeshi-yoshimura Thanks again for the guidance! |
25033f8 to
8769bb3
Compare
|
@kk47 |
|
or just run |
|
OK, I‘ll run this and make it passed befor push again. |
Add a new 'fgds' copier for direct GPU storage access,
The FGDS provides a high-performance, easy-to-use alternative to existing
GDS implementations.
FGDS micro benchmark shows that FGDS outperforms GDS by 11%~109%, and POSIX by
40%~143%. When loading the Qwen-32B model in the H100+NVME SSD environment,
fgds copier shows 2.6x improvement than nogds copier.
Key changes:
- Add fastsafetensors/cpp/fgds_ext.cpp: C++ pybind11 extension wrapping libfgds
- Add fastsafetensors/cpp/fgds_ext.pyi: Type stubs for the extension
- Add fastsafetensors/copier/fgds.py: Python FgdsFileCopier implementation
- Register 'fgds' copier in fastsafetensors/copier/__init__.py
- Add fgds_ext build target in setup.py
- Add use_fgds parameter to SafeTensorsFileLoader and fastsafe_open
The FGDS copier follows the same pattern as the existing GDS copier:
- Supports O_DIRECT file access
- Uses async multi-threaded read submission
- Registers GPU memory for direct I/O
- Falls back to nogds copier when libfgds is unavailable
- Properly deregisters memory and cleans up resources
Usage:
loader = SafeTensorsFileLoader(pg, device='cuda:0', use_fgds=True)
# or
with fastsafe_open(files, device='cuda:0', use_fgds=True) as f:
...
Signed-off-by: Kai Kuang <kuangkai@kylinos.cn>
|
@kk47 |
2 participants
Add a new 'fgds' copier for direct GPU storage access, Providing an alternative solution with higher performance and easy-to-use alternative solution for existing GDS implementations.
FGDS micro benchmark shows that FGDS outperforms GDS by 11%~109%, and POSIX by 40%~143%. When loading the Qwen-32B model in the H100+NVME SSD environment, fgds copier shows 2.6x improvement than nogds copier.
Key changes:
The FGDS copier follows the same pattern as the existing GDS copier:
Usage:
loader = SafeTensorsFileLoader(pg, device='cuda:0', use_fgds=True)
# or
with fastsafe_open(files, device='cuda:0', use_fgds=True) as f:
...