-
Notifications
You must be signed in to change notification settings - Fork 2
Initial implementation of acomp-dpusm provider #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gcabiddu
wants to merge
3
commits into
intel:main
Choose a base branch
from
gcabiddu:devel
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # SPDX-License-Identifier: GPL-2.0-only | ||
| # Copyright (C) 2025 Intel Corporation | ||
| # These contents may have been developed with support from one or more | ||
| # Intel-operated generative artificial intelligence solutions. | ||
| .* | ||
| *.o | ||
| *.o.* | ||
| *.a | ||
| *.s | ||
| *.ko | ||
| *.so | ||
| *.so.dbg | ||
| *.mod | ||
| *.mod.c | ||
| *.i | ||
| *.lst | ||
| *.symtypes | ||
| *.order | ||
| modules.builtin | ||
| *.elf | ||
| *.bin | ||
| *.gz | ||
| *.bz2 | ||
| *.lzma | ||
| *.xz | ||
| *.lz4 | ||
| *.lzo | ||
| *.patch | ||
| *.patch.conv | ||
| *.gcno | ||
|
|
||
| # | ||
| # Top-level generic files | ||
| # | ||
| /tags | ||
| /TAGS | ||
| /linux | ||
| /vmlinux | ||
| /vmlinuz | ||
| /System.map | ||
| /Module.markers | ||
| Module.symvers | ||
|
|
||
| # stgit generated dirs | ||
| patches-* | ||
|
|
||
| # quilt's files | ||
| patches | ||
| series | ||
|
|
||
| # cscope files | ||
| cscope.* | ||
| ncscope.* | ||
|
|
||
| # gnu global files | ||
| GPATH | ||
| GRTAGS | ||
| GSYMS | ||
| GTAGS | ||
|
|
||
| *.orig | ||
| *~ | ||
| \#*# |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| # Installation Guide for Acompress Provider | ||
|
|
||
| This document provides step-by-step instructions for installing and configuring the Acompress Provider for DPUSM and ZFS. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before you begin, ensure that you have the following: | ||
|
|
||
| - A system with Linux Kernel support for the Crypto API's acomp interface and zlib-deflate (versions 6.3 to 6.6). | ||
| - Administrative access to the system for installation and configuration tasks. | ||
| - Setup top level source directory. | ||
|
|
||
| ```bash | ||
| export HOME_PATH=/devel/zfs | ||
| sudo mkdir -p $HOME_PATH | ||
| sudo chown -R $USER:$USER $HOME_PATH | ||
| ``` | ||
|
|
||
| ## Component 1: DPUSM | ||
|
|
||
| ### Download DPUSM | ||
|
|
||
| 1. Clone the DPUSM repository: | ||
|
|
||
| ```bash | ||
| cd $HOME_PATH | ||
| git clone https://github.com/hpc/dpusm | ||
| cd dpusm | ||
| ``` | ||
|
|
||
| ### Build and Install DPUSM | ||
|
|
||
| 2. Build DPUSM | ||
|
|
||
| ```bash | ||
| make | ||
| ``` | ||
|
|
||
| 3. Load the module: | ||
|
|
||
| ```bash | ||
| sudo insmod dpusm.ko | ||
| ``` | ||
|
|
||
| 4. Verify module is loaded: | ||
|
|
||
| ```bash | ||
| lsmod | grep dpusm | ||
| ``` | ||
|
|
||
| Example output: | ||
|
|
||
| ```bash | ||
| lsmod | grep dpusm | ||
| dpusm 36864 0 | ||
| ``` | ||
|
|
||
| ## Component 2: ZFS with Z.I.A. | ||
|
|
||
| ### Obtain ZFS with Z.I.A. | ||
|
|
||
| To use ZFS with the ZFS Interface for Accelerators (Z.I.A.), you need to obtain the specific branch that includes Z.I.A. support. Follow these steps: | ||
|
|
||
| 1. Clone the repository with the Z.I.A.: | ||
|
|
||
| ```bash | ||
| cd $HOME_PATH | ||
| git clone https://github.com/hpc/zfs.git | ||
| cd zfs | ||
| ``` | ||
|
|
||
| ### Build and Install ZFS | ||
|
|
||
| 1. Build ZFS | ||
|
|
||
| Notes: If DPUSM is installed to another directory, update the following command accordingly. | ||
|
|
||
| ```bash | ||
| ./autogen.sh | ||
| ./configure --with-zia=$HOME_PATH/dpusm | ||
| make -j$(nproc) | ||
| ``` | ||
|
|
||
| 3. Install ZFS: | ||
|
|
||
| ```bash | ||
| sudo make install | ||
| ``` | ||
|
|
||
| 4. Load ZFS modules: | ||
|
|
||
| ```bash | ||
| sudo modprobe zfs | ||
| ``` | ||
|
|
||
| 5. Verify ZFS with Z.I.A was loaded: | ||
|
|
||
| ```bash | ||
| sudo dmesg | grep -E "Z\.I\.A|ZFS" | ||
| ``` | ||
|
|
||
| The output will look like: | ||
|
|
||
| ```bash | ||
| user@server:~/zfs$ sudo dmesg | grep -E "Z\.I\.A|ZFS" | ||
| [59203.606953] Z.I.A. initialized (0000000029342ebc) | ||
| [59204.818383] ZFS: Loaded module v2.3.99-313_g325a5e241, ZFS pool version 5000, ZFS filesystem version 5 | ||
| ``` | ||
|
|
||
| ## Component 3: Acompress Provider | ||
|
|
||
| ### Clone the Acompress Provider Repository | ||
|
|
||
| 1. Clone the Acompress Provider repository: | ||
|
|
||
| ```bash | ||
| cd $HOME_PATH | ||
| git clone https://github.com/intel/acomp-dpusm.git | ||
| cd acomp-dpusm | ||
| ``` | ||
|
|
||
| ### Build and Install Provider: | ||
|
|
||
| 1. Install the Acompress Provider: | ||
|
|
||
| ```bash | ||
| make | ||
| ``` | ||
|
|
||
| 3. Install the Provider: | ||
|
|
||
| ```bash | ||
| sudo insmod acomp-dpusm.ko | ||
| ``` | ||
|
|
||
| 4. Verify provided was loaded: | ||
|
|
||
| ```bash | ||
| sudo dmesg | grep acomp_dpusm | ||
| ``` | ||
|
|
||
| Example output: | ||
|
|
||
| ```bash | ||
| user@server:~/acomp-dpusm$ sudo dmesg | grep acomp_dpusm | ||
| [62503.851101] Provider acomp_dpusm supports GZIP Level 1 Compress | ||
| [62503.851112] Provider acomp_dpusm supports GZIP Level 2 Compress | ||
| [62503.851114] Provider acomp_dpusm supports GZIP Level 3 Compress | ||
| [62503.851116] Provider acomp_dpusm supports GZIP Level 4 Compress | ||
| [62503.851117] Provider acomp_dpusm supports GZIP Level 5 Compress | ||
| [62503.851118] Provider acomp_dpusm supports GZIP Level 6 Compress | ||
| [62503.851119] Provider acomp_dpusm supports GZIP Level 7 Compress | ||
| [62503.851120] Provider acomp_dpusm supports GZIP Level 8 Compress | ||
| [62503.851121] Provider acomp_dpusm supports GZIP Level 9 Compress | ||
| [62503.851122] Provider acomp_dpusm supports GZIP Level 1 Decompress | ||
| [62503.851123] Provider acomp_dpusm supports GZIP Level 2 Decompress | ||
| [62503.851124] Provider acomp_dpusm supports GZIP Level 3 Decompress | ||
| [62503.851125] Provider acomp_dpusm supports GZIP Level 4 Decompress | ||
| [62503.851126] Provider acomp_dpusm supports GZIP Level 5 Decompress | ||
| [62503.851127] Provider acomp_dpusm supports GZIP Level 6 Decompress | ||
| [62503.851128] Provider acomp_dpusm supports GZIP Level 7 Decompress | ||
| [62503.851129] Provider acomp_dpusm supports GZIP Level 8 Decompress | ||
| [62503.851130] Provider acomp_dpusm supports GZIP Level 9 Decompress | ||
| [62503.851131] dpusm_provider_register: DPUSM Provider "acomp_dpusm" (00000000863c55ce) added. Now 1 providers registered. | ||
| ``` | ||
|
|
||
| ## Runtime Configuration | ||
|
|
||
| Ensure the zpools are configured to use the provider and enable compression/decompression. The operations would look like: | ||
|
|
||
| ```bash | ||
| sudo $HOME_PATH/zfs/zpool create -f test_zpool /dev/sda | ||
| sudo $HOME_PATH/zfs/zpool set zia_provider=acomp_dpusm test_zpool | ||
| sudo $HOME_PATH/zfs/zpool zia_compress=on test_zpool | ||
| sudo $HOME_PATH/zfs/zpool zia_decompress=on test_zpool | ||
| sudo $HOME_PATH/zfs/zfs set compression=gzip test_zpool | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # SPDX-License-Identifier: GPL-2.0-only | ||
| # Copyright (C) 2025 Intel Corporation | ||
| # These contents may have been developed with support from one or more | ||
| # Intel-operated generative artificial intelligence solutions. | ||
|
|
||
| # Define the kernel source directory | ||
| KERNELDIR ?= /lib/modules/$(shell uname -r)/build | ||
|
|
||
| # Define the module name | ||
| MODULE_NAME := acomp-dpusm | ||
|
|
||
| # Define the home path for the DPUSM and ZFS source directories | ||
| HOME_PATH ?= /devel/zfs | ||
|
|
||
| # Define the include paths for DPUSM | ||
| DPUSM_INCLUDE := $(HOME_PATH)/dpusm/include | ||
|
|
||
| # Define the include paths for ZFS | ||
| ZFS_INCLUDE := $(HOME_PATH)/zfs/include | ||
|
|
||
| # Define the source files for the module | ||
| SRC_FILES := provider.c compress.c | ||
|
|
||
| # Define the object files for the module | ||
| OBJ_FILES := $(SRC_FILES:.c=.o) | ||
|
|
||
| # Define the location of the module symvers file for DPUSM | ||
| KBUILD_EXTRA_SYMBOLS := $(HOME_PATH)/dpusm/Module.symvers | ||
|
|
||
| ifneq ($(KERNELRELEASE),) | ||
| ccflags-y := -I$(DPUSM_INCLUDE) -I$(ZFS_INCLUDE) | ||
| obj-m := $(MODULE_NAME).o | ||
| $(MODULE_NAME)-y := $(OBJ_FILES) | ||
| else | ||
| PWD := $(shell pwd) | ||
|
|
||
| default: | ||
| $(MAKE) -C $(KERNELDIR) M=$(PWD) W=1 modules | ||
|
|
||
| clean: | ||
| $(MAKE) -C $(KERNELDIR) M=$(PWD) clean | ||
| endif | ||
|
|
||
| .PHONY: clean |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # Acompress Provider for DPUSM | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Introduction](#introduction) | ||
| - [Licensing](#licensing) | ||
| - [Software Requirements](#software-requirements) | ||
| - [Limitations](#limitations) | ||
| - [Installation](#installation) | ||
| - [Legal](#legal) | ||
|
|
||
| ## Introduction | ||
|
|
||
| The Acompress Provider is a plugin designed to integrate with DPUSM (Data Processing Unit Storage Management) and ZFS, utilizing the Z.I.A (ZFS Interface for Accelerators) framework. It enhances data compression capabilities within ZFS environments by leveraging the Crypto API's acomp interface for efficient data processing and storage management. | ||
|
|
||
| <img src="docs/images/stack.png" alt="stack diagram" width="500"/> | ||
|
|
||
|
|
||
| ## Licensing | ||
|
|
||
| This project is licensed under the GNU General Public License v2.0. Please see the `LICENSE` file contained in the top-level folder for details. | ||
|
|
||
| ## Software Requirements | ||
|
|
||
| - Linux Kernel with support for the Crypto API's acomp interface and zlib-deflate (currently covering versions 6.3 to 6.6). | ||
| - ZFS installed and operational configured with support for Z.I.A. | ||
| - DPUSM installed and configured. | ||
| - zpool configured to use this provider and compression offloading enabled | ||
|
|
||
|
|
||
| ## Limitations | ||
|
|
||
| 1. The Acompress Provider currently supports specific compression algorithms available through the Crypto API's acomp interface. | ||
| 2. Ensure compatibility with your existing DPUSM and ZFS setup before integration. | ||
| 3. Performance may vary based on hardware and system configuration. | ||
|
|
||
| ## Installation | ||
|
|
||
| For detailed installation instructions, please refer to the `INSTALL.md` file in this repository. | ||
|
|
||
| ## Legal | ||
|
|
||
| Intel® disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. | ||
|
|
||
| This document contains information on products, services, and/or processes in development. All information provided here is subject to change without notice. Contact your Intel® representative to obtain the latest forecast, schedule, specifications, and roadmaps. | ||
|
|
||
| The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request. | ||
|
|
||
| Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm. | ||
|
|
||
| Intel, the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. | ||
|
|
||
| \*Other names and brands may be claimed as the property of others | ||
|
|
||
| --- | ||
|
|
||
| Thank you for using the Acompress Provider! We hope it enhances your data processing and storage management experience. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation lists kernel versions 6.3–6.6, but the PR description specifies 6.4–6.6; update the version range for consistency.