issue: 4610737 Updating XLIO Readme with threads mode

1. Updating README with threads mode information.
2. Updating README.md with latest changes.

Signed-off-by: Alexander Grissik <[email protected]>

Author: AlexanderGrissik

README

@@ -8,41 +8,33 @@ accelerated network hardware.
 XLIO dynamically links with these applications at run-time, redirect standard socket API calls allowing them to be
 be accelerated without modification.
 
-Build library from source
-========================
-
-Prerequisites:
-1. rdma-core upstream kernel and userspace verbs libraries (libibverbs, libmlx4, libmlx5, librdmacm)
-2. Autoconf, Automake, libtool, unzip, patch, libnl-devel (netlink 1 or 3)
-
-Build:
-1. ./autogen.sh
-2. ./configure --prefix=/usr
-3. make
-4. sudo make install
-
-
-Install library from rpm or debian
-=================================
-
-Installing:
-Install the package as any other rpm or debian package [rpm -i libxlio.X.Y.Z-R.rpm].
-The installation copies the XLIO library to: /usr/lib[64]/libxlio.so
-The XLIO monitoring utility is installed at: /usr/bin/xlio_stats
-The XLIO extra socket API is located at: /usr/include/mellanox/xlio_extra.h
-
-Upgrading:
-Use rpm update procedure: # rpm -U libxlio.X.Y.Z-R.rpm
-You can upgrade by uninstalling (rpm -e) the previously installed package
-before starting to install the new library rpm.
-
-Uninstalling:
-When uninstalling remember to uninstall (rpm -e) the package before you
-uninstall ofed.
+XLIO execution modes
+====================
+XLIO supports two execution modes.
+1 - Run to completion (R2C)
+2 - Worker Threads
 
-Running:
-Set the environment variable LD_PRELOAD to libxlio.so and run your application.
-Example: # LD_PRELOAD=libxlio.so iperf -uc 224.22.22.22 -t 5
+Run to completion
+-----------------
+By default, XLIO works in R2C mode, meaning, the execution context is
+provided to XLIO by the application. In particular, XLIO code such as reading/writing packets,
+polling CQs, handling sockets, etc, is performed as part of POSIX socket calls or Ultra API calls,
+on the same thread which called the API.
+In terms of performance this mode is the preferred one, however,
+this mode requires from the application to work with sockets in an efficient way,
+by avoiding sharing sockets between threads, provide enough execution time to XLIO, use per thread epoll and more.
+In case of listen sockets, each thread should have its own listen socket.
+
+Worker Threads
+--------------
+In this mode XLIO spawns XLIO worker threads. These threads run in the background and perform network operations.
+This mode requires minimal network awareness from the aplication,
+i.e applications may share sockets between threads, use single listening thread,
+use single epoll context or rarely call socket APIs.
+While R2C mode provides best performance, Worker Threads mode provides greater flexibility.
+The number of XLIO worker threads is controlled by the performance.threading.worker_threads parameter.
+XLIO Ultra API is not supported with this mode.
+Please see User Manual for additional details and current limitations.
 
 Configuration Subsystem
 =======================
@@ -986,8 +978,22 @@ automatically to reach a desired rate of interrupts.
 4. Disable CQ moderation with XLIO_CQ_MODERATION_ENABLE=0
 5. Disable Adaptive CQ moderation with XLIO_CQ_AIM_INTERVAL_MSEC=0
 
+Install library from rpm or debian
+=================================
 
+Installing:
+Install the package as any other rpm or debian package [rpm -i libxlio.X.Y.Z-R.rpm].
+The installation copies the XLIO library to: /usr/lib[64]/libxlio.so
+The XLIO monitoring utility is installed at: /usr/bin/xlio_stats
+The XLIO extra socket API is located at: /usr/include/mellanox/xlio_extra.h
+
+Upgrading:
+Use rpm update procedure: # rpm -U libxlio.X.Y.Z-R.rpm
+You can upgrade by uninstalling (rpm -e) the previously installed package
+before starting to install the new library rpm.
 
+Uninstalling:
+When uninstalling remember to uninstall (rpm -e) the package before you uninstall DOCA.
 
 Troubleshooting
 ===============
@@ -1004,7 +1010,6 @@ This warning message means that you are using XLIO with high log level:
 XLIO_TRACELEVEL variable value is set to 4 or more.
 In order to fix it - set XLIO_TRACELEVEL to it's default value: 3
 
-
 * CAP_NET_RAW and root access
 
  XLIO_WARNING: ******************************************************************************

README.md

@@ -1,10 +1,13 @@
 # XLIO - Accelerated IO SW library
 
-The NVIDIA® Accelerated IO (XLIO) SW library boosts the performance of TCP/IP applications based on NGINX (CDN, DoH Etc.) and storage solutions as part of the SPDK. XLIO is a user-space software library that exposes standard socket APIs with kernel-bypass architecture, enabling a hardware-based direct copy between an application’s user-space memory and the network interface.
+The NVIDIA® Accelerated IO (XLIO) SW library boosts the performance of TCP/IP network applications providing high bandwidth and low CPU usage.
+XLIO is a user-space software library which exposes standard POSIX and XLIO Ultra socket APIs with kernel-bypass architecture, enabling a hardware-based direct copy between an application user-space memory and the network interface.
 
-Coupling XLIO with NVIDIA ConnectX®-6 Dx, NVIDIA ConnectX®-7 or NVIDIA BlueField®-2 data processing unit (DPU) acceleration capabilities provides breakthrough performance of Transport Layer Security (TLS) encryption/decryption. It also enables working with features such as HW LRO/TSO and Striding-RQ which increase TCP performance, without application code changes and using a standard socket API.
+Coupling XLIO with Crypto Enabled NVIDIA ConnectX, NVIDIA BlueField data processing unit (DPU) acceleration capabilities, provides a breakthrough performance of Transport Layer Security (TLS) encryption and decryption.
+XLIO is capable of utilizing HW features such as LRO/TSO and Striding-RQ which increase TCP performance, without application code changes for both
+POSIX socket and XLIO Ultra APIs.
 
-Please visit our [documentation site](https://docs.nvidia.com/networking/category/xlio) for more details.
+Please visit our [documentation website](https://docs.nvidia.com/networking/category/xlio) for more details.
 
 <hr>
 
@@ -16,24 +19,28 @@ Please visit our [documentation site](https://docs.nvidia.com/networking/categor
 * [Licenses](#licenses)
 * [Information](#information)
 * [Contribution](#contribution)
-* [Configuration](#configuration)
 
 <hr>
 
 ## Getting Started
 
-### Prerequisits
+### Installing XLIO
 
-#### Driver Stack
+XLIO is available as part of DOCA Software Stack.
+Please visit [DOCA website](https://developer.nvidia.com/networking/doca) for more details.
 
-XLIO can work on top of both MLNX_OFED driver stack that supports Ethernet and on a lighter driver stack, MLNX_EN that supports only Ethernet.
+### Building XLIO
+
+#### Prerequisits
 
-[MLNX_OFED](https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/)
-[MLNX_EN](https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/)
+##### DOCA Software stack
 
-#### DPCP
+XLIO requires NVIDIA DOCA-Host software stack or NVIDIA Bluefiled bundle to be installed on the system.
+Please visit [DOCA website](https://developer.nvidia.com/networking/doca) for more details.
 
-DPCP (Direct Packet Control Plane) is mandatory to enable advanced HW features for supported NVIDIA HW.
+##### DPCP
+
+DPCP (Direct Packet Control Plane) is mandatory to run XLIO.
 Repository: [libdpcp](https://github.com/Mellanox/libdpcp.git)
 
 ```sh
@@ -43,7 +50,11 @@ $ make -j
 $ make install
 ```
 
-### Building XLIO
+##### Tools
+
+Autoconf, Automake, libtool, unzip, patch, libnl-devel (netlink 3)
+
+#### Compiling XLIO
 
 Run the following commands from within the directory at the top of the tree:
 
@@ -55,7 +66,7 @@ $ make install
 ```
 --enable-utls : Enables uTLS HW offload for supported NVIDIA HW.
 
-### Building XLIO using preinstalled dpcp
+#### Compiling XLIO using preinstalled dpcp
 
 ```sh
 $ ./autogen.sh
@@ -64,6 +75,9 @@ $ make -j
 $ make install
 ```
 
+#### Configure
+See more [Options](./docs/configuration.md)
+
 ### Usage Examples
 
 #### Sockperf
@@ -88,26 +102,20 @@ N - Number of Nginx workers.
 * TCP
 * UDP
 
-## Supported CPU Architectures
+## Supported CPU Architecturess
 
 * [x86_64](https://en.wikipedia.org/wiki/X86-64)
 * [Arm](https://www.arm.com/)
 
-## Supported NICs
+## Supported HW
 
-* NVIDIA ConnectX®-6
-* NVIDIA ConnectX®-7
-* NVIDIA BlueField®-2
-* NVIDIA BlueField®-3
+* Please refer to the [User Manual](https://docs.nvidia.com/networking/software/accelerator-software/index.html#xlio) for supported device list.
 
 ## Licenses
 See [LICENSE](./LICENSE) file
 
 ## Information
-See [README](./README) file
+See [README](./README) file for XLIO features and parameters.
 
 ## Contribution
 [Contribution](./docs/contributing.md) guidelines for this project
-
-## Configuration
-[Configuration](./docs/configuration.md) options description