TAO Toolkit is a Python package hosted on the NVIDIA Python Package Index. It interacts with lower-level TAO dockers available from the NVIDIA GPU Accelerated Container Registry (NGC). The TAO containers come pre-installed with all dependencies required for training. The output of the TAO workflow is a trained model that can be deployed for inference on NVIDIA devices using DeepStream, TensorRT and Triton.
This repository contains the required implementation to facilitate data annotation, augmentation, labeling and analytics. These routines are packaged as part of the TAO Toolkit Data-services container in the Toolkit package.
As soon as the repository is cloned, run the envsetup.sh file to check
if the build environment has the necessary dependencies, and the required
environment variables are set.
source scripts/envsetup.shWe recommend adding this command to your local ~/.bashrc file, so that every new terminal instance receives this.
- 8 GB system RAM
- 4 GB of GPU RAM
- 8 core CPU
- 1 NVIDIA GPU
- 100 GB of SSD space
- 32 GB system RAM
- 32 GB of GPU RAM
- 8 core CPU
- 1 NVIDIA GPU
- 100 GB of SSD space
| Software | Version |
|---|---|
| Ubuntu LTS | >=18.04 |
| python | >=3.8.x |
| docker-ce | >19.03.5 |
| docker-API | 1.40 |
nvidia-container-toolkit |
>1.3.0-1 |
| nvidia-container-runtime | 3.4.0-1 |
| nvidia-docker2 | 2.5.0-1 |
| nvidia-driver | >525.85 |
| python-pip | >21.06 |
In order to maintain a uniform development environment across all users, TAO Toolkit provides a base environment docker that has been built and uploaded to NGC for the developers. For instantiating the docker, simply run the tao_ds CLI. The usage for the command line launcher is mentioned below.
usage: tao_ds [-h] [--gpus GPUS] [--volume VOLUME] [--env ENV] [--mounts_file MOUNTS_FILE] [--shm_size SHM_SIZE] [--run_as_user] [--tag TAG] [--ulimit ULIMIT]
Tool to run the TAO Toolkit Data-services container.
optional arguments:
-h, --help show this help message and exit
--gpus GPUS Comma separated GPU indices to be exposed to the docker.
--volume VOLUME Volumes to bind.
--env ENV Environment variables to bind.
--mounts_file MOUNTS_FILE
Path to the mounts file.
--shm_size SHM_SIZE Shared memory size for docker
--run_as_user Flag to run as user
--tag TAG The tag value for the local dev docker.
--ulimit ULIMIT Docker ulimits for the host machine.
A sample command to instantiate an interactive session in the base development docker is mentioned below.
tao_ds --gpus all --volume /path/to/data/on/host:/path/to/data/on/container --volume /path/to/results/on/host:/path/to/results/in/containerThere will be situations where developers would be required to update the third party dependancies to newer versions, or upgrade CUDA etc. In such a case, please follow the steps below:
The base dev docker is defined in $NV_TAO_DS_TOP/docker/Dockerfile. The python packages required for the TAO dev is defined in $NV_TAO_DS_TOP/docker/requirements-pip.txt. Once you have made the required change, please update the base docker using the build script in the same directory.
git submodule update --init --recursive
git submodule foreach git pull origin main
cd $NV_TAO_DS_TOP/docker
./build.sh --buildThe build script now supports cross-platform builds for x86_64 and ARM64 architectures. By default, it builds for your host architecture, but you can specify the target platform:
# Build for x86_64/AMD64 (default on x86_64 hosts)
./build.sh --build --x86
# Build for ARM64 (for Jetson/ARM devices)
./build.sh --build --arm
# Build for both platforms (requires --push flag)
./build.sh --build --multiplatform --pushFor more build options, use the help flag:
./build.sh --helpThe build script tags the newly built base docker with the username of the account in the user's local machine. Therefore, the developers may tests their new docker by using the tao_ds command with the --tag option.
tao_ds --tag $USER -- script argsOnce you are sufficiently confident about the newly built base docker, please do the following
-
Push the newly built base docker to the registry
# For single platform (x86 or arm) bash $NV_TAO_DS_TOP/docker/build.sh --build --push --x86 # For multi-platform (both x86 and arm) bash $NV_TAO_DS_TOP/docker/build.sh --build --push --multiplatform
-
The above step produces a digest file associated with the docker. This is a unique identifier for the docker. So please note this, and update all references of the old digest in the repository with the new digest. The manifest file at
$NV_TAO_DS_TOP/docker/manifest.jsonnow contains platform-specific digests for both x86 and ARM architectures. Update the appropriate digest(s) based on which platform(s) you built.
Push you final updated changes to the repository so that other developers can leverage and sync with the new dev environment.
Please note that if for some reason you would like to force build the docker without using a cache from the previous docker, you may do so by using the --force option.
bash $NV_TAO_DS_TOP/docker/build.sh --build --push --force --x86The TAO container is built on top of the TAO Data-services base dev container, by building a python wheel for the nvidia_tao_ds module in this repository and installing the wheel in the Dockerfile defined in release/docker/Dockerfile. The whole build process is captured in a single shell script which may be run as follows:
source scripts/envsetup.sh
cd $NV_TAO_DS_TOP/release/docker
./deploy.sh --build --wheelIn order to build a new docker, please edit the deploy.sh file in $NV_TAO_DS_TOP/release/docker to update the patch version and re-run the steps above.
TAO Toolkit Data-services is not accepting contributions as part of the TAO 5.0 release, but will be open in the future.
This project is licensed under the Apache-2.0 License.