Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 38 additions & 30 deletions containers/2_ApplicationSpecific/conda/README.md
Original file line number Diff line number Diff line change
@@ -1,69 +1,77 @@
# Example conda container

CCR does not support running Anaconda on its HPC systems natively for [these reasons](https://docs.ccr.buffalo.edu/en/latest/software/modules/#anaconda-python). We realize many applications are only available via conda so, as an alternative, users can utilize conda in an Apptainer container. Utilizing a container for conda eliminates many of the problems it creates on our systems. This is not a perfect solution though and does require some technical expertise on your part as the CCR user. We provide this information as an example for installing and creating a conda environment. You can modify these files to install the conda packages you require. Any issues you may encounter will need to be worked out on your own.
CCR does not support running Anaconda on its HPC systems natively for [these reasons](https://docs.ccr.buffalo.edu/en/latest/software/modules/#anaconda-python). We realize many applications are only available via conda so, as an alternative, users can utilize conda in an Apptainer container. Utilizing a container for conda eliminates many of the problems it creates on our systems.

## Building the container
This is not a perfect solution though and does require some technical expertise on your part as the CCR user. We provide this information as an example for installing and creating a conda environment. You can modify these files to install the conda packages you require. Any issues you may encounter will need to be worked out on your own.

Please refer to CCR's [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/) for information on building and using Apptainer. This example provides an Apptainer definition file (`conda.def`) which lists a number of packages and installation steps so Apptainer can install these in an Ubuntu container environment. This example also uses an environment file (`environment.yml`) to specify which conda packages to install during the build. For most use cases, you will only need to modify the `environment.yml` file where you'll add additional Python and conda packages to install in your environment. This example utilizes the [conda-forge environment](https://conda-forge.org/) to align with the most recent appropriate use license guidance from Anaconda.
## Building the container

1. Start an interactive job
This example provides an Apptainer definition file (`conda.def`) which lists a number of packages and installation steps so Apptainer can install these in an Ubuntu container environment. This example also uses an environment file (`environment.yml`) to specify which conda packages to install during the build.

Apptainer is not available on the CCR login nodes and the compile nodes may not provide enough resources for you to build a container. We recommend requesting an interactive job on a compute node to conduct this build process. This will allow you to test your build after completion as well. Please refer to our documentation on [running jobs](https://docs.ccr.buffalo.edu/en/latest/hpc/jobs/#interactive-job-submission) for more information. This is provide as an example only and not all users will have access to the resources in this example:
For most use cases, you will only need to modify the `environment.yml` file where you'll add additional Python and conda packages to install in your environment. This example utilizes the [conda-forge environment](https://conda-forge.org/) to align with the most recent appropriate use license guidance from Anaconda.

```
$ salloc --cluster=ub-hpc --partition=general-compute --qos=general-compute --mem=32GB --time=02:00:00
salloc: Pending job allocation 19319338
salloc: job 19319338 queued and waiting for resources
salloc: job 19319338 has been allocated resources
salloc: Granted job allocation 19319338
salloc: Nodes cpn-h23-04 are ready for job
CCRusername@cpn-h23-04:~$

```
The build process follows the same steps detailed in the [introductory container example](../../0_Introductory/README.md#pulling-the-container), which you can use as a guide. Please refer to CCR's [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/) for more information on using Apptainer.

2. Navigate to your build directory & set a temp directory for cache
Log into a compute node, navigate to your project directory, and set a temporary directory for cache there.

You should now be on the compute node allocated to you. In this example we're using our project directory for our build directory. Ensure you've placed your `conda.def` and `environment.yml` file in your build directory
Once ready, build the container:

```
CCRusername@cpn-h23-04:~$ cd /projects/academic/[YourGroupName]/[CCRusername]
CCRusername@cpn-h23-04:~$ mkdir cache
CCRusername@cpn-h23-04:~$ export APPTAINER_CACHEDIR=/projects/academic/[YourGroupName]/[CCRusername]/cache

export APPTAINER_CACHEDIR=/projects/academic/[YourGroupName]/[CCRUsername]/cache
apptainer build conda-$(arch).sif conda.def
```

3. Build your container
Sample output:

```
CCRusername@cpn-h23-04:~$ apptainer build conda-$(arch).sif conda.def
...
...
INFO: Adding environment to container
INFO: Creating SIF file...
INFO: Build complete: conda.sif
```

NOTE: Whenever you modify the `environment.yml` file to add more Python packages to install via conda or pip, you will need to rebuild your container.
> [!WARNING]
> Whenever you modify the `environment.yml` file to add more Python packages to install via conda or pip, you will need to rebuild your container.

## Running the container

For our example, we have no additional conda packages installed. However, we can test this container by running Python:
Ensure you are on a compute node and that you have copied the `conda.def` and `environment.yml` files to your build directory.

For our example, we have no additional conda packages installed. However, we can test this container by running Python.

Run the following command:
```
apptainer exec conda-$(arch).sif python
```

Sample output:
```
CCRusername@cpn-h23-04:~$ apptainer exec conda-$(arch).sif python
Python 3.10.16 | packaged by conda-forge | (main, Apr 8 2025, 20:53:32) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

Alternatively, you can first get shell access into the container and then run Python, as shown here:
Alternatively, you can first get shell access into the container and then run Python.

Get shell access:
```
apptainer shell conda-$(arch).sif
```

Once in the container, run:
```
python
```

Sample output:
```
CCRusername@cpn-h23-04:~$ apptainer shell conda-$(arch).sif
Apptainer> python
Python 3.10.16 | packaged by conda-forge | (main, Apr 8 2025, 20:53:32) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

Please refer to the CCR [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/) for more info on accessing project directories, using GPUs, and other important topics.
## Additional Information

- The [Placeholders](../../../README.md#placeholders) section lists the available options for each placeholder used in the example scripts.
- For more info on accessing shared project and global scratch directories, resource options, and other important container topics, please refer to the CCR [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/)
36 changes: 28 additions & 8 deletions containers/2_ApplicationSpecific/juicer/README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,28 @@
# Example Juicer container
This example goes over how to use Juicer at CCR. The most straightforward way would be to use a containerized version of Juicer. If you are unfamiliar with containerization, please go through [CCR's documentation on the topic](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/) before proceeding with this example. You can find instructions to pull Juicer containers on [GitHub](https://github.com/ArimaGenomics/Juicer_pipeline_containers), however we have some slightly modified guidance on pulling these containers.

## Pulling container
Juicer is a pipeline for generating and analyzing Hi-C maps. For more information and usage details, visit the [official Juicer GitHub repository](https://github.com/aidenlab/juicer).

Instead of using the pre-built `Singularity` image, CCR recommends you pull the available `docker` image on CCR's systems. The commands to do so are:
## Pulling the container

The Juicer software can be pulled from Docker Hub to the CCR's HPC environment using Apptainer. This process follows the same steps detailed in the [introductory container example](../../0_Introductory/README.md#pulling-the-container), which you can use as a guide. Please refer to CCR's [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/) for more information on using Apptainer.

Log into a compute node, navigate to your project directory, and set a temporary directory for cache there.

Once ready, pull the Juicer container from Docker Hub:

```
export APPTAINER_CACHEDIR=/projects/academic/[YourGroupName]/[CCRusername]/cache
apptainer pull juicer.sif docker://arimaxiang/arima_juicer
apptainer pull juicer.sif docker://arimaxiang/arima_juicer:1.6
```

## Running container
After the pull completes, the Apptainer image will be saved as `juicer.sif` in your current working directory.

> [!NOTE]
> Juicer containers are also listed on [GitHub](https://github.com/ArimaGenomics/Juicer_pipeline_containers), but CCR recommends using the pulling instructions shown here for the best compatibility with our environment.

With the container image pulled down, you can run the Juicer software with the instructions from the GitHub repository. You can also find more detailed instructions for how to use Juicer in this [GitHub Repository](https://github.com/aidenlab/juicer/wiki/Usage).
## Running the container

With the container image pulled, you can run the Juicer software with the instructions from the GitHub repository. You can also find more detailed instructions for how to use Juicer in this [GitHub Repository](https://github.com/aidenlab/juicer/wiki/Usage).

First, you make a directory to store your `fastq` files and move your files there:
```
Expand All @@ -28,7 +38,17 @@ export INPUT_reference="/projects/academic/[YourGroupName]/[CCRusername]/referen

Finally, you'll run `juicer.sh` with the following command. You should modify the reference files as necessary for your use case.
```
apptainer exec -B $OUTPUT_DIR -B $INPUT_reference juicer.sif juicer.sh -d $OUTPUT_DIR -D /opt/ -p $INPUT_reference/hg38.chrom.sizes -y $INPUT_reference/hg38_GATC_GANTC.txt -z $INPUT_reference/hg38.fa -t 20 &> $OUTPUT_DIR"/juicer.log"
apptainer exec -B $OUTPUT_DIR -B $INPUT_reference juicer.sif juicer.sh \
-d $OUTPUT_DIR -D /opt/ \
-p $INPUT_reference/hg38.chrom.sizes \
-y $INPUT_reference/hg38_GATC_GANTC.txt \
-z $INPUT_reference/hg38.fa \
-t 20 &> $OUTPUT_DIR"/juicer.log"
```

If you need to use `HiCCUP` with a GPU when using Juicer, you will need to add the `--nv` flag to the `apptainer exec` command. More instructions on using containers with GPUs can be found in [CCR's documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/#gpu-enabled-containers-with-apptainer).
If you need to use `HiCCUP` with a GPU when using Juicer, you will need to add the `--nv` flag to the `apptainer exec` command. More instructions on using containers with GPUs can be found in [CCR's documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/#gpu-enabled-containers-with-apptainer).

## Additional Information

- The [Placeholders](../../../README.md#placeholders) section lists the available options for each placeholder used in the example scripts.
- For more info on accessing shared project and global scratch directories, resource options, and other important container topics, please refer to the CCR [container documentation](https://docs.ccr.buffalo.edu/en/latest/howto/containerization/)