Update rclone setup for Docker (#803)

* Update rclone setup for Docker

* Added explanation of placeholders

* Update README

* Added note that MinIO is the default bucket to use

Author: TaperChipmunk32

.devcontainer/Dockerfile

@@ -39,10 +39,6 @@ RUN poetry config virtualenvs.create true && \
     poetry config virtualenvs.in-project true
 # Clean up
 RUN rm -rf /var/lib/apt/lists/*
-# Set up the MinIO/Backblaze bucket
-RUN mkdir -p ~/M
-RUN mkdir -p ~/B
-RUN mkdir -p ~/.config/rclone
 # Set environment variables
 ENV CLEARML_API_HOST="https://api.sil.hosted.allegro.ai"
 ENV EFLOMAL_PATH=/workspaces/silnlp/.venv/lib/python3.10/site-packages/eflomal/bin

.devcontainer/devcontainer.json

@@ -81,7 +81,7 @@
 			]
 		}
 	},
-	"postStartCommand": "poetry install && sh /workspaces/silnlp/.devcontainer/update_hosts.sh && sh /workspaces/silnlp/minio_bucket_setup.sh"
+	"postStartCommand": "poetry install && sh /workspaces/silnlp/.devcontainer/update_hosts.sh && sh /workspaces/silnlp/rclone_setup.sh minio"
 	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
 	// "remoteUser": "root"
 }

README.md

@@ -23,6 +23,24 @@ These are the main requirements for the SILNLP code to run on a local machine. S
 | Environment variables | To tell SILNLP where to find the data, etc.                       |
 
 ## Environment Setup
+Create file for environment variables
+   
+   Create a text file with the following content and edit as necessary:
+   ```
+   CLEARML_API_HOST="https://api.sil.hosted.allegro.ai"
+   CLEARML_API_ACCESS_KEY=xxxxxxx
+   CLEARML_API_SECRET_KEY=xxxxxxx
+   MINIO_ENDPOINT_URL=https://truenas.psonet.languagetechnology.org:9000
+   MINIO_ACCESS_KEY=xxxxxxxxx
+   MINIO_SECRET_KEY=xxxxxxx
+   B2_ENDPOINT_URL=https://s3.us-east-005.backblazeb2.com
+   B2_KEY_ID=xxxxxxxx
+   B2_APPLICATION_KEY=xxxxxxxx
+   ```
+   * Include SIL_NLP_DATA_PATH="/silnlp" if you are not using MinIO or B2 and will be storing files locally.
+   * If you do not intend to use SILNLP with ClearML, MinIO, and/or B2, you can leave out the respective variables. If you need to generate ClearML credentials, see [ClearML setup](clear_ml_setup.md).
+   * Note that this does not give you direct access to a MinIO or B2 bucket from within the Docker container, it only allows you to run scripts referencing files in the bucket.
+
 ### Option 1: Docker
 1. If using a local GPU, install the corresponding [NVIDIA driver](https://www.nvidia.com/download/index.aspx)
 
@@ -49,43 +67,46 @@ These are the main requirements for the SILNLP code to run on a local machine. S
 
    If you're using a local GPU, then in a terminal, run:
    ```
-   docker create -it --gpus all --name silnlp ghcr.io/sillsdev/silnlp:latest
+   docker create -it --gpus all --name silnlp --device /dev/fuse --cap-add SYS_ADMIN --env-file path/to/env-vars-file --security-opt apparmor=path/to/docker-apparmor ghcr.io/sillsdev/silnlp:latest
    ```
    Otherwise, run:
    ```
-   docker create -it --name silnlp ghcr.io/sillsdev/silnlp:latest
+   docker create -it --name silnlp --device /dev/fuse --cap-add SYS_ADMIN --env-file path/to/env-vars-file --security-opt apparmor=path/to/docker-apparmor ghcr.io/sillsdev/silnlp:latest
    ```
-   A docker container should be created. You should be able to see a container named 'silnlp' on the Containers page of Docker Desktop.
+   If you do not intend to use SILNLP with ClearML, MinIO, and/or B2, you can exclude the --device, --cap-add, and --security-opt flags.
 
-5. Create file for environment variables
+   You will need to replace the placehoders "path/to/env-vars-file" and "path/to/docker-apparmor" with the respective real paths.
+   * The env-vars-file is the file you created at the beginning of the Environment Setup section.
+   * The docker-apparmor file is in the silnlp repo. You do not need to clone the entire repo, just download the docker-apparmor file.
 
-   Create a text file with the following content and edit as necessary:
-   ```
-   CLEARML_API_HOST="https://api.sil.hosted.allegro.ai"
-   CLEARML_API_ACCESS_KEY=xxxxxxx
-   CLEARML_API_SECRET_KEY=xxxxxxx
-   MINIO_ENDPOINT_URL=https://truenas.psonet.languagetechnology.org:9000
-   MINIO_ACCESS_KEY=xxxxxxxxx
-   MINIO_SECRET_KEY=xxxxxxx
-   B2_ENDPOINT_URL=https://s3.us-east-005.backblazeb2.com
-   B2_KEY_ID=xxxxxxxx
-   B2_APPLICATION_KEY=xxxxxxxx
-   ```
-   * Include SIL_NLP_DATA_PATH="/silnlp" if you are not using MinIO or B2 and will be storing files locally.
-   * If you do not intend to use SILNLP with ClearML, MinIO, and/or B2, you can leave out the respective variables. If you need to generate ClearML credentials, see [ClearML setup](clear_ml_setup.md).
-   * Note that this does not give you direct access to a MinIO or B2 bucket from within the Docker container, it only allows you to run scripts referencing files in the bucket.
+   A docker container should be created. You should be able to see a container named 'silnlp' on the Containers page of Docker Desktop.
 
 6. Start container
 
    In a terminal, run:
    ```
       docker start silnlp
-      docker exec -it --env-file path/to/env_vars_file silnlp bash
+      docker exec -it silnlp bash
    ```
 
    * After this step, the terminal should change to say `root@xxxxx:~/silnlp#`, where `xxxxx` is a string of letters and numbers, instead of your current working directory. This is the command line for the docker container, and you're able to run SILNLP scripts from here.
    * To leave the container, run `exit`, and to stop it, run `docker stop silnlp`. It can be started again by repeating step 6. Stopping the container will not erase any changes made in the container environment, but removing  it will.
 
+7. (Optional) Mount the rclone bucket
+
+   While in the /root/silnlp directory (the default on startup), run the following command:
+
+   If you are using rclone to mount MinIO (This is the default option):
+   ```
+      ./rclone_setup.sh minio
+   ```
+   If you are using rclone to mount Backblaze (Only used as a backup option):
+   ```
+      ./rclone_setup.sh backblaze
+   ```
+
+   This will mount the specified bucket within the docker container.
+
 ### Option 2: Conda
 1. If using a local GPU, install the corresponding [NVIDIA driver](https://www.nvidia.com/download/index.aspx)
 
@@ -132,24 +153,7 @@ These are the main requirements for the SILNLP code to run on a local machine. S
    poetry install
    ```
 
-10. If using ClearML, MinIO, and/or B2, set the following environment variables:
-   ```
-   CLEARML_API_HOST="https://api.sil.hosted.allegro.ai"
-   CLEARML_API_ACCESS_KEY=xxxxxxx
-   CLEARML_API_SECRET_KEY=xxxxxxx
-   MINIO_ENDPOINT_URL=https://truenas.psonet.languagetechnology.org:9000
-   MINIO_ACCESS_KEY=xxxxxxxxx
-   MINIO_SECRET_KEY=xxxxxxx
-   B2_ENDPOINT_URL=https://s3.us-east-005.backblazeb2.com
-   B2_KEY_ID=xxxxxxxx
-   B2_APPLICATION_KEY=xxxxxxxx
-   ```
-   * Include SIL_NLP_DATA_PATH="/silnlp" if you are not using MinIO or B2 and will be storing files locally.
-   * If you need to generate ClearML credentials, see [ClearML setup](clear_ml_setup.md).
-   * Note that this does not give you direct access to a MinIO or B2 bucket from within the Docker container, it only allows you to run scripts referencing files in the bucket.
-   * For instructions on how to permanently set up environment variables for your operating system, see the corresponding section under the Development Environment Setup header below.
-
-11. If using MinIO or B2, you will need to set up rclone:
+10. If using MinIO or B2, you will need to set up rclone:
    * Mount the bucket to your filesystem following the instructions under [Install and Configure Rclone](https://github.com/sillsdev/silnlp/blob/master/bucket_setup.md#install-and-configure-rclone).
 
 ## Development Environment Setup

backblaze_bucket_setup.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+apt-get install --no-install-recommends -y fuse3 rclone
+mkdir -p /root/.config/rclone
+cp scripts/rclone/rclone.conf /root/.config/rclone/
+BUCKET_TYPE=$1
+if [ "$BUCKET_TYPE" = "minio" ]; then
+    export SIL_NLP_DATA_PATH="/root/M"
+    mkdir -p /root/M
+    sed -i -e "s#access_key_id = x*#access_key_id = $MINIO_ACCESS_KEY#" /root/.config/rclone/rclone.conf
+    sed -i -e "s#secret_access_key = x*#secret_access_key = $MINIO_SECRET_KEY#" /root/.config/rclone/rclone.conf
+
+    echo "Mounting MinIO bucket..."
+    rclone mount --daemon --log-file=rclone_log.txt --log-level=DEBUG  --vfs-cache-mode full --use-server-modtime miniosilnlp:nlp-research ~/M
+    echo "Done"
+elif [ "$BUCKET_TYPE" = "backblaze" ]; then
+    export SIL_NLP_DATA_PATH="/root/B"
+    mkdir -p /root/B
+    sed -i -e "s#account = x*#account = $B2_KEY_ID#" /root/.config/rclone/rclone.conf
+    sed -i -e "s#key = x*#key = $B2_APPLICATION_KEY#" /root/.config/rclone/rclone.conf
+
+    echo "Mounting Backblaze bucket..."
+    rclone mount --daemon --log-file=rclone_log.txt --log-level=DEBUG  --vfs-cache-mode full --use-server-modtime b2silnlp:silnlp ~/B
+    echo "Done"
+fi