Merge main into contribution guide

README.md

@@ -12,6 +12,8 @@ a continual AI pipeline. It was originally part of a bachelor thesis project by
   called "front-end", it also contains a back-end component in the form of a Flask API that communicates with the rest
   of the CHIMP system).
 - **mlflow-tracking:** MLFlow is used to track the different models and log metrics for these models.
+- **minio-datastore:** Minio is used as a centralized datastore for storing and managing datasets.
+- **message-queue:** Redis is used as a message queue to coordinate jobs between the APIs and the workers.
 
 The table below shows which ports are used by the different components. The "Local Dev Port" is the port that is used when a component is run on the host instead of in Docker.
 
@@ -22,17 +24,19 @@ The table below shows which ports are used by the different components. The "Loc
 | ml-frontend      | 5252           | 5252             | 8000                  |
 | mlflow-tracking  | n.a.           | 8999             | 8999                  |
 | minio-datastore  | n.a.           | 9000, 9001 (web) | 9000, 9001            |
+| message-queue    | n.a.           | 6379             | 6379                  |
 
 ```mermaid
 graph RL
 ;
     subgraph Application
         afe[Emotion recognition front-end<br/>- HTML/CSS/JS] --> abe[Emotion recognition back-end<br/>- Python/Flask];
     end
-    abe --> tapi[Training service API<br/>- Python/Flask<br/>- TalosML/Tensorflow];
+    abe --> tapi[Training service API<br/>- Python/Flask];
     abe --> srv[Serving_api service<br/>- Python/Flask];
     subgraph Services
-        tapi --> twork[Training service worker<br/>- Python];
+        tapi --> mq[Redis message queue];
+        mq --> twork[Training service worker<br/>- Python];
         twork --> mlf[Tracking<br/>- MLFlow];
         srv --> mlf;
         mlf --> db[Database<br/>- SQLite];
@@ -69,6 +73,8 @@ run the external services, such as MLFlow and RabbitMQ (e.g. when running the Py
 follows: `docker-compose --profile services up -d`. Please note that the `--profile <PROFILE_NAME>` should come before any
 further commands, such as `up -d`. On some setups, to use the default profile (denoted by an empty string, or ''), it should be explicitly included in the call, for example: `docker-compose --profile '' up -d`.
 
+To monitor and work with the Redis message queue during development, a tool like "Another Redis Desktop Manager" can be used.
+
 ### Local development setup (on host outside of Docker)
 To run the Python/Flask based CHIMP components outside of Docker (for example, when you want to run a component with a debugger attached), you can use the following steps:
 - Run the MLFlow service in Docker using `docker-compose --profile services up -d`

docker-compose.yml

@@ -52,8 +52,8 @@ services:
         environment:
             - LEGACY_PLUGIN_NAME=Emotion Recognition
             - TRACKING_URI=http://monitoring-server:8999
-            - CELERY_BROKER_URL=amqp://message-queue
-            - CELERY_RESULTS_BACKEND=rpc://message-queue
+            - CELERY_BROKER_URL=redis://message-queue:6379
+            - CELERY_RESULT_BACKEND=redis://message-queue:6379
             - DATASTORE_ACCESS_KEY=yZmhrURuUhaeVSUagMRa
             - DATASTORE_SECRET_KEY=cnk0OxGuIgVx4La0prNaWUv7JpriCnxZfq2417ba
             - DATASTORE_URI=datastore:9000
@@ -80,8 +80,8 @@ services:
         environment:
             - LEGACY_PLUGIN_NAME=Emotion Recognition
             - TRACKING_URI=http://monitoring-server:8999
-            - CELERY_BROKER_URL=amqp://message-queue
-            - CELERY_RESULTS_BACKEND=rpc://message-queue
+            - CELERY_BROKER_URL=redis://message-queue:6379
+            - CELERY_RESULT_BACKEND=redis://message-queue:6379
             - DATASTORE_ACCESS_KEY=yZmhrURuUhaeVSUagMRa
             - DATASTORE_SECRET_KEY=cnk0OxGuIgVx4La0prNaWUv7JpriCnxZfq2417ba
             - DATASTORE_URI=datastore:9000
@@ -114,8 +114,8 @@ services:
         environment:
             - LEGACY_PLUGIN_NAME=Emotion Recognition
             - TRACKING_URI=http://monitoring-server:8999
-            - CELERY_BROKER_URL=amqp://message-queue
-            - CELERY_RESULTS_BACKEND=rpc://message-queue
+            - CELERY_BROKER_URL=redis://message-queue:6379
+            - CELERY_RESULT_BACKEND=redis://message-queue:6379
             - DATASTORE_ACCESS_KEY=yZmhrURuUhaeVSUagMRa
             - DATASTORE_SECRET_KEY=cnk0OxGuIgVx4La0prNaWUv7JpriCnxZfq2417ba
             - DATASTORE_URI=datastore:9000
@@ -193,20 +193,13 @@ services:
     # : serves as a message broker for the CHIMP system.
     message-queue:
         restart: unless-stopped
-        image: rabbitmq:3.13-management
+        image: redis:7.4.2
         container_name: message-queue
-        healthcheck:
-            test: rabbitmq-diagnosis check_port_connectivity
-            start_period: 30s
-            interval: 1m
-            timeout: 30s
-            retries: 5
         networks:
             - public-network
             - backend-network
         ports:
-            - "5672:5672"
-            - "15672:15672"
+            - "6379:6379"
         profiles:
             - ''
             - 'gpu'

training/app/config.py

@@ -24,8 +24,8 @@
     basedir, "../datasets"
 )
 
-CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL") or "amqp://localhost"
-CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND") or "rpc://localhost"
+CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL") or "redis://localhost"
+CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND") or "redis://localhost"
 
 DATASTORE_URI = os.environ.get("DATASTORE_URI") or "localhost:9000"
 DATASTORE_ACCESS_KEY = os.environ.get("DATASTORE_ACCESS_KEY") or ""

training/requirements.txt

@@ -3,6 +3,6 @@ flask-cors==4.0.0
 gevent==24.2.1
 gunicorn==21.2.0
 python-dotenv==1.0.1
-celery==5.4.0
+celery[redis]==5.4.0
 mlflow==2.11.3
 minio==7.2.7