Skip to content

Commit 1eb2b81

Browse files
zhiltsov-maxMarishka17ivhusDzeranovBobronium
authored
[CVAT-M2] Oracles M2 API (#2094)
* Fix job event handling when no assignments exist * Implement boxes from points task creation * Refactor cloud storage api * Implement job downloading * Estimate max bbox * Little refactoring * Copy some shared code from excor to recor * Refactor task creation and export * Implement boxes from points validation * Integrate SDK updates * Fix escrow access * [Exchange/Recoring oracles] Add GCS support * Remove local dev mocks * Remove some extra changes * Update tests * Update some tests * Remove extra launcher change * Refactor code * Add google-cloud-storage dependency * Fix exception classes * Set env variables in docker-compose.test.yml * [CVAT] Points to boxes task (#1560) * Fix job event handling when no assignments exist * Implement boxes from points task creation * Refactor cloud storage api * Implement job downloading * Estimate max bbox * Little refactoring * Copy some shared code from excor to recor * Refactor task creation and export * Implement boxes from points validation * Integrate SDK updates * Fix escrow access * Remove local dev mocks * Remove some extra changes * Update tests * Update some tests * Remove extra launcher change * Use virtual hosted bucket style * Fix creating CS in CVAT * Fix tests * Add basic implementation for skeletons from boxes * Refactor and fix some errors * Implement job uploading * Implement downloading * Fix extra gt boxes in merged results from excor * Implement skeleton matching * Fix labels in merged annotations * Update project events handling in excor * Improve oks sigma comment * Remove local testing assets * Fix label mapping * Add extra skeleton validations in manifest * Update .env template * Update several comments * Fix failing tests * Update enum name * Add more annotation validations for skeletons * Extend input annotations validation * Fix quality computation, refactor * Unify assignment accuracy checks between different job types * Refactor GT downloading for validation * Add gcs support - fixes (#1) * Update code formatting * Update poetry lock * Refactor * Use dict for gcs file contents * Update class name * Add backward compatibility for data bucket env var * Remove extra changes * Remove extra import * Fix enum name * Update bucket access info parsing * Fix error type in cloud provider parsing * Update tests * Add job annotation mode param into assignment links * Add CD trigger for experimental cvat oracles * Update code formatting * Fix test * Format code * Fix bucket uses * Fix result sending * Remove extra changes * Fix extra code * Remove unused code * Align enum naming convention * Update oracle-webhook endpoints * Update webhooks for m2 * Update imports * Use relative paths in bucket in task creation * Improve error messages * Improve skeleton and bbox validation * Fix escrow manifest downloading * Update tests * Fix tests * Implement general image bans * Fix import * Make default gt ban threshold more strict * Fix comparison for absent points * Fix comparison for omitted points in jobs * Finish escrows with too many unverifiable assignments * Check if an increased healthcheck interval will fix unhealthy container (#1700) * Fix validations for boxes from points task creation * Clean code * disable roi estimation for unreliable cases * Fix manifest parsing * Remove m0 launcher stubs * Remove m0 rep or stubs * Make max assignment time optional in manifest * Make label type fully optional in manifest * Fix test * Enable and fix a disabled m1 task creation test * Fix RecOr tests * Update poetry lock * Rename rejected_jobs to assignments * Update imports * Fix dict traversal * [Dependent][CVAT-M2] Oracles M2 API - endpoints (#1649) * Upgrade deps, upgrade pydantic to v2 and fastapi to 0.109 * Migrate code to pydantic v2 * Add filtering and pagination utilities * Add draft implementation for updated excor endpoints, refactor enum names * Fixes and refactoring * Update endpoint response fields and filters * Implement response field selection * Add new endpoints * Support JWT auth * Update .env template * Move type definition * Refactor pagination * Add page size env var * Add jwt lib project dependency * Fix authorization header integration with swagger * Adopt pagination changes * Fix assignment status filter * Add job status field and filter * Add redis and rate limiting for stats * Update recording update deps, align versions * Remove job_title from /job * Update poetry lock * Refactor some code * Add resign endpoint * Take updated_at from created_at for active assignments * Update register endpoint api * Add qualifications stubs in API * Remove deprecated max_time from manifest * Add qualifications into job responses * Update code formatting * Update deprecated method .dict() uses * Update deprecated parse_obj calls on pydantic models * Update comment * Remove human app signature header from /register * Fix throttling implementation * Add new filters in API * Update fastapi to 0.111.1 * Update email and method name in /register * Add missing chain_id filter in list_assignments * Rename amount to total in stats * Don't allow updating user * [Exchange Oracle] Update lock file * [Exchange Oracle] Mark @field_validator as a classmethod decorator * [Exchange Oracle] Fix Ruff violations * [Exchange Oracle] Remove deprecated param * [Recording Oracle] Fix ruff violations * [Recording Oracle] Update poetry.lock * Add redis auth * Refactor some code, allow empty wallet_address in JWT if role is human app * Fix CI checks * Update dev env configs * [Exchange oracle] return 401 instead of 403 when there is no authorization header * [Exchange oracle] list assignments: make filter params optional && resign assignment: make assignment_id a request body param * [Exchange oracle] Update api tests * [Exchange oracle] Update integraton/services/test_exchange.py * Use renamed enum members * t * Update poetry.lock * t * Small fixes * Import StrEnum from strenum directly * [Exchange oracle] Fix pagination * [Exchange oracle] GET /job fields param: support json array * [Exchange oracle] more tests * [Exchange oracle] Fix sorting * [Exchange oracle] RecordingOracleEvent_SubmissionRejected::RejectedAssignmentInfo::task_id -> assignment_id * [Recording oracle] RecordingOracleEvent_SubmissionRejected::RejectedAssignmentInfo::task_id -> assignment_id * Remove todo * [Exchange oracle] Fix missed updated_at in GET /job response * [Exchange oracle] Add description and examples for JobsFilter::fields * Update packages/examples/cvat/exchange-oracle/src/endpoints/filtering.py * Remove extra file after merge * Improve comment * Refactor some code * Improve comment * Improve comment * Remove outdated comment * Fix formatting * Apply comments * Fix invalid updated_after filtering for assignments * Fix invalid assignment status filter and refactor job status filter * Make docker images lighter for oracles * Refactor some code * Extend list assignments test * Fix linter errors * Touch parents updated_at when updating a child && update tests * Update packages/examples/cvat/recording-oracle/src/handlers/process_intermediate_results.py * Update packages/examples/cvat/exchange-oracle/src/crons/cvat/state_trackers.py * Fix ordering value register in API * Allow ordering in any register * Unify parent update mechanism on ORM models * Unify touch in cvat service * Remove touch methods from models * Update tests * Add dummy joints * Remove .vscode dir * Update cryptography to 43.0.1 GHSA-h4gh-qq45-vh27 * Replace python-jose with pyjwt GHSA-cjwg-qfpm-7377 GHSA-6c5p-j8vq-pqhj --------- Co-authored-by: maya <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: Dzeranov <[email protected]> Co-authored-by: Arseny Boykov <[email protected]> Co-authored-by: Arseny Boykov <[email protected]>
1 parent cb7153a commit 1eb2b81

File tree

78 files changed

+4234
-960
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+4234
-960
lines changed

packages/examples/cvat/exchange-oracle/Dockerfile

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,13 @@ WORKDIR /app
44

55
RUN apt-get update -y && \
66
apt-get install -y jq ffmpeg libsm6 libxext6 && \
7-
pip install poetry
7+
pip install --no-cache poetry
88

99
COPY pyproject.toml poetry.lock ./
1010

1111
RUN poetry config virtualenvs.create false \
12-
&& poetry install --no-interaction --no-ansi --no-root
12+
&& poetry install --no-interaction --no-ansi --no-root \
13+
&& poetry cache clear pypi --all
1314

1415
COPY . .
1516

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
"""add-assignment-updated-at
2+
3+
Revision ID: 2cbc85686054
4+
Revises: 0f3fb7bfcbcf
5+
Create Date: 2024-09-24 19:18:23.248579
6+
7+
"""
8+
9+
import enum
10+
11+
import sqlalchemy as sa
12+
from sqlalchemy import Column, DateTime, Enum, String
13+
from sqlalchemy.orm import Session, declarative_base
14+
from sqlalchemy.sql import func
15+
16+
from alembic import op
17+
18+
# revision identifiers, used by Alembic.
19+
revision = "2cbc85686054"
20+
down_revision = "0f3fb7bfcbcf"
21+
branch_labels = None
22+
depends_on = None
23+
24+
25+
Base = declarative_base()
26+
27+
28+
class AssignmentStatuses(str, enum.Enum):
29+
created = "created"
30+
completed = "completed"
31+
expired = "expired"
32+
rejected = "rejected"
33+
canceled = "canceled"
34+
35+
36+
class Assignment(Base):
37+
# Represents the model before the transaction is applied
38+
39+
__tablename__ = "assignments"
40+
id = Column(String, primary_key=True, index=True)
41+
created_at = Column(DateTime(timezone=True), server_default=func.now())
42+
updated_at = Column(DateTime(timezone=True), onupdate=func.now(), server_default=func.now())
43+
expires_at = Column(DateTime(timezone=True), nullable=False)
44+
completed_at = Column(DateTime(timezone=True), nullable=True, server_default=None)
45+
status = Column(
46+
String,
47+
Enum(AssignmentStatuses),
48+
server_default=AssignmentStatuses.created.value,
49+
nullable=False,
50+
)
51+
52+
53+
def define_initial_updated_at():
54+
bind = op.get_bind()
55+
session = Session(bind=bind)
56+
57+
session.query(Assignment).filter(
58+
Assignment.updated_at == None,
59+
Assignment.status.in_(
60+
[AssignmentStatuses.expired, AssignmentStatuses.rejected, AssignmentStatuses.canceled]
61+
),
62+
).update({Assignment.updated_at: Assignment.expires_at})
63+
64+
session.query(Assignment).filter(
65+
Assignment.updated_at == None,
66+
Assignment.status == AssignmentStatuses.completed,
67+
).update({Assignment.updated_at: Assignment.completed_at})
68+
69+
session.query(Assignment).filter(
70+
Assignment.updated_at == None,
71+
# fallback for invalid entries above + handling of status == "created"
72+
).update({Assignment.updated_at: Assignment.created_at})
73+
74+
75+
def upgrade() -> None:
76+
# This operation can fail because of null values, it's intentional.
77+
# Invalid entries in the DB must be fixed in order to proceed
78+
op.alter_column("assignments", "created_at", nullable=False)
79+
80+
op.add_column("assignments", sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True))
81+
define_initial_updated_at()
82+
op.alter_column("assignments", "updated_at", server_default=sa.text("now()"), nullable=False)
83+
84+
85+
def downgrade() -> None:
86+
# ### commands auto generated by Alembic - please adjust! ###
87+
op.drop_column("assignments", "updated_at")
88+
# ### end Alembic commands ###

packages/examples/cvat/exchange-oracle/docker-compose.dev.yml

Lines changed: 78 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
version: '3.8'
2-
31
services:
42
postgres:
3+
container_name: oracle-postgres
54
image: postgres:14.4
65
restart: on-failure
76
environment:
@@ -15,5 +14,81 @@ services:
1514
- 5431:5432
1615
command: ["postgres", "-c", "log_statement=all"]
1716

17+
redis:
18+
container_name: oracle-redis
19+
image: redis:7.2.3-alpine
20+
restart: always
21+
command: [
22+
"redis-server",
23+
"--save", "60", "100",
24+
"--appendonly", "yes",
25+
# Uncomment if you want to configure a password
26+
# "--requirepass", "password",
27+
# "--user", "username", "on", ">password", "~*", "allcommands",
28+
# "--user", "default", "off", "nopass", "nocommands",
29+
]
30+
volumes:
31+
- redis:/data
32+
ports:
33+
- 6380:6379
34+
networks:
35+
- test-network
36+
37+
minio:
38+
container_name: minio
39+
image: minio/minio:RELEASE.2022-05-26T05-48-41Z
40+
environment:
41+
MINIO_ROOT_USER: dev
42+
MINIO_ROOT_PASSWORD: devdevdev
43+
volumes:
44+
- ./tests/utils/datasets:/tmp/datasets
45+
entrypoint: 'sh'
46+
ports:
47+
- 9010:9010
48+
- 9011:9011
49+
command:
50+
-c "
51+
cp -r /tmp/datasets /data &&
52+
minio server /data --console-address ':9011' --address ':9010'"
53+
healthcheck:
54+
test: ["CMD", "curl", "-f", "http://localhost:9010/minio/health/live"]
55+
interval: 5s
56+
timeout: 5s
57+
retries: 3
58+
networks:
59+
- test-network
60+
- cvat-human-bridge
61+
62+
minio-mc:
63+
container_name: minio-mc
64+
image: minio/mc
65+
depends_on:
66+
minio:
67+
condition: service_healthy
68+
entrypoint: >
69+
/bin/sh -c "
70+
/usr/bin/mc config host add myminio http://minio:9010 dev devdevdev;
71+
/usr/bin/mc anonymous set public myminio/manifests;
72+
/usr/bin/mc mb myminio/excor-data;
73+
/usr/bin/mc anonymous set public myminio/excor-data;
74+
/usr/bin/mc mb myminio/recor-data;
75+
/usr/bin/mc anonymous set public myminio/recor-data;
76+
/usr/bin/mc mb myminio/datasets;
77+
/usr/bin/mc anonymous set public myminio/datasets;
78+
/usr/bin/mc mb myminio/launcher;
79+
/usr/bin/mc anonymous set public myminio/launcher;
80+
"
81+
networks:
82+
- test-network
83+
1884
volumes:
19-
postgres:
85+
postgres:
86+
redis:
87+
88+
networks:
89+
test-network:
90+
cvat-human-bridge:
91+
name: cvat-human-bridge
92+
external: true
93+
# Create with:
94+
# docker network create -o "com.docker.network.bridge.enable_icc=true" -o "com.docker.network.driver.mtu=1500" -o "com.docker.network.bridge.enable_ip_masquerade=true" cvat-human-bridge

packages/examples/cvat/exchange-oracle/docker-compose.test.yml

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
version: '3.8'
2-
31
services:
42
postgres:
53
image: postgres:14.4
@@ -14,6 +12,20 @@ services:
1412
ports:
1513
- 5432:5432
1614

15+
redis:
16+
container_name: oracle-redis
17+
image: redis:7.2.3-alpine
18+
restart: always
19+
command: [
20+
"redis-server",
21+
"--save", "60", "100",
22+
"--appendonly", "yes",
23+
]
24+
ports:
25+
- 6379:6379
26+
networks:
27+
- test-network
28+
1729
minio:
1830
container_name: minio
1931
image: minio/minio:RELEASE.2022-05-26T05-48-41Z
@@ -72,9 +84,12 @@ services:
7284
STORAGE_USE_SSL: False
7385
STORAGE_PROVIDER: 'aws'
7486
ENABLE_CUSTOM_CLOUD_HOST: Yes
87+
REDIS_HOST: 'redis'
7588
depends_on:
7689
postgres:
7790
condition: service_started
91+
redis:
92+
condition: service_started
7893
minio-mc:
7994
condition: service_completed_successfully
8095
# Used to reference localhost since minio:9000 is rejected by the SDK URL validator

packages/examples/cvat/exchange-oracle/dockerfiles/test.Dockerfile

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@ WORKDIR /app
44

55
RUN apt-get update -y && \
66
apt-get install -y jq ffmpeg libsm6 libxext6 && \
7-
pip install poetry
7+
pip install --no-cache poetry
88

99
COPY pyproject.toml poetry.lock ./
1010

1111
RUN poetry config virtualenvs.create false \
12-
&& poetry install --no-interaction --no-ansi --no-root
12+
&& poetry install --no-interaction --no-ansi --no-root \
13+
&& poetry cache clear pypi --all
1314

1415
COPY . .
1516

16-
CMD ["pytest", "-W", "ignore::DeprecationWarning"]
17+
CMD ["pytest", "-W", "ignore::DeprecationWarning", "-v"]

0 commit comments

Comments
 (0)