feat: create task #207

JaeAeich · 2024-09-05T18:15:44Z

Summary by Sourcery

Add functionality to create TES tasks and convert them into Kubernetes jobs, including new modules and API endpoints. Refactor existing code for consistency and enhance error handling.

New Features:

Introduce a new feature to create TES tasks and convert them into Kubernetes jobs, including handling of taskmaster and executor jobs.
Add a new module for converting TES tasks to Kubernetes jobs, including handling of taskmaster and executor jobs.
Implement a new API endpoint for creating tasks in the TESK API, which processes task requests and returns responses.

Enhancements:

Refactor the existing code to use lowercase constants for better consistency and readability.
Enhance the KubernetesError class to include a method for checking if an object name is duplicated.

sourcery-ai · 2024-09-05T18:15:50Z

Reviewer's Guide by Sourcery

This pull request implements the creation of a task in the TESK (TES on Kubernetes) system. It introduces new modules for converting TES tasks to Kubernetes jobs, handling task creation requests, and managing the execution of tasks. The changes primarily focus on the backend implementation of task creation, including the conversion of TES task specifications to Kubernetes job configurations.

File-Level Changes

Change	Details	Files
Implement TES task to Kubernetes job conversion	Create TesKubernetesConverter class for converting TES tasks to Kubernetes jobs Implement methods to convert TES executors to Kubernetes jobs Add utility methods for generating Kubernetes job and config map templates	`tesk/k8s/converter/converter.py` `tesk/k8s/converter/template.py`
Implement task creation API endpoint	Create CreateTask function in the API controller Implement CreateTesTask class for handling task creation requests Add error handling and retry logic for task creation	`tesk/api/ga4gh/tes/controllers.py` `tesk/api/ga4gh/tes/task/create_task.py`
Add data structures for managing Kubernetes resources	Create Job class to represent Kubernetes job objects Implement Task class to manage the relationship between taskmaster and executor jobs Add utility methods for manipulating job and task objects	`tesk/k8s/converter/data/job.py` `tesk/k8s/converter/data/task.py`
Update utility functions and constants	Modify get_taskmaster_template function to include new configuration options Update constants and configuration structures Add new utility functions for handling Pydantic models and Kubernetes resources	`tesk/utils.py` `tesk/exceptions.py` `tesk/api/ga4gh/tes/models.py`

Tips

Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
Continue your discussion with Sourcery by replying directly to review comments.
You can change your review settings at any time by accessing your dashboard:
- Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
- Change the review language;
You can always contact us if you have any questions or feedback.

sourcery-ai

Hey @JaeAeich - I've reviewed your changes - here's some feedback:

Overall Comments:

Consider adding more unit tests for the new converter and task creation logic to ensure robustness and catch potential edge cases.
The error handling in CreateTesTask is good, but consider adding more detailed logging throughout the conversion process to aid in debugging and monitoring.

Here's what I looked at during the review

🟡 General issues: 4 issues found
🟡 Security: 1 issue found
🟢 Testing: all looks good
🟡 Complexity: 1 issue found
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

tesk/utils.py

tesk/k8s/converter/converter.py

tesk/api/ga4gh/tes/controllers.py

tesk/k8s/converter/executor_command_wrapper.py

tesk/api/ga4gh/tes/task/create_task.py

tesk/k8s/converter/data/job.py

tesk/k8s/converter/data/task.py

tesk/k8s/converter/executor_command_wrapper.py

tesk/utils.py

jemaltahir

Similar comments from PR 206, can be revisited here as well. All in all great work :)

tesk/api/ga4gh/tes/task/create_task.py

tesk/k8s/converter/data/task.py

tesk/k8s/converter/template.py

tesk/utils.py

uniqueg

Haven't yet managed to review the converter subpackage. Let's see that we get #206 merged and all of the other comments addressed first as things are becoming too complex to review effectively.

tesk/api/ga4gh/tes/controllers.py

tesk/exceptions.py

tesk/api/ga4gh/tes/task/create_task.py

codecov · 2024-10-13T17:13:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.21%. Comparing base (a3e2cc2) to head (5607e62).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #207   +/-   ##
=======================================
  Coverage   98.21%   98.21%           
=======================================
  Files           8        8           
  Lines         561      561           
=======================================
  Hits          551      551           
  Misses         10       10

Flag	Coverage Δ
test_unit	`98.21% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

uniqueg

I need to leave it off here for now - it's very late. I'll try my best to go through this, but note that it is 2000 lines or so of complex code. It will take several more hours to go through this.

tesk/api/ga4gh/tes/controllers.py

uniqueg · 2024-10-14T00:50:50Z

tesk/api/ga4gh/tes/task/task_request.py

Yeah, that second part is where I would see this going then. Maybe even ending up in FOCA.

I mean, if we already have 21 changed files (not including tests) for the addition of a single controller, we might as well go all the way, right? 😛

uniqueg · 2024-10-14T01:00:18Z

tesk/api/ga4gh/tes/task/task_request.py

@JaeAeich: Why is it that with your PRs I often see the entire file changed, even though it's just an iteration over the last time I've viewed? I don't have this with other people, so I guess it's something with your editor or Git flow.

Please have a look at this, because it's really quite annoying. Instead of just focusing on what changed since the last time, I have to look through the entire file again - which is not only not fun, but also holds up the reviews big time, especially when they tend to end up being huge.

And even apart from that, it's also really not good practice in terms of provenance. If this were previously existing code (e.g., maybe some of the old TES code from TES-core still remains), you'd end up being listed as git blame for every line, taking credit and blame for other people's work.

tesk/k8s/converter/converter.py

uniqueg · 2024-10-14T01:16:14Z

Please also note that some checks are failing and unit tests for the code are entirely missing as of yet. I suppose you can already go ahead addressing/adding these while I finalize the review (it's obviously mostly good code, so I wouldn't expect that a lot of the tests will need to be rewritten).

uniqueg

A few minor open issues to fix

tesk/utils.py

uniqueg · 2025-03-24T13:35:39Z

tesk/utils.py

+import json
 import os
 from pathlib import Path
+from typing import List, Sequence


From Py 3.9 onwards, you can use list, dict etc. - no need to import the capitalized version from typing (but you still need to import Sequence). And I hope we're not using Py 3.8 or lower, since they are beyond end of life.

uniqueg · 2025-03-24T13:38:21Z

tesk/k8s/wrapper.py

+        batch_api: Kubernetes batch API client
+        core_api: Kubernetes core API client
+        namespace: Kubernetes namespace
+        tesk_k8s_constant: TESK Kubernetes constants


Please be consistent with the naming. It's multiple constants, so please name it constants throughout, as arg and as attribute, in the code and in the docs.

uniqueg · 2025-03-24T13:51:21Z

tesk/api/ga4gh/tes/task/task_request.py

+        try:
+            res: dict = json.loads(json.dumps(response))
+            return res
+        except (TypeError, ValueError) as e:
+            logger.info(e)


What's the error? Do you think it's a bug in Pydantic? If you really have good reason to use this code as it is, I think you should document inside the code exactly why you couldn't just validate the model with Pydantic, and do so by linking to an existing issue.

Currently my assumption is that there's something wrong with your code and this workaround just masks that.

uniqueg · 2025-03-24T13:53:06Z

tesk/api/ga4gh/tes/task/task_request.py

+        self.tesk_k8s_constants = tesk_k8s_constants
+
+    @abstractmethod
+    def handle_request(self) -> BaseModel:


Shouldn't this be a more specific model that is returned? I mean the function is called handle_request(), not handle_model().

uniqueg · 2025-03-24T14:05:23Z

tesk/api/ga4gh/tes/task/task_request.py

I guess I meant that it happens between reviews. If I have seen and checked a file in review #1, it shouldn't be entirely green again in review #2, unless the entire file changed between reviews #1 and #2 - which shouldn't really happen.

Anyway... please rename it somehow. If it's generic to a degree that at some point we might put it in FOCA (which it certainly isn't in its current form), maybe just use request.py and Request for module and class, respectively? Otherwise maybe tes_request.py and TesRequest?

JaeAeich added 2 commits September 5, 2024 16:09

feat: get taskmaster values from config

06602d1

feat: create task

b5a917d

JaeAeich requested a review from lvarin September 5, 2024 18:15

sourcery-ai bot reviewed Sep 5, 2024

View reviewed changes

fix: add podinfo volumes

3fa2eca

JaeAeich requested a review from jemaltahir September 6, 2024 12:06

minoe

ffecf22

JaeAeich requested a review from uniqueg September 6, 2024 12:14

minor

0acd4b0

jemaltahir requested changes Sep 9, 2024

View reviewed changes

tesk/api/ga4gh/tes/task/create_task.py Outdated Show resolved Hide resolved

tesk/k8s/converter/data/task.py Outdated Show resolved Hide resolved

tesk/k8s/converter/template.py Show resolved Hide resolved

tesk/utils.py Outdated Show resolved Hide resolved

merge

dd14159

uniqueg requested changes Sep 10, 2024

View reviewed changes

JaeAeich and others added 10 commits September 11, 2024 00:00

review changes

2948508

remove getting env from os

7f21d1f

mypy

71d9be1

merge with new changes from template

1a56253

remove inline typing and use constants

41d8d19

merge template

0dfe256

change int to str to make it json marshalable

e4993df

change int to str to make it json marshalable

2ac6959

review

a4edd0e

Merge branch 'template' into createT

8586b4d

JaeAeich requested a review from uniqueg October 8, 2024 16:03

JaeAeich added 6 commits October 11, 2024 10:25

reviiews

0d7981a

Merge remote-tracking branch 'origin/createT' into createT

3eaad7f

reviiews

c45724b

docs till here

b182a87

format

b4570a5

error

925d63a

Base automatically changed from template to main October 13, 2024 17:09

chore: merge conflic

11976e1

uniqueg requested changes Oct 14, 2024

View reviewed changes

JaeAeich added 2 commits October 14, 2024 10:48

review

c6e3964

remove todo and fixme

5607e62

uniqueg requested changes Mar 24, 2025

View reviewed changes

feat: create task #207

Are you sure you want to change the base?

feat: create task #207

Uh oh!

Conversation

JaeAeich commented Sep 5, 2024 • edited by sourcery-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by Sourcery

Uh oh!

sourcery-ai bot commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer's Guide by Sourcery

File-Level Changes

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jemaltahir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uniqueg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Oct 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

uniqueg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

uniqueg Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

uniqueg Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uniqueg commented Oct 14, 2024

Uh oh!

uniqueg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

uniqueg Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

uniqueg Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

uniqueg Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

uniqueg Mar 24, 2025

Choose a reason for hiding this comment

JaeAeich commented Sep 5, 2024 •

edited by sourcery-ai bot

Loading

sourcery-ai bot commented Sep 5, 2024 •

edited

Loading

codecov bot commented Oct 13, 2024 •

edited

Loading