add swf_common_lib.rucio_utils derriverd from swf-common-lib/rucio_comms/utils.py by veprbl · Pull Request #42 · BNLNPPS/swf-common-lib

veprbl · 2026-07-18T15:26:48Z

rucio_comms is not a real import from the package and relies on RUCIO_COMMS_PATH or manual PYTHONPATH entries.

The new implementations don't use classes. We only use static functions and it's okay to have most simplest forms until we know we need something special.

This will be used by BNLNPPS/swf-testbed#62

github-actions · 2026-07-18T15:40:41Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       136    136     0%   5-336
-------------------------------------------------------------------
TOTAL                                   807    774     4%
============================== 5 passed in 0.32s ===============================

…mms/utils.py

calculate_file_checksum is almost the same impl as in rucio_comms, with chunk size parameter added

github-actions · 2026-07-18T15:53:35Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       136    136     0%   5-336
-------------------------------------------------------------------
TOTAL                                   807    774     4%
============================== 5 passed in 0.34s ===============================

zhaoyuyoung · 2026-07-21T20:26:31Z

+            logger.info(f"Dataset created: {scope}:{name}")
+        except DataIdentifierAlreadyExists:
+            logger.info(f"Dataset already exists: {scope}:{name}")
+


For an existing dataset, is it expected to refresh the lifetime? rucio_comms.DatasetManager.create_dataset() attempts to set lifetime for existing datasets here it's not handled.

For my usage to run a prompt processing workflow, this is not an issue because it's been always a new simulated run with a new dataset. Just think of the case when we needs to update an existing dataset.

From what I understand one can not simply change a lifetime for a dataset, but can submit an extension request via https://rucio.cern.ch/documentation/html/client_api/lifetimeclient.html

Also, our datasets are supposed to never expire, and even for testbed they are still relatively long-living.

And I found some slides on the topic:
https://indico.cern.ch/event/331406/contributions/770929/attachments/645580/888086/20141029TimDataLifeCycle.pdf

Now I understand why you removed the set_metadata part. For an existing DID, that's questionable to extend the lifecycle.

Actually I do set lifetime=7days for prompt processing runs on the testbed. It used to be 1 day and I found inconvenient when I need to find some recent runs but disappear next day.

Then I think this PR is good to merge.

Actually, maybe set_metadata works to extend. Seems like it just recalculates expired_at as utcnow() + lifetime and writes that into database.

I've not actually used it so far because all prompt processing workflows are newly simulated runs. In rucio_comms, it's after DataIdentifierAlreadyExists so I think it attempts to refresh an existing DID lifetime.

Whether set_metadata is valid needs to be tested if we don't know. However I think your question is worth consideration. Should an existing DID be renewed? Is it something about the experiment policy?

Dug out some old logs and I found this output:
2026-07-01 15:44:35,510 - INFO - rucio_comms.rucio_comms - Set lifetime to 7 days for group.daq:swf.102682.run
Looks like even for a new run, that block is executed actually.

I don't see why we would want to have finite dataset lifetimes. In any case, this code does not consider replica policies. But I did reinstate the lifetime extension, I think it won't hurt to have it, and it matches original PanDA behavior.

zhaoyuyoung · 2026-07-21T20:31:14Z

+
+        # Step 2: Check if DID already exists
+        try:
+            existing_did = data_obj.did_client.get_did(data_obj.rucio_scope, file_name)


I noticed this is a pre-existing issue that's also in rucio_comms. Can it be fixed here? This helper checks data_obj.did_client, but data agent doesn't have the attribute. Should it be rucio_did_client so there is no need to fix in data agent?

I've not encountered any issue yet with this bug because the prompt processing always registers new datasets :-) If not looking at this brach, I've never noticed this issue.

Nice catch, I've updated in both places.

github-actions · 2026-07-22T19:43:30Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       136    136     0%   5-336
-------------------------------------------------------------------
TOTAL                                   807    774     4%
============================== 5 passed in 0.43s ===============================

github-actions · 2026-07-22T22:39:15Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       139    139     0%   5-343
-------------------------------------------------------------------
TOTAL                                   810    777     4%
============================== 5 passed in 0.33s ===============================

github-actions · 2026-07-22T22:39:56Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       139    139     0%   5-343
-------------------------------------------------------------------
TOTAL                                   810    777     4%
============================== 5 passed in 0.38s ===============================

Copilot

Pull request overview

This PR moves Rucio-related helpers into the installable swf_common_lib package (src/ layout) and removes the legacy rucio_comms module that required PYTHONPATH/RUCIO_COMMS_PATH to import reliably. This aligns the Rucio utilities with how the rest of the library is consumed by downstream projects (e.g., swf-testbed).

Changes:

Added src/swf_common_lib/rucio_utils.py with standalone utility functions for scope extraction, checksums, dataset creation, and file attachment.
Removed the legacy rucio_comms package/modules from the repository.
Updated pyproject.toml mypy configuration to ignore missing imports for rucio.*.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/swf_common_lib/rucio_utils.py	New packaged Rucio utility module intended to replace `rucio_comms` imports.
rucio_comms/utils.py	Removed legacy utilities module (previous home of similar helpers).
rucio_comms/rucio_comms.py	Removed legacy higher-level orchestration/classes module.
rucio_comms/README.md	Removed README describing the legacy `rucio_comms` package.
rucio_comms/exceptions.py	Removed legacy exception definitions tied to `rucio_comms`.
rucio_comms/init.py	Removed legacy package initializer.
pyproject.toml	Added mypy override to ignore missing imports for `rucio.*`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-07-22T22:45:45Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       138    138     0%   5-341
-------------------------------------------------------------------
TOTAL                                   809    776     4%
============================== 5 passed in 0.36s ===============================

github-actions · 2026-07-22T22:47:02Z

Test Coverage Summary

src/swf_common_lib/__init__.py            0      0   100%
src/swf_common_lib/api_utils.py          87     87     0%   8-198
src/swf_common_lib/base_agent.py        441    441     0%   5-877
src/swf_common_lib/config_utils.py       30     30     0%   7-102
src/swf_common_lib/logging_utils.py      33      0   100%
src/swf_common_lib/rest_logging.py       80     80     0%   8-157
src/swf_common_lib/rucio_utils.py       139    139     0%   5-343
-------------------------------------------------------------------
TOTAL                                   810    777     4%
============================== 5 passed in 0.39s ===============================

zhaoyuyoung · 2026-07-23T00:31:09Z

Merge is approved. After that, I think you need a follow up to remove the fallback to rucio_comms in swf-testbed, because you've removed rucio_comms.

veprbl added 2 commits July 18, 2026 11:53

add swf_common_lib.rucio_utils derriverd from swf-common-lib/rucio_co…

535327b

…mms/utils.py

fix mypy issues

2a36373

calculate_file_checksum is almost the same impl as in rucio_comms, with chunk size parameter added

veprbl force-pushed the pr/swf_common_lib_rucio_utils branch from 66dd2bf to 2a36373 Compare July 18, 2026 15:53

veprbl requested a review from zhaoyuyoung July 20, 2026 14:40

zhaoyuyoung reviewed Jul 21, 2026

View reviewed changes

did_client -> rucio_did_client

c1f320d

recover behavior when lifetime_days is applied to existing datasets too

7a7f8ad

veprbl requested a review from Copilot July 22, 2026 22:38

remove rucio_comms

7d86908

Copilot started reviewing on behalf of veprbl July 22, 2026 22:39 View session

veprbl requested review from Copilot and removed request for Copilot July 22, 2026 22:39

Copilot started reviewing on behalf of veprbl July 22, 2026 22:40 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread src/swf_common_lib/rucio_utils.py Outdated

Comment thread src/swf_common_lib/rucio_utils.py

Comment thread src/swf_common_lib/rucio_utils.py Outdated

Comment thread src/swf_common_lib/rucio_utils.py Outdated

Comment thread src/swf_common_lib/rucio_utils.py

veprbl and others added 2 commits July 22, 2026 18:45

Potential fix for pull request finding

f861c9d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

bc18935

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

address copilot suggestions regarding logging/exceptions

d269fc8

zhaoyuyoung merged commit 0035410 into main Jul 23, 2026
4 checks passed

veprbl deleted the pr/swf_common_lib_rucio_utils branch July 23, 2026 05:35

Uh oh!

Conversation

veprbl commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 18, 2026

Test Coverage Summary

Uh oh!

github-actions Bot commented Jul 18, 2026

Test Coverage Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 22, 2026

Test Coverage Summary

Uh oh!

github-actions Bot commented Jul 22, 2026

Test Coverage Summary

Uh oh!

github-actions Bot commented Jul 22, 2026

Test Coverage Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026

Test Coverage Summary

Uh oh!

github-actions Bot commented Jul 22, 2026

Test Coverage Summary

Uh oh!

zhaoyuyoung commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

veprbl commented Jul 18, 2026 •

edited

Loading