Dataset add operator. Fixes #17 #19
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…, query builders, constants. (Motivation: Will be helpful to extend code base to Pangaea) 2. SUpport multi valued parameters by changing input type to str/list[str], coercing list values to a string using |(pipe) and adding respetive {name}AndOr flag to query. 3. Set base towards using logging package. 4. Change input params with kwargs.
…am; changed patches for utils.api.http.get
doswal
commented
Sep 11, 2025
pyleotups/core/Dataset.py
Outdated
| if not same: | ||
| log.warning( | ||
| "Dataset union: duplicate StudyID %s with differing content. " | ||
| "Keeping left-hand version.", sid |
There was a problem hiding this comment.
Add:
(If doing C = A + B, contents of A will be kept)
doswal
commented
Sep 11, 2025
pyleotups/core/Dataset.py
Outdated
| for sid, study in other.studies.items(): | ||
| if sid in merged.studies: | ||
| try: | ||
| same = (merged.studies[sid].to_dict() == study.to_dict()) |
There was a problem hiding this comment.
Comment the line to state that same refers to content and not StudyId
khider
requested changes
Sep 11, 2025
There was a problem hiding this comment.
Please add pytest for A=A+B and C=A+B.
Test with 2 datasets (do a xmlid or NOAA study ID query).
Case 1: And B have different IDs so C should contain both A and B IDs.
Case 2: A and B have the same ID so C should look like A.
Case 3: A and B have the same ID but different content, so the warning should be printed and C should still look A.
khider
approved these changes
Sep 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This features implements set union like addition of two Dataset Objects. In simpler words, two different searches can be combined together.
Dataset holds the StudyId(s) and respective study object(s) from the response retrieved from the query.
With two or more dataset Objects, one can now Add datasets as:
C = A+B
A = A+B
Multiple Study objects can be created over same StudyId in multiple concurrent searches. However, the data stored in these objects will always be same. Programmatically, this is checked before merging two dataset when an overlapping ID is found to avoid incorrect deletion.
Freehand tests added in example notebook. pyTest tests to be added soon.