Design document for the Zenodo like DOI per dandiset #2012

yarikoptic · 2024-08-22T01:45:53Z

A design doc composed with @djarecka to avoid dummy DOIs for dandisets

refs:

TODOs

complete initial pass
seek review
make explicit that DOI generation is optional overall, since LINC does not even need it (ref)
potentially add a sequence diagram of interactions between user, archive, datacite across different stages of embargoed -> public dandisets

but could already be checked out by @dandi/archive-maintainers folks since overall idea is formulated already and some early concerns/questions could already be asked/answered

doc/design/doi-generation-2.md

djarecka · 2025-01-19T00:43:08Z

I created some test to simulate the workflow in dandi/dandi-schema#275

yarikoptic · 2025-03-05T20:34:37Z

doc/design/doi-generation-2.md

+  - We might want a dedicated 404 page for deleted dandisets, or at least a message that the dandiset was deleted, and ideally describe the reason why it was deleted ("Upon request of maintainer", "Due to violation of terms of service", etc.)
+  - Then we adjust DOI record to point to that page.
+
+- Should we do anything at dandischema level?


@djarecka we are to answer those I think before we could call this "done" ;-)

doc/design/doi-generation-2.md

yarikoptic · 2025-04-14T20:09:14Z

doc/design/doi-generation-2.md

+- Upon changes to a non-embargoed, draft dandiset metadata record:
+  - If `Draft DOI`, attempt to "promote" it to `Findable`.
+    - If validation fails - keep `Draft DOI` (very limited validation), attempt to update datacite metadata record while keeping the same target URL.
+    - **Question to clear up**: what happens to `Draft DOI` if metadata record is invalid? It seems to create one with no metadata, but does it update only the fields it knows about?


@djarecka I feel like you clarified on this but we did not put it "in writing" here. What do you remember on this aspect?

ping @djarecka

yes, sorry, i just gave "thumbs up", since you were right, Draft DOI doesn't have to have metadata, it can only have url, and if has more, it will be added

doc/design/doi-generation-2.md

asmacdo · 2025-04-23T13:08:17Z

doc/design/doi-generation-2.md

+
+A django-admin script should be created and executed to create a `Dandiset DOI` for all existing dandisets.
+
+**Question to address**: Will adding a `Dandiset DOI` in addition to `Version DOI` require a db migration?


In the POC I've added a doi field to the Dandiset model which does add a migration.

isn't there also a "draft" Version with DOI (and that's where I guess we inject a fake one), i.e. could we avoid changing DB model?

I have to admit that I "just dont like that" since the Dandiset DOI semantically belongs to the Dandiset. I cant think of a reason this wouldnt work, but just feels messy.

To retrieve the Dandiset DOI via Django, someversion.dandiset.draft_version.doi IMO violates the "principle of least surprise".

Prior to publication the Dandiset DOI will point to the draft version (via the DLP) but after publication the dandiset DOI will point to the latest publication, so that would also be surprising

doc/design/doi-generation-2.md

asmacdo · 2025-04-23T13:12:10Z

doc/design/doi-generation-2.md

+  - **Follow up concern**: after dandiset and DOI publish, metadata of the Draft version of the dandiset could still be changed.
+    This potentially making changed record again "invalid".
+    Should be Ok'ish
+- Test site of datacite had different result of validation that the primary one


Do we have any more information about this?

It might have been a bad memory, @djarecka do you have any information on this or should we just remove this?

Suggested change

- Test site of datacite had different result of validation that the primary one

let's remove it. It was based on my memory of some old issue, but wasn't able to reproduce, or find old records.

doc/design/doi-generation-2.md

asmacdo · 2025-04-24T16:25:49Z

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Either way, we need to consider using some kind of validation. @djarecka @yarikoptic I suggest this as topic for our discussion tomorrow.

djarecka · 2025-04-25T15:13:58Z

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

correct!

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

That's probably not good, if the validation is not used before publishing. Was not aware.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Yes, no documentation, but that's the case...

doc/design/doi-generation-2.md

yarikoptic · 2025-05-09T21:45:30Z

doc/design/doi-generation-2.md

+## Concerns to keep in mind/address
+
+- **Question to clear up**: what happens to `Draft DOI` if metadata record is invalid?
+    - It seems to create one with no metadata, but does it update only the fields it knows about?


@djarecka did you check in your experiments?

doc/design/doi-generation-2.md

could not reproduce

Co-authored-by: Yaroslav Halchenko <[email protected]>

fallback to Dandiset pydantic model would require changes to that model.

yarikoptic · 2025-06-04T18:20:17Z

@asmacdo -- do you think this one is good enough and reflects everything you learned from implementing #2350 ? then we likely should just merge it. Any objections? @djarecka what are you thoughts?

asmacdo · 2025-06-04T18:28:11Z

@yarikoptic I think its good enough to merge. There are remaining "unknowns" relating to how datacite will respond with requests for Findable DOIs that might not have fully valid metadata, which will hopefully be illuminated by incorporating NSKEY so I can create permanent Findable DOIs that will not cause collisions. Whatever comes of that, I think its fine to open a new PR to the design doc if necessary.

yarikoptic · 2025-06-05T18:22:34Z

@asmacdo then please approve this PR "officially" -- I can't since I am the author. I have enabled auto-merge though ;)

asmacdo

Oh right, forgot this wasnt mine :)

dandibot · 2025-06-05T18:57:24Z

🚀 PR was released in v0.11.1 🚀

- Dandiset DOI will redirect to the DLP - Example: 10.80507/dandi.000004 - Dandiset DOI is stored in the doi field of the draft version - Dandiset DOI metadata (on Datacite) will match the draft version until first publication - Once a Dandiset is published, the Dandiset DOI metadata will match the latest publication See the design document for more details: dandi#2012

yarikoptic mentioned this pull request Aug 23, 2024

Create and maintain a "Findable" DOI for the Dandiset as a whole #1319

Open

djarecka reviewed Sep 30, 2024

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic marked this pull request as ready for review December 10, 2024 17:44

yarikoptic mentioned this pull request Dec 13, 2024

Concept DOI to reference multiple versions of a dataset OpenNeuroOrg/openneuro#2650

Open

satra reviewed Jan 17, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented Jan 17, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented Mar 5, 2025

View reviewed changes

guittari mentioned this pull request Mar 20, 2025

Add DOI steps to Class I-III aplbrain/BBQS-EMBER-Data-Model#14

Closed

yarikoptic commented Apr 7, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented Apr 7, 2025

View reviewed changes

doc/design/doi-generation-2.md Show resolved Hide resolved

yarikoptic commented Apr 7, 2025

View reviewed changes

doc/design/doi-generation-2.md Show resolved Hide resolved

asmacdo reviewed Apr 9, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

asmacdo reviewed Apr 11, 2025

View reviewed changes

doc/design/doi-generation-2.md Show resolved Hide resolved

yarikoptic mentioned this pull request Apr 14, 2025

Do try to (re)mint DOIs for prior failed to be minted but released dandisets #1953

Closed

2 tasks

yarikoptic commented Apr 14, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented Apr 14, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented Apr 14, 2025

View reviewed changes

asmacdo mentioned this pull request Apr 22, 2025

Introducing Dandiset DOIs #2350

Draft

19 tasks

asmacdo reviewed Apr 23, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

asmacdo reviewed Apr 30, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

asmacdo force-pushed the enh-doi-draft branch from bcf2318 to a67633f Compare May 5, 2025 17:40

yarikoptic commented May 9, 2025

View reviewed changes

doc/design/doi-generation-2.md Show resolved Hide resolved

yarikoptic commented May 9, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented May 9, 2025

View reviewed changes

doc/design/doi-generation-2.md Outdated Show resolved Hide resolved

yarikoptic commented May 9, 2025

View reviewed changes

doc/design/doi-generation-2.md Show resolved Hide resolved

yarikoptic and others added 12 commits June 2, 2025 10:44

Decide upon deletion of dandiset and DOI

51dc9d3

Alter text after discussion with @yarikoptic and @djarecka

f8b560f

Chart Remove: Findable DOI prior to publication

83718ee

remove concern about test vs prod validation

a3f13e2

could not reproduce

Add Dandiset DOI to dandi-schema TODO

20bf562

fixup: mermaid syntax and language

6600849

Apply suggestions from code review

8562dbe

Co-authored-by: Yaroslav Halchenko <[email protected]>

Update Schema changes based on discussion with @yarikoptic

c499cc1

Add cautions (gotchas) to be explicit about settings behavior

04092bc

fixup formatting

b6b4338

chore: cleanup format and clarify language

c54bd38

We opted to fallback to unvalidated

24ea6eb

fallback to Dandiset pydantic model would require changes to that model.

asmacdo force-pushed the enh-doi-draft branch from 02d83b3 to 24ea6eb Compare June 2, 2025 15:44

yarikoptic added documentation Changes only affect the documentation design-doc Involves creating or discussing a design document labels Jun 5, 2025

yarikoptic enabled auto-merge June 5, 2025 18:21

yarikoptic disabled auto-merge June 5, 2025 18:21

yarikoptic enabled auto-merge June 5, 2025 18:22

asmacdo approved these changes Jun 5, 2025

View reviewed changes

yarikoptic merged commit 3a9a346 into master Jun 5, 2025
7 of 11 checks passed

yarikoptic deleted the enh-doi-draft branch June 5, 2025 18:24

dandibot added the released This issue/pull request has been released. label Jun 5, 2025


		A django-admin script should be created and executed to create a `Dandiset DOI` for all existing dandisets.

		Question to address: Will adding a `Dandiset DOI` in addition to `Version DOI` require a db migration?

Design document for the Zenodo like DOI per dandiset #2012

Design document for the Zenodo like DOI per dandiset #2012

Uh oh!

Conversation

yarikoptic commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

djarecka commented Jan 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

djarecka Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmacdo commented Apr 24, 2025

Uh oh!

djarecka commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yarikoptic commented Jun 4, 2025

Uh oh!

asmacdo commented Jun 4, 2025

Uh oh!

yarikoptic commented Jun 5, 2025

Uh oh!

asmacdo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dandibot commented Jun 5, 2025

Uh oh!

Uh oh!

yarikoptic commented Aug 22, 2024 •

edited

Loading

djarecka commented Jan 19, 2025 •

edited

Loading

djarecka Apr 25, 2025 •

edited

Loading