MaDaBi - Mannheim Data Bibliography

This repo contains work in progress on creating Mannheim Data Bibliography (or datagraphy), i.e., a registry of metadata of all data, created or collected by the employees of the University of Mannheim.

Motivation

• FAIRness (Findability, Accessibility, Interoperability, and Reusability) of data at the University of Mannheim,
• a single point of access to metadata of data, created or collected by the employees of the University of Mannheim (MADATA),
• performance evaluation using metrics for data sharing, data reuse and data citation in the University of Mannheim,
• improving culture of data sharing, data reuse and data citation in the University of Mannheim.

Problem statement

Mannheim Data Bibliography MADATA contains only a part of all (meta)data published by the employees of the University of Mannheim. All other data, created or collected by employees of the University of Mannheim, are stored somewhere else.

We want to:

• collect metadata of data, created or collected by employees of University of Mannheim,
• update the data bibliography regularly,
• store the collected metadata in MADATA,
• evaluate metrics for data sharing, data reuse and data citation,
• create data dashboard.

Scope of work: Resource Types

What can data mean?

1. dataset,
2. software (code, script, package),
3. executable notebook (Jupyter notebook),
4. data management plan,
5. software management plan,
6. workflow,
7. model,
8. figure,
9. table,
10. image,
11. video,
12. text,
13. interview,
14. project (e.g., https://doi.org/10.3886/E124902V2),
15. reproducibility (or replication) package.

Controlled vocabulary for resource types of da|ra

• Audiovisual
• Collection
• DataPaper
• Dataset
• Event
• Image
• InteractiveResource
• Model
• PhysicalObject
• Service
• Software
• Sound
• Text
• Workflow
• Other

The resource types for DataCite DOIs:

• Audiovisual
• Book
• BookChapter
• Collection
• ComputationalNotebook
• ConferencePaper
• ConferenceProceeding
• DataPaper
• Dataset
• Dissertation
• Event
• Image
• InteractiveResource
• Journal
• JournalArticle
• Model
• OutputManagementPlan
• PeerReview
• PhysicalObject
• Preprint
• Report
• Service
• Software
• Sound
• Standard
• Text
• Workflow
• Other

Scope of work: Madoc and Uni-resources

Scope of work and plan:

• Extract dataset mentions, "Data availability statements" and "Supplemented Materials" from publications at MADOC, i.e., Mannheim University bibliography and publication server:
  • From full texts in PDF-files at MADOC
  • From external online versions of publications (using DOIs from MADOC)
• Add metadata for data resources (e.g., databases, digital editions, etc.) hosted at uni-mannheim.de to MADATA
  • Digital Editions in MaObjects

Scope of work: Repositories and portals

Plan:

• Harvesting metadata of data from data repositories and metadata portals
  • Searching for "University of Mannheim" or "Mannheim University" or "Universität Mannheim" ⏳
  • Searching for names of employees of University of Mannheim ⏳

Repositories and portals:

1. External data repositories with links to search queries "University of Mannheim":

   • Zenodo
   • GESIS Search
   • Dataverse
   • Figshare
   • GitHub and GitLab
     • https://github.com/dwslab
     • https://github.com/wbsg-uni-mannheim
     • https://github.com/socialsciencedatalab
     • https://github.com/UB-Mannheim
     • https://gitlab.uni-mannheim.de/processanalytics
   • CESSDA Data Catalogue
   • repositories described in subject-specific publication policies

2. Metadata portals with links to search queries "University of Mannheim":

   • BASE
   • Google Dataset Search
   • OpenAIRE Search

Harvested Metadata

See folder ./data/.

Building the datagraphy with AI (metadata extraction)

Much of the data produced by University of Mannheim authors is only discoverable inside their publications — in data-availability statements, supplementary materials, and citations. We mine MADOC full texts with an LLM-based pipeline, recover the data/code links, harvest metadata for each, attribute it to Mannheim authors, and assemble a unified datagraphy.

Extraction (code/extraction/):

1. python code/harvester/madoc.py → data/madoc.csv, then python code/extraction/fetch_madoc_ubma_flags.py to add UB EPrints flags → data/madoc_with_ubma.csv.
2. python code/extraction/download_madoc_papers.py downloads the open-access PDFs into data/pdf/ (download_missing_pdfs.py handles long filenames).
3. Extract structured metadata via RAG (FAISS + sentence-transformers) and an LLM — extract_metadata_api.py (hosted API: MaIA / GWDG, current default) or the local-Ollama extract_metadata.py / extract_metadata_reduced.py. Outputs land in data/from_papers/.

Datagraphy assembly (code/ + code/from_papers/):

• Repository-link discovery from the extraction output (notebooks/analyze_repository_links_from_papers.ipynb).
• Per-source metadata harvesters: run_osf_metadata.py, run_figshare_metadata.py, run_somef_github.py + run_github_contributors.py.
• Provenance, Mannheim-affiliation scoring, and unification: build_paper_lookup_from_madoc.py → scoring.py → build_unified_metadata.py.

📖 Full walkthrough, pipeline diagram, and reproduce commands: docs/building_datagraphy_with_ai.md.

Dependencies for the extraction scripts live in code/extraction/requirements.txt; see code/extraction/README.md for tooling notes. Note: full texts, the employee list, and personal-data outputs are git-ignored (see .gitignore).

Retrieval-augmented search prototype

The code/rag/ folder contains a prototype “GraphRAG” pipeline (rag_madata.py) that harvests MADATA via OAI-PMH, normalizes Dublin Core fields, creates TF‑IDF + sentence-transformer indices, wires relationship graphs (authors, subjects, DOIs), and answers natural-language questions with graph-aware retrieval. Typical usage:

python code/rag/rag_madata.py harvest --out data/madata_harvest.json
python code/rag/rag_madata.py query \
  --results data/madata_harvest.json \
  --question "Datasets from Professor Alpers since 2018"

The script can optionally call a local Ollama model (--ollama) to summarize results. See code/rag/README.md for setup instructions and more examples, and code/rag/requirements.txt for its dependency set.

License

All scripts in this repo are licensed under the MIT license. The metadata records are licensed under the CC0 license if not otherwise stated. Everything else in this repo is licensed under the Creative Commons Attribution 4.0 International license.