Cache data fetched from GitHub

[IgorTatarnikov]

Is your feature request related to a problem? Please describe.
The GitHub related metrics are fetched fresh each time the fetchers are run. This leads to long run times, specifically when computing metrics related to issues.

Describe the solution you'd like
We should cache all stats fetched from GitHub in a tabular format. Storing all metadata for each issue and contribution. This will allow us to check the "oldest" entry in the cache and only fetch content after that date, instead of using the configured [since]](

dashboard/config.yml

Line 6 in 180c63f

since: '2018-01-02'

). If no cache is present (to account for local runs, or if the cache goes stale, there should be fallback mechanisms to fetch all data).

We should also cache the overall repository stats (stars, contributors, downloads, etc...). This will allow us to integrate plots showing trajectory over time for most stats tracked by the dashboard.

[SKM2227229725]

Hello @IgorTatarnikov,

I would like to work on this issue.

From my understanding, the goal is to:

• Cache GitHub-fetched metrics (issues, contributions, stars, contributors, etc.)
• Store them in a structured tabular format
• Use the latest cached timestamp to incrementally fetch new data
• Provide a fallback mechanism if cache is missing or stale

I am thinking of implementing a caching layer that:

1. Stores issue metadata (created_at, updated_at, state, labels, author, etc.)
2. Tracks the latest fetched timestamp
3. Supports incremental updates instead of full re-fetch
4. Allows storing historical repository statistics to enable time-series plots

Before starting, I would like to clarify:

• Is there a preferred storage format? (e.g., CSV, SQLite, Parquet)
• Should the cache be local-only, or configurable via config.yml?
• Are there any existing utilities in the repository for persistent storage?

Looking forward to your guidance.

[IgorTatarnikov]

1. Parquet is preferred as that's already being used to store data fetched from conda-forge (see https://github.com/neuroinformatics-unit/dashboard/blob/main/oss_dashboard/fetchers/fetch_parquet.py)

2. The cache should be stored in the same location as the conda-forge download cache, to make it compatible with the current CI set up.

3. See fetch_parquet.py and the below for the current strategy

   dashboard/.github/workflows/fetch_and_deploy.yml

   Lines 43 to 52 in 00bbbad

   	- name: Get current date
   	id: date
   	run: echo "DATE=$(date +'%Y-%m')" >> $GITHUB_ENV
   	- name: Restore conda downloads cache
   	uses: actions/cache@v4
   	with:
   	path: |
   	~/.dashboard
   	key: conda-download-cache-${{ env.DATE }}

[SKM2227229725]

Hi @IgorTatarnikov,

Thanks for the clarification. I’ll implement the caching layer using Parquet and store it in ~/.dashboard following the existing CI caching setup. I’ll enable incremental fetching based on the latest cached timestamp and add a fallback for missing or stale cache.

I’ll review fetch_parquet.py and proceed accordingly.

Best regards,
Shailesh Kumar