Skip to content

[curriculum-eval] curriculum-assessment: checkpoint_quality — CHECKPOINT_RE regex misses ✅ shortcode, zeroing scores corpus-wide #2472

Description

@github-actions

File: .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py (systemic — all 87 workshop files affected)
Overall Impact: All 87 learning pages score checkpoint_quality: 0.0 despite having valid ## :white_check_mark: Checkpoint sections.

Flagged Dimensions:

Dimension Score Benchmark Delta
checkpoint_quality 0.0 10.0 (items × 2.5, cap 10) −10.0

Root Cause (≤ 2 sentences):
CHECKPOINT_RE in curriculum_assessment.py matches only the literal Unicode emoji ✅ Checkpoint, but every workshop file uses the GitHub Markdown shortcode :white_check_mark: Checkpoint. Because the regex never matches, has_checkpoint is always False and checkpoint_quality is always 0.0.

Evidence (quoted from the file):

## :white_check_mark: Checkpoint — present in every learning page (e.g., 07d-confirm-model-access.md line 143, 04-github-actions-intro.md line 129, 05-agentic-workflows-intro.md line 120)

The failing regex:

CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE)

Learning Science Rationale:
Checkpoint quality is the second-highest-weighted rubric dimension (weight 2.0). Zeroing it artificially deflates every file's overall score by approximately 1.5–2.0 points and masks the true quality signal — preventing the rubric from measuring whether learners can self-assess their progress against explicit success criteria, a core principle of Bloom's formative assessment.

Improvement Prompt (for an agent):

In `.github/skills/curriculum-quantitative-assessment/curriculum_assessment.py`,
update CHECKPOINT_RE to match both the Unicode emoji and the GitHub Markdown shortcode:

  CHECKPOINT_RE = re.compile(
      r"##\s+(?:✅|:white_check_mark:)\s*Checkpoint",
      re.IGNORECASE
  )

After changing, re-run the rubric against the full workshop corpus and verify that
`has_checkpoint` is True for all learning pages that contain `## :white_check_mark: Checkpoint`.
Also audit PREREQ_RE and any other emoji-based regexes for the same mismatch pattern.

Expected Score After Fix: Corpus mean rises from 6.09 to approximately 7.5 / 10.0 once checkpoint_quality is correctly measured across all files.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔬 Curriculum Quality Evaluator · 75.2 AIC · ⌖ 6.21 AIC · ⊞ 6.2K ·

  • expires on Aug 4, 2026, 1:43 PM UTC

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions