RFC: Store Hyperion Consistency Checks as Exercise Reviews (multi-location)

[FelixTJDietrich]

Status: Draft
Depends on: #10967 (versioning)
Related: #10969 (inline reviews) · #11339 (consistency check UX)

Summary

Treat AI-found consistency issues as first-class review comments so they persist across reloads, drive unresolved counts, and support instructor actions.

Motivation (Why)

• Make AI findings durable and actionable (approve/dismiss/resolve/unresolve).
• Show unresolved counts in file tree and repository switcher.
• Correlate runs with Langfuse/OTel for debugging and product insights.

Proposal (Approach)

• Each Hyperion run creates one ExerciseReview for the active ExerciseVersion with review_type = AI_CONSISTENCY.
• Each AI “consistency issue” becomes an ExerciseReviewComment on that review.
• Instructor actions update comment status; unresolved counts derive from status.
• Support issues that span multiple places (statement + files) via a single JSONB locations array (no extra tables).

Schema delta (builds on #10969)

erDiagram
    Exercise ||--o{ ExerciseVersion : "has versions"
    ExerciseVersion ||--o{ ExerciseReview : "reviewed in"
    ExerciseReview ||--o{ ExerciseReviewComment : "contains"
    User ||--o{ ExerciseReview : "authors"

    Exercise {
        long id PK
    }

    ExerciseVersion {
        long id PK
        long exercise_id FK
        text problem_statement
        text template_commit_sha
        text solution_commit_sha
        text tests_commit_sha
        timestamp created_at
    }

    ExerciseReview {
        long id PK
        long exercise_version_id FK
        long author_id FK "user who triggered the consistency check"
        enum review_type "MANUAL | AI_CONSISTENCY"
        jsonb ai_metadata "model, prompt_version, trace_id (nullable)"
        timestamp submitted_at "for AI: set at creation"
    }

    ExerciseReviewComment {
        long id PK
        long review_id FK
        enum status "OPEN | APPROVED | DISMISSED | RESOLVED"
        jsonb locations "[] of {component_type, path?, start_line?, end_line?, start_offset?, end_offset?}"
        enum consistency_issue_severity "LOW | MEDIUM | HIGH (nullable)"
        enum consistency_issue_category "<CONSISTENCY ISSUE TYPES> (nullable)"
        text body
    }

    User {
        long id PK
    }

Loading

locations example

[
  {"component_type":"PROBLEM_STATEMENT","start_line":10,"end_line":12},
  {"component_type":"REPOSITORY_TESTS","path":"test/FooTest.java","start_line":5,"end_line":9}
]

Counting & Telemetry

• Unresolved counts: status IN (OPEN, APPROVED); exclude DISMISSED, RESOLVED.
• Tracing: store model/prompt/trace in ExerciseReview.ai_metadata for Langfuse/OTel correlation.

Feedback welcome.

[wasnertobias]

ExerciseReviewComment needs suggested_fix and probably an extension of suggested_patch as well!

Edit: Ok, suggested_fix could also be simply saved in the attribute text instead.

[IcanCUthere]

erDiagram
    Exercise ||--o{ CommentThread : "has threads"
    CommentThread ||--o{ Comment : "contains"

    Exercise {
        long id PK
        text test_repository_url
        text template_repository_url
        text solution_repository_url
        text auxiliary_repository_url
    }

    CommentThread {
        long id PK
        long group_id
        long exercise_id FK
        enum target_type "PROBLEM_STATEMENT | TEMPLATE_REPO | SOLUTION_REPO | TEST_REPO | AUXILIARY_REPO"
        long auxiliary_repo_id "optional"
        text file_path
        long line_number
        long initial_version_id
        long initial_line_number
        enum state "ACTIVE | OUTDATED | RESOLVED"
    }

    Comment {
        long id PK
        long thread_id FK
        long author_id FK
        text comment_text
        timestamp created_date
        timestamp last_modified_date
    }

Loading

This is what @Elfari1028 and I discussed, that takes into consideration the new versioning system and the focus more on general comments instead of only AI. Explanation:

Overall changes:

• Comments should not reference any version but newest version, as we don't want to address comments in older version. What if we go back versions? Right now the plan is to go back versions by putting old versions as the newest version, so this won't be a problem.
• Why no ExerciseReview anymore? The data in the table is not really that useful. If we want to group comments for a review, this can be UI only.
• Added a CommentThread table that stores multiple comments. This enables replying to comments beneath a comment.

Specifics:

CommentThread:

• group_id: In consistency checks, we return multiple comments that are shown in different locations which belong together (group). If we e.g. resolve one of these comments, the other comments should also be resolved.
• target_type: In which repo (or problem statement) should the thread be visible
• file_path, line_number: self explanatory, but line number has to be updated if a new version is created and the file was changed
• initial_version_id, initial_line_number: If a version changes, the comment can potentially not be shown anymore. If we still want to show a code preview of the initial commented code, we need these attributes.

Comment:

• Should be self-explanatory
• Maybe needs more attributes

[FelixTJDietrich]

I'm against not referencing the version. For the UI we don't need it but I want to know which state the exercise was in when the comment was given so I understand the history of changes and comments / detected consistency issues for auditing purposes. This will help creating a dataset in the future for the programming exercise generation / consistency checking.

Comments in the UI should always reference the latest version but you can do it via a database query.

The question is, can you review a diff? Like version 1 vs version 2? Then you need something like "side" on CommentThread as GitHub has it.

[FelixTJDietrich]

Maybe you want to have a field in_reply_to as GitHub has it in their schema for Comment then you don't have to rely on creation date for sorting. The root comment has in_reply_to=NULL

[IcanCUthere]

erDiagram
    Exercise ||--o{ ExerciseVersion : "has versions"
    Exercise ||--o{ CommentThread : "has threads"
    CommentThread ||--o{ Comment : "contains"
    User ||--o{ Comment : "authors"
    ExerciseVersion ||--o{ CommentThread : "initial version"

    Exercise {
        long id PK
        text test_repository_url
        text template_repository_url
        text solution_repository_url
        text auxiliary_repository_url
    }

    ExerciseVersion {                                                                                                                            
          bigint id PK                                                                                                                              
          bigint exercise_id FK                                                                                                                                                           
          json exercise_snapshot "ExerciseSnapshotDTO with problem statement, commit SHAs, etc."                                                                                                                                                                 
          datetime created_date                                                                                                                                                                                                                              
      } 

    CommentThread {
        long id PK
        long group_id
        long exercise_id FK
        enum target_type "PROBLEM_STATEMENT | TEMPLATE_REPO | SOLUTION_REPO | TEST_REPO | AUXILIARY_REPO"
        long auxiliary_repo_id "optional"
        text file_path
        long line_number
        long initial_version_id FK
        long initial_line_number
        enum state "ACTIVE | OUTDATED | RESOLVED"
    }

    Comment {
        long id PK
        long thread_id FK
        long author_id FK
        long in_reply_to FK
        text comment_text
        timestamp created_date
        timestamp last_modified_date
    }

    User {
        long id PK
    }

Loading

Yes it makes sense to have a in_reply_to field and a version_id for comments, I added them. For viewing diffs, we can use versionon_id in the Comment and initial_line_number in the CommentThread which is one side, and the other side is the latest version id of an exercise (which doesn't need to be stored in the comments i guess?) and the current line_number in CommentThread.

[FelixTJDietrich]

I think you would only have initial_version_id and initial_line_number on the CommentThread other than that I think it's fine. You need to check how the ExerciseVersion table actually looks like now, it might have changed since the initial proposal.