Add DVP&R traceability and Playwright coverage for requirement validation by Copilot · Pull Request #154 · JosephPBaruch/CharAI

Copilot · 2026-04-14T22:23:27Z

Adds a DVP&R document that connects the product requirements and documented user scenarios to the current Playwright suite and CI reporting path, and extends the Playwright suite to cover additional documented requirements. The updated scope makes the existing validation coverage scannable, converts the regression-model validation language to R²-based pipeline validation, and turns previously documented gaps into executable browser tests.

New DVP&R traceability document
- Added docs/06_Design_Validation/DVPR.md
- References the authoritative requirement and scenario sources:
  - docs/01_Problem_Definition/Requirements.md
  - docs/01_Problem_Definition/CharAI.feature
  - frontend/tests/CharAI.feature
  - frontend/tests/charai.spec.ts
- References model accuracy enforcement files:
  - backend/YieldPredictionModel/CreateAndTrainYieldCalculatorModel.py
  - backend/YieldPredictionModel/README.md
  - backend/Dockerfile
Requirements-to-test mapping
- Maps core product requirements and Gherkin scenarios to specific Playwright specs
- Updates the matrix to reflect newly covered scenarios:
  - previous prescription-map access after sign-in
  - multiple-user support
  - yield-prediction model accuracy (R² gate in CI build)
- Keeps remaining uncovered scenarios clearly marked as gaps
- Uses spec names and file paths instead of brittle line-number references
New Playwright coverage
- Added a Playwright test for accessing previous prescription maps after signing back in
- Added a Playwright test for multiple-user support using separate browser contexts
- Strengthened the previous-map test to verify persisted analysis summary data after re-login
- Refactored repeated field-creation steps in frontend/tests/charai.spec.ts into shared helpers to keep the new coverage maintainable
Model R² validation in CI
- Added a dedicated "Model R² Validation in CI" section to the DVP&R documenting how the CI enforces R² during Docker builds
- Raised MIN_R2_THRESHOLD from 0.0 to 0.2 in the training script and updated backend/YieldPredictionModel/README.md to match
- Documents the full enforcement chain: ci.yaml → docker/bake-action with CACHEBUST → Dockerfile → training script accuracy gate
- Includes the structured accuracy report format printed to build logs
- Includes threshold history table and key file references
- Updated docs/06_Design_Validation/ValidationPlan.md with the specific R² threshold (0.2) and CI enforcement details
CI execution and reporting
- Documents how Playwright runs in .github/workflows/ci.yaml
- Captures where results are generated and retained:
  - frontend/tests/playwright-report/
  - uploaded GitHub Actions artifact: playwright-report
- Notes current coverage tracking limitations and where to update the doc if paths/reporting change
Merged main branch changes
- Pulled in the R² accuracy gate, Crop feature, BatchNormalization, and CACHEBUST build arg changes from main
Maintenance guidance
- Records the path dependencies that must stay in sync (including model-related files)
- Notes how to update the matrix when new documented scenarios are automated
- Notes how to update the DVP&R when MIN_R2_THRESHOLD changes

Example traceability entry shape:

| Requirement / intent | Related scenario(s) | Playwright evidence | Status |
| --- | --- | --- | --- |
| Users can export prescription-map results. | `Export prescription maps` | `frontend/tests/charai.spec.ts`: `User creates field and views prescription map` | Covered |

Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/abdf826f-8470-4ba7-815a-24b80e839b68 Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

JosephPBaruch · 2026-04-14T22:43:12Z

+- When a documented scenario is added to Playwright, update the matrix
+  from `Gap` to `Covered` and record the exact spec name.
+- If reporting changes from the HTML reporter to another format, update
+  the CI execution and result-location sections above.


@copilot Find where the model requiremetns have accuracy requirements but this isn't good for a regression model. We will use r2 instead and this will be validated on model greation in the pipeline. There is currently a PR up fo this.

Updated the regression-model validation wording to use R²-based pipeline validation in 764996d.

JosephPBaruch · 2026-04-14T22:44:23Z

+| Prescription-map output exposes ROI-oriented map data rather than only raw form submission success. | `docs/01_Problem_Definition/Requirements.md` (`## Functional Requirements` -> `### User-Facing Input/Output`) | `User requests prescription maps`; `Export prescription maps` | `frontend/tests/charai.spec.ts`: `User creates field and views prescription map` (line 104) | Covered | The test validates `Analysis Summary`, `Total Grid Cells`, and exported GeoJSON grid cells containing a `paybackPeriod` property, which is the ROI-oriented field used by the product. |
+| Users can export prescription-map results. | `docs/01_Problem_Definition/Requirements.md` (`## Functional Requirements` -> `### User-Facing Input/Output`) | `Export prescription maps` | `frontend/tests/charai.spec.ts`: `User creates field and views prescription map` (line 104) | Covered | Verifies download of a JSON file and parses the downloaded content. |
+| Users can provide land area by uploading a file. | `docs/01_Problem_Definition/Requirements.md` (`## Functional Requirements` -> `### User-Facing Input/Output`) | `User enters land area using an uploaded file` | No current Playwright spec in `frontend/tests/charai.spec.ts` | Gap | The behavior is documented in both feature files, but there is not yet an executable Playwright test covering the upload path. |
+| A signed-in user can access previous prescription-map data. | `docs/01_Problem_Definition/CharAI.feature` only | `User signs in` | No current Playwright assertion specifically proving retrieval of a previously created map after a new sign-in session | Gap | Authentication is covered, but this scenario detail is not yet explicitly automated. |


@copilot Was this: stateed in the .feature: A signed-in user can access previous prescription-map data. if it was, then create a test fo rthis now, you should be able to do this. If not, remove it from this file.

This requirement is stated in the .feature, and I added Playwright coverage plus updated the traceability entry in 764996d.

JosephPBaruch · 2026-04-14T22:45:03Z

+| Users can export prescription-map results. | `docs/01_Problem_Definition/Requirements.md` (`## Functional Requirements` -> `### User-Facing Input/Output`) | `Export prescription maps` | `frontend/tests/charai.spec.ts`: `User creates field and views prescription map` (line 104) | Covered | Verifies download of a JSON file and parses the downloaded content. |
+| Users can provide land area by uploading a file. | `docs/01_Problem_Definition/Requirements.md` (`## Functional Requirements` -> `### User-Facing Input/Output`) | `User enters land area using an uploaded file` | No current Playwright spec in `frontend/tests/charai.spec.ts` | Gap | The behavior is documented in both feature files, but there is not yet an executable Playwright test covering the upload path. |
+| A signed-in user can access previous prescription-map data. | `docs/01_Problem_Definition/CharAI.feature` only | `User signs in` | No current Playwright assertion specifically proving retrieval of a previously created map after a new sign-in session | Gap | Authentication is covered, but this scenario detail is not yet explicitly automated. |
+| The system supports multiple users concurrently. | `docs/01_Problem_Definition/CharAI.feature` only | `System supports multiple users` | No current Playwright spec in `frontend/tests/charai.spec.ts` | Gap | This scenario is documented but not yet represented as a Playwright test. |


@copilot add functional tests to test multiple users. using playwright

Added a Playwright multi-user test using separate browser contexts and updated the DVP&R coverage entry in 764996d.

JosephPBaruch · 2026-04-14T22:45:27Z

@@ -0,0 +1,110 @@
+# Playwright Requirements Traceability DVP&R


@copilot Just title this DVP&R

Retitled the document to DVP&R in 764996d.

Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/5d452bef-0877-4d31-9cb1-c38d5e52a0db Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

Copilot · 2026-04-14T23:04:00Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

portal.opentopography.org
- Triggering command: /usr/local/bin/python python -u CreateAndTrainYieldCalculatorModel.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

JosephPBaruch · 2026-04-14T23:30:21Z

@copilot Pull in recent changes from main. Then add details on how the ci is checking that the r2 of the accuracy and add details to the dvp and r. Change the r2 threshold in the ci to be higher (.2). Rename the dvp and r file to just be dvp and r.

…mentation-for-tests Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

…I R2 details Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/f4a1a779-95b2-4314-8177-79b505637f0c Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

Copilot · 2026-04-14T23:35:28Z

@copilot Pull in recent changes from main. Then add details on how the ci is checking that the r2 of the accuracy and add details to the dvp and r. Change the r2 threshold in the ci to...

Done in f6cf189:

Merged main into this branch (bringing in the R² accuracy gate, Crop feature, CACHEBUST changes)
Raised MIN_R2_THRESHOLD from 0.0 to 0.2 in the training script and updated the README table
Renamed PlaywrightRequirementsTraceability.md to DVPR.md
Added a new "Model R² Validation in CI" section to the DVP&R with details on how the CI enforces R² (Docker build gate, CACHEBUST, accuracy report format, threshold history, key files)
Added a model accuracy row to the traceability matrix
Updated ValidationPlan.md with the specific 0.2 threshold and CI enforcement details

Initial plan

843f326

Copilot AI assigned Copilot and JosephPBaruch Apr 14, 2026

Copilot started work on behalf of JosephPBaruch April 14, 2026 22:23 View session

Copilot AI linked an issue Apr 14, 2026 that may be closed by this pull request

DVP&R: Playwright test validation and requirements traceability #153

Closed

docs(docs): add playwright requirements traceability dvpr

4cc041e

Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/abdf826f-8470-4ba7-815a-24b80e839b68 Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add DVP&R document for Playwright test validation~~ Add DVP&R traceability for Playwright requirement validation Apr 14, 2026

Copilot finished work on behalf of JosephPBaruch April 14, 2026 22:27

Copilot AI requested a review from JosephPBaruch April 14, 2026 22:27

JosephPBaruch marked this pull request as ready for review April 14, 2026 22:38

JosephPBaruch requested changes Apr 14, 2026

View reviewed changes

Copilot started work on behalf of JosephPBaruch April 14, 2026 22:46 View session

JosephPBaruch requested review from B-Proctor, MatthewM60 and joshnorlin April 14, 2026 22:46

JosephPBaruch and others added 3 commits April 14, 2026 15:57

Moving presentations around

196b9f1

test(frontend): add previous map and multi-user coverage

764996d

Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/5d452bef-0877-4d31-9cb1-c38d5e52a0db Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

test(frontend): verify persisted map summary after sign in

18e283b

Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/5d452bef-0877-4d31-9cb1-c38d5e52a0db Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

Copilot AI changed the title ~~Add DVP&R traceability for Playwright requirement validation~~ Add DVP&R traceability and Playwright coverage for requirement validation Apr 14, 2026

Copilot finished work on behalf of JosephPBaruch April 14, 2026 23:04

Copilot AI requested a review from JosephPBaruch April 14, 2026 23:04

Copilot started work on behalf of JosephPBaruch April 14, 2026 23:30 View session

Copilot AI and others added 2 commits April 14, 2026 23:32

Merge remote-tracking branch 'origin/main' into copilot/add-dvpr-docu…

6d12664

…mentation-for-tests Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

docs(docs): merge main, raise R2 threshold to 0.2, rename DVPR, add C…

f6cf189

…I R2 details Agent-Logs-Url: https://github.com/JosephPBaruch/CharAI/sessions/f4a1a779-95b2-4314-8177-79b505637f0c Co-authored-by: JosephPBaruch <132173774+JosephPBaruch@users.noreply.github.com>

Copilot finished work on behalf of JosephPBaruch April 14, 2026 23:36

JosephPBaruch changed the title ~~Add DVP&R traceability and Playwright coverage for requirement validation~~ Add DVP&R traceability and Playwright coverage for requirement validation Apr 15, 2026

JosephPBaruch merged commit f36d92b into main Apr 15, 2026
1 check passed

JosephPBaruch deleted the copilot/add-dvpr-documentation-for-tests branch April 15, 2026 22:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DVP&R traceability and Playwright coverage for requirement validation#154

Add DVP&R traceability and Playwright coverage for requirement validation#154
JosephPBaruch merged 7 commits intomainfrom
copilot/add-dvpr-documentation-for-tests

Copilot AI commented Apr 14, 2026 •

edited

Loading

Uh oh!

JosephPBaruch Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

JosephPBaruch Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

JosephPBaruch Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

JosephPBaruch Apr 14, 2026

Uh oh!

Copilot AI Apr 14, 2026

Uh oh!

Copilot AI commented Apr 14, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

JosephPBaruch commented Apr 14, 2026

Uh oh!

Copilot AI commented Apr 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,110 @@
		# Playwright Requirements Traceability DVP&R

Conversation

Copilot AI commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JosephPBaruch Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

JosephPBaruch Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

JosephPBaruch Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

JosephPBaruch Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 14, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

JosephPBaruch commented Apr 14, 2026

Uh oh!

Copilot AI commented Apr 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Apr 14, 2026 •

edited

Loading