feat(backend): enforce model accuracy gate in CI, fix negative R² by adding Crop feature

[Copilot]

The training script had no accuracy enforcement — a broken model could be baked into the Docker image silently. Additionally, the model produced negative R² because it excluded crop type (which explains ~58% of yield variance) and lacked feature scaling. The Feature Sensitivity Analysis ran on every build (including CI), and zero-yield rows (failed crops) polluted training data.

Fix negative R² — model training improvements

• Added Crop to MODEL_FEATURE_COLUMNS: Crop type was the dominant predictor of yield (means range from ~116 g/m² for Winter Lentil to ~488 g/m² for Winter Wheat) but was excluded. The model now uses 5 features: Crop, elev_mean_m, slope_mean_deg, aspect_eastness, aspect_northness.
• Deterministic crop encoding: Added CROP_ENCODING dict to YieldCalculator for consistent crop-to-integer mapping across training and inference, replacing sklearn.LabelEncoder which produces different encodings depending on input data.
• Feature scaling via BatchNormalization: Added a BatchNormalization layer as the first layer in the model to handle scale differences (elevation ~750-800 vs aspect ~-1..1). Learned statistics are stored in the .keras file.
• Inference-time crop encoding: _calculate() in YieldCalculator now automatically encodes string crop codes, and raises ValueError on unknown crop codes (fail-fast).

Training script (CreateAndTrainYieldCalculatorModel.py)

• Accuracy gate: Added MIN_R2_THRESHOLD = 0.0 — script exits non-zero if R² on the held-out test set is below this, failing the Docker build and CI pipeline
• Structured accuracy report: Logs MSE, MAE, RMSE, R², threshold, row/feature counts in a delimited block for easy CI log inspection
• Data quality: Filter out zero-yield rows (GrainYieldAirDry <= 0) — these are planting failures, not valid observations
• Removed Feature Sensitivity Analysis: No longer runs during build

Helpers (helpers.py)

• Updated encode() to use the fixed CROP_ENCODING mapping instead of sklearn.LabelEncoder, with validation that raises ValueError on unknown crop codes
• Updated Create_Model() to include BatchNormalization after the input layer

Dockerfile

• Updated error message to point at the accuracy report in build logs
• Added comment documenting the accuracy enforcement behavior
• Added CACHEBUST build arg: Placed before the training RUN step so passing a unique value (e.g. git SHA, timestamp) invalidates Docker's layer cache and forces model retraining

CI (ci.yaml)

• Model retrains on every CI build: CACHEBUST is set to ${{ github.sha }} so the training layer cache is busted on every push, ensuring the model is never stale

Docker Compose (docker-compose.yml)

• Passes CACHEBUST build arg to the backend service, defaulting to empty (cached) for local dev; set CACHEBUST=$(date +%s) to force retraining locally

Test updates

• test_calculator.py: Added Crop column to test data to match updated MODEL_FEATURE_COLUMNS

Documentation (YieldPredictionModel/README.md)

• Added Model Accuracy Requirements section: threshold table, rationale for R², failure behavior, how to update the threshold, example log output
• Added Model Features section: feature table, crop encoding details, and feature scaling documentation
• Added Data Quality section: Cook harvest cleaning steps

[JosephPBaruch]

@copilot The R^2 error is too low (less than 0) investigate why this might be happening problems with how I am training the model and using the data.

[JosephPBaruch]

@copilot Also, set the r^2 to 0 and not 0.05

[Copilot]

@copilot The R^2 error is too low (less than 0) investigate why this might be happening problems with how I am training the model and using the data.

Investigated the root cause: the model was trained on only 4 terrain features (elev_mean_m, slope_mean_deg, aspect_eastness, aspect_northness) from a single small farm (~1km²) where terrain values have very low variance. Meanwhile, crop type alone explains ~58% of yield variance (mean yields range from ~116 g/m² for Winter Lentil to ~488 g/m² for Winter Wheat) but was excluded as a feature.

Three fixes in commits 0709ebf and e8d7d62:

1. Added Crop to MODEL_FEATURE_COLUMNS with a fixed CROP_ENCODING dict for deterministic encoding across training and inference (replaced sklearn.LabelEncoder which produces inconsistent mappings).
2. Added BatchNormalization as the first model layer to handle the scale mismatch (elevation ~750-800 vs aspect ~-1..1) that caused gradient dominance.
3. Updated _calculate() to automatically encode string crop codes at inference time, and fail fast with a clear ValueError on unknown crop codes.

[Copilot]

@copilot Also, set the r^2 to 0 and not 0.05

Updated MIN_R2_THRESHOLD from 0.05 to 0.0 in the training script and the README table in 9a3c33a.

[JosephPBaruch]

@copilot Is there a way to train the model every time. I don't like that the model itself is being cached.

[Copilot]

@copilot Is there a way to train the model every time. I don't like that the model itself is being cached.

Added a CACHEBUST build arg to the Dockerfile right before the training RUN step in 6c79401. In CI, it's set to ${{ github.sha }} so every push gets a unique value and Docker cannot reuse the cached training layer. For local builds, run CACHEBUST=$(date +%s) docker compose build backend to force retraining.