Agentic EHR — patient-friendly summaries over EHR risk models

A reproducible prototype that wraps a classical EHR risk model (XGBoost baseline) with an agent layer that turns raw predictions, feature attributions, and the patient's current state into a cautious, non-diagnostic, patient-friendly health summary.

The predictive model and the summary agent are completely decoupled: they communicate only through a RiskProfile object. You can swap XGBoost for other models without changing the agent (see Replacing the baseline model).

The AI (LLM) summary agent will generate health summary based on RiskProfile. Current support llms includes Gemini, Claude, or GPT.

Every llm provider is driven by one shared, non-diagnostic safety prompt. There are two output modes (the same safety guardrails apply to both):

• template (default) — constrained to structured outputs, yielding the fixed five sections defined in agent/templates.py.
• no-template — one cohesive free-form narrative, no fixed sections. Select it with agentic-ehr demo --no-template.

⚠️ Not medical advice. Outputs are educational, generated by a model, and must not be used for diagnosis or treatment. The demo uses synthetic, non-PHI data. No patient-identifying data is required anywhere in the system.

---

Architecture

structured EHR events
        │  data/  (ingest → CountFeaturizer → EHRDataset / MIMIC)
        ▼
  feature matrix X ───────────────► models/  BaseModel
        │                                  │  XGBoostClassifierModel (proba + uncertainty)
        │                                  │  XGBoostRegressionModel (point estimate)
        │                                  ▼
        │                          explain/  Attributor (SHAP / fallback)
        │                                  + ConceptMap (codes → plain language)
        │                                  │
        │                                  ▼
        └──────────────►  RiskProfile  /  HealthRiskProfile  ◄─ the ONLY model→agent contract
                          (single task)    (multi-task)
                                           │
                                           ▼
                                    agent/  SummaryAgent + guardrails
                                           │  (LLM agent: Gemini | Claude | GPT)
                                           ▼
                                    PatientSummary (5 sections)
                                           │
                                           ▼  (optional) eval/response_log
                                    JSON / MD / TXT record for review

The five summary sections are always: What we found · What may be contributing · What this means · What to do next · When to seek care urgently, followed by a standing disclaimer. SummaryAgent consumes a single-task RiskProfile or a multi-task HealthRiskProfile (the MIMIC prediction panel) through the same interface.

Layers (src/agentic_ehr/)

Module	Responsibility
data/	Schema, synthetic generator, FEMR/EHR-shot loader, CountFeaturizer, EHRDataset
data/mimic/	MIMIC-IV cohort/feature/label build (DuckDB), task registry, multi-task train + inference
models/	BaseModel ABC, XGBoostClassifierModel + XGBoostRegressionModel, registry
explain/	Attributor (SHAP + fallback), ConceptMap, RiskProfile / HealthRiskProfile
agent/	SummaryAgent (LLM-only, no-fallback), provider backends (Gemini/Claude/GPT), guardrails
eval/	Predictive metrics (classification + regression) + summary quality + response logging
pipeline.py	End-to-end orchestration (train, InferenceService)

---

Setup

cd AgenticEHR
python -m venv .venv && source .venv/bin/activate
pip install -e .            # core deps (incl. google-genai + anthropic LLM agents)
export GEMINI_API_KEY=...   # required for the default (Gemini) LLM summary agent
# optional extras:
pip install -e ".[openai]"  # the GPT provider (then set agent.llm.provider: openai)
pip install -e ".[explain]" # SHAP attributions (otherwise a built-in fallback is used)
pip install -e ".[mimic]"   # DuckDB + pyarrow for the MIMIC-IV multi-task pipeline
pip install -e ".[dev]"     # pytest

train and the evaluate metrics don't call llm so need no key; only summary generation (demo, evaluate --n-summaries) calls the API. The test suite injects an offline backend double, so it runs without any key.

---

End-to-end example

# 1) Train the XGBoost baseline on the synthetic EHR-shot-style task and save it.
agentic-ehr train
#   → artifacts/models/xgboost.joblib  + printed test metrics (AUROC, AUPRC, Brier, ECE)

# 2) Evaluate predictive performance AND summary quality.
agentic-ehr evaluate --n-summaries 25

# 3) Summarise a specific patient (fixed five-section template by default).
agentic-ehr demo --patient-id SYN000123
# 3b) Same patient, free-form narrative instead of the template.
agentic-ehr demo --patient-id SYN000123 --no-template

If the agentic-ehr console script isn't on your PATH, run it as a module: python -m agentic_ehr.cli train, etc.

Demo inference output

1. With Template:

========================================================================
PATIENT: SYN000116   |   risk tier: elevated   |   estimate: 80%   |   confidence: moderate
========================================================================
## What we found
Based on a review of your health records, a computer model estimates there is about an 80% chance of an unplanned hospital stay over the next 12 months. This is a statistical estimate based on patterns in health data, not a certain prediction of what will happen.

## What may be contributing
The model identified several factors associated with this estimate. These include how much recent healthcare activity you've had, reduced kidney function on record, a history of type 2 diabetes, your age of 73, and your recent kidney-function (eGFR) results. These are patterns the model noticed in the data, not proven direct causes.

## What this means
This estimate falls into the elevated risk category. Because the model's confidence in this estimate is moderate, it should be viewed as a helpful guide rather than an absolute certainty. It is a tool to help you and your healthcare team look ahead and plan.

## What to do next
We recommend sharing this summary with a qualified clinician. You can use it to start a conversation about managing your health, reviewing your current care plan, and discussing ways to support your kidney health and diabetes management.

## When to seek care urgently
Please seek immediate medical attention if you experience severe warning signs such as chest pain, sudden shortness of breath, severe dizziness, confusion, or any other sudden and serious changes in your health.

---
_This summary is generated by a computer model to help you understand your health information. It is not a diagnosis and not a substitute for advice from a qualified clinician. Always talk to your healthcare provider about your results and any decisions._

2. No Template:

========================================================================
PATIENT: SYN000116   |   risk tier: elevated   |   estimate: 80%   |   confidence: moderate   |   mode: free
========================================================================
A computer model has analyzed your health information to estimate the likelihood of an elevated overall health risk in the near future. The model calculated this statistical estimate to be about 80%. Please keep in mind that this is a mathematical estimation based on patterns in your health data, not a prediction of what will happen to you. Because the model’s confidence in this estimate is moderate, please treat this information with extra caution.

This estimate is associated with several factors found in your records, including your age of 73, a history of type 2 diabetes, and records indicating reduced kidney function. The model also noted your recent healthcare activity as a factor that influences this calculation. It is important to understand that these factors are simply associated with the model's output and do not necessarily explain the underlying cause of your health status.

This information is intended to help you prepare for a conversation with your healthcare provider. Because this is a synthetic estimate, it does not replace a professional clinical evaluation. You should discuss these findings with your doctor to better understand your current health status and determine if any changes to your care plan are appropriate.

If you experience any concerning symptoms, such as sudden shortness of breath, chest pain, confusion, or severe weakness, please seek urgent medical attention immediately. These are signs that require prompt evaluation by a medical professional, regardless of any model-based risk estimates.

---
_This summary is generated by a computer model to help you understand your health information. It is not a diagnosis and not a substitute for advice from a qualified clinician. Always talk to your healthcare provider about your results and any decisions._

---

Using real EHR-shot / FEMR data

The demo is synthetic so it is reproducible and PHI-free. To run on real EHR-shot data:

1. Use FEMR to extract events and task labels.
2. Export them to two tables (parquet or CSV):
   • events: patient_id, time, code, value[, description]
   • labels: patient_id, label_time, value[, age, sex]
3. Point the config at them and switch the source:

data:
  source: femr
  femr:
    events_path: /path/to/events.parquet
    labels_path: /path/to/labels.parquet

The CountFeaturizer and the rest of the pipeline are unchanged. Anything benchmark-specific (lookback window, vocabulary size, task wording, risk tiers, the concept map) is configurable in config/default.yaml, never hard-coded. Extend the plain-language ConceptMap for a new vocabulary via explain.concept_map_path.

---

Using MIMIC-IV Data

Prerequisites

1. Download MIMIC-IV to agentic-ehr/datasets/ directory ([TODO]: download mimic-iv-ed?).
2. Install the mimic extra requirements: pip install -e ".[mimic]".

Tasks Two groups of prediction tasks are trained on MIMIC-IV on a shared feature set, playing complementary roles in the health summary.

1. Chronic panel (context). A set of binary classifiers that identify which chronic conditions the patient currently has, painting the patient's present health picture. To prevent label leakage, each chronic task drops the feature columns that define its own target, including the previous diagnosis and definitive lab test results. For example, the "diabetes" task drops HbA1c, glucose, and diabetes codes. All per-target exclusions are declared in DX_DEFINING_FEATURES.

• Open question: Do we really need to exclude the defining labs? Clinicians themselves rely on these "definitive" results to diagnose, so dropping them may make the model harder than the real clinical setting. Worth revisiting.

2. Forward panel (future). Forward-looking outcomes that drive advice.

The following table lists detailed tasks included in these two panels. All are defined in data/mimic/tasks.py.

Panel	Task	Type	Input observed up to
forward	1-year mortality, 30-day readmission	binary	discharge
forward	prolonged stay (≥7d)	binary	admission + 24h
forward	length of stay	regression (days)	admission + 24h
chronic	diabetes, hypertension, hyperlipidemia, cardiovascular, respiratory, depression/anxiety	binary	discharge

Observation window & inputs. One index admission per adult survivor (first hospital stay containing an ICU stay). Each task declares an observation window — the point up to which its input features may be observed, a TaskSpec.window given in hours after admission (None = up to discharge):

• Most tasks observe up to discharge (window = None).
• Length-of-stay tasks (prolonged_stay, los_days) observe only the first 24h (window = 24), because predicting how long a stay lasts must not see end-of-stay data — otherwise the task is circular. The build emits one events file per distinct window (events.parquet, events_24h.parquet) and each task trains/infers on its own window.

Shared features (per window): vital-sign summary statistics, a lab panel (latest value), prior-admission comorbidity history, prior-admission count, and demographics. See No data leakage for the additionally dropped label-defining columns (e.g. HbA1c/glucose for the diabetes task).

Run it (see the walkthrough below):

# 1) Build the FEMR-style events + per-task label tables (one-time, ~5 min).
python -m agentic_ehr.data.mimic.build --root agentic-ehr/datasets/physionet.org/files/mimiciv/3.1 --out-dir artifacts/mimic

# 2) Train all tasks (dispatched by data.source: mimic).
agentic-ehr --config config/mimic.yaml train

# 3) Per-task held-out metrics (no LLM call).
agentic-ehr --config config/mimic.yaml evaluate

# 4) Prediction panel + LLM health report for one patient (needs GEMINI_API_KEY).
agentic-ehr --config config/mimic.yaml demo --patient-id <hadm_id>
#    optionally persist the record (predictions + logits + features on top of the report):
agentic-ehr --config config/mimic.yaml demo --save-response artifacts/responses --response-format json,md

Each patient becomes a HealthRiskProfile — a panel of per-task predictions (each with its probability/point-estimate, confidence, and SHAP-attributed contributors) sharing one snapshot. The agent reads it exactly like a single RiskProfile; --save-response DIR writes the metadata, per-task logits, feature input, and report for clinician review, to:

DIR/<dataset>/<provider>_<patient_id>.<ext>
# e.g. artifacts/responses/mimic/claude_29079034.json  (+ .md / .txt per --response-format)

where <dataset> is data.source (mimic / synthetic / femr) and <provider> is agent.llm.provider (gemini / claude / openai).

---

Replacing the baseline model

The agent depends on RiskProfile, not on XGBoost. To plug in a new model:

1. Implement the BaseModel interface (src/agentic_ehr/models/base.py): fit, predict_output (returns ModelOutput with a probability or a regression point estimate + uncertainty), feature_importance, feature_names, save/load. predict_proba is classification-only (the base class raises NotImplementedError by default, which regression models keep).
2. Register it:

   from agentic_ehr.models.registry import register_model
   register_model("<new_model_name>", MotorTModel)

3. Set model.name: new_model_name in the config and add a loader branch in pipeline._load_model.

Because attributions flow through the same Attributor → RiskProfileBuilder path, the agent, guardrails, and evaluation all keep working unchanged. For a model without feature attributions, return an empty feature_importance() and the agent will gracefully produce a summary without a contributors list.

---

No data leakage (temporal integrity)

A prediction model must never see its answer during training.

• Temporal causality — features use only information available strictly before the prediction time (or within a window that ends at it). Nothing recorded afterward may enter the inputs.
• No label-defining inputs — any signal used to define a label is excluded from that task's features. E.g. the "diabetes" task drops HbA1c, glucose, and diabetes codes; otherwise the task is circular.
• No future / same-episode outcome leakage — discharge diagnoses, discharge time, and death flags are labels, never features; comorbidity history is taken only from prior admissions.
• Train and inference use the identical, leakage-free feature set, built by the same code path.

---

Evaluation

• Predictive (eval/model_eval.py): AUROC, AUPRC, Brier, ECE (classification) and MAE / RMSE / R² (regression, e.g. length of stay).
• Summary quality (eval/summary_eval.py), pragmatic automatic checks:
  • factual consistency — stated risk % matches the RiskProfile;
  • no unsupported claims — no banned diagnostic/overclaiming phrases, and named contributors are grounded in the profile;
  • uncertainty faithfulness — low model confidence is acknowledged;
  • clarity — readability proxy + all sections present.

These are heuristic proxies, not a substitute for clinical review.

---

Project layout

config/default.yaml          configuration (all tunables)
src/agentic_ehr/             package (data, models, explain, agent, eval, cli)
tests/                       pytest suite