Skip to content

Commit ea315c6

Browse files
committed
docs(readme): tighten mvp delivery copy
Clarify the README opening, add workflow diagrams, and make the public docs read less like an internal status report. Keep the checked-in run summary aligned with the report writer template. Tested: - uv run ruff check . - uv run ruff format --check . - uv run basedpyright - uv run pytest -v - uv run pytest --cov=clean_data_export_api --cov-report=term-missing --cov-fail-under=90 - uv run python -m compileall src tests - git diff --check
1 parent 600d349 commit ea315c6

7 files changed

Lines changed: 135 additions & 136 deletions

File tree

‎README.md‎

Lines changed: 39 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,11 @@
11
# Clean Data Export API
22

3-
Clean Data Export API is a local workflow for turning imperfect job records into
4-
review-ready CSV and JSON exports, with duplicate handling, rejected rows, run
5-
history, and handoff notes.
3+
Clean Data Export API turns incomplete FieldOps Desk job exports into files a
4+
team can review: clean CSV, matching JSON, rejected rows, run history, and a
5+
short run summary.
66

7-
The workflow is intentionally local and uses fictional data. It models an
8-
operations tool called **FieldOps Desk** and focuses on exporting `jobs` and
9-
related `job_updates` records. It does not connect to a real customer system or
10-
use real credentials.
11-
12-
## Current Status
13-
14-
The first local MVP is implemented. The repository includes deterministic
15-
FieldOps Desk fixture data, a reusable export service, FastAPI and CLI entry
16-
points, clean CSV/JSON outputs, rejected rows, SQLite run history, and tests.
7+
The project is local by design. It uses fictional data, reads only `jobs` and
8+
`job_updates`, and never connects to a real customer system.
179

1810
The demo export for `2026-06-01` through `2026-06-05` returns:
1911

@@ -24,44 +16,45 @@ duplicate_count=1
2416
rejected_count=6
2517
```
2618

27-
## What the workflow delivers
19+
## What it does
2820

29-
This is not a broad backend platform. The workflow focuses on one handoff-ready
30-
export package:
21+
This is one export workflow, not a broad backend platform. It:
3122

32-
- pull a limited set of source records.
33-
- map source fields into a buyer-friendly output shape.
34-
- keep invalid records out of clean exports without hiding them.
35-
- remove duplicate records deterministically.
36-
- write output files that can be reviewed in a spreadsheet or handed to another
23+
- pulls a limited set of source records.
24+
- maps source fields into clean output columns.
25+
- keeps invalid records out of the clean export without hiding them.
26+
- removes later duplicate records for the same `job_id`.
27+
- writes files that can be reviewed in a spreadsheet or handed to another
3728
system.
38-
- record enough run history to explain what happened.
29+
- records run history in SQLite.
3930

4031
## Scenario
4132

42-
A fictional operations team uses FieldOps Desk to manage field jobs. Their
43-
built-in export is limited, and manual spreadsheet updates have created missing
44-
fields and duplicate records.
33+
FieldOps Desk is a fictional operations tool for field jobs. Its export is
34+
limited, and manual spreadsheet edits have left missing fields and duplicate
35+
rows.
4536

46-
They need a repeatable local workflow that produces:
37+
The local workflow produces:
4738

4839
- a clean job export for review.
49-
- a JSON copy for technical handoff.
50-
- a rejected-row file explaining records that were not accepted.
40+
- a JSON copy with the same accepted records.
41+
- a rejected-row file that explains what failed.
5142
- a short run summary with counts and file paths.
5243

5344
## Workflow
5445

55-
```text
56-
source fixture
57-
-> paginated job fetch
58-
-> related job update fetch
59-
-> field mapping
60-
-> validation
61-
-> duplicate handling
62-
-> output files
63-
-> SQLite run history
64-
-> API/CLI response
46+
```mermaid
47+
flowchart LR
48+
source["FieldOps Desk fixture"]
49+
fetch["Fetch jobs and updates"]
50+
map["Map fields"]
51+
validate["Validate rows"]
52+
dedupe["Remove later duplicates"]
53+
write["Write CSV, JSON, and rejected rows"]
54+
history["Store run history"]
55+
response["Return API or CLI summary"]
56+
57+
source --> fetch --> map --> validate --> dedupe --> write --> history --> response
6558
```
6659

6760
The FastAPI endpoint and CLI use the same export service, so both entry points
@@ -77,12 +70,12 @@ outputs/
7770
run_summary.md
7871
```
7972

80-
`clean_jobs.csv` is the main spreadsheet-ready export. `clean_jobs.json` carries
81-
the same accepted records in JSON form. `rejected_jobs.csv` keeps invalid or
82-
duplicate records visible with reason codes. `run_summary.md` gives a
83-
human-readable handoff summary.
73+
`clean_jobs.csv` is the spreadsheet export. `clean_jobs.json` contains the same
74+
accepted records in JSON form. `rejected_jobs.csv` keeps invalid and duplicate
75+
records visible with reason codes. `run_summary.md` records the run counts and
76+
file paths.
8477

85-
## Project Structure
78+
## Project structure
8679

8780
```text
8881
clean-data-export-api/
@@ -116,7 +109,7 @@ clean-data-export-api/
116109
tests/
117110
```
118111

119-
## Usage
112+
## Run it
120113

121114
Run the CLI export:
122115

@@ -154,9 +147,8 @@ local runs.
154147

155148
## Safety and limits
156149

157-
This project does not use real third-party credentials, scraping, login bypass,
158-
paid APIs, or real customer data. All sample data should be fictional and safe
159-
to share.
150+
This project does not use real credentials, scraping, login bypass, paid APIs,
151+
or customer data. All sample data is fictional.
160152

161153
This project should not claim readiness for live operations, guaranteed business
162154
outcomes, advanced security guarantees, or support for a real vendor API before

‎docs/ARCHITECTURE.md‎

Lines changed: 33 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,31 @@
22

33
## Purpose
44

5-
This document describes the technical design for Clean Data Export API. The
6-
design keeps the workflow small, local, deterministic, and reusable from both
7-
FastAPI and the CLI.
5+
This document describes how Clean Data Export API is wired. The design keeps the
6+
workflow local, deterministic, and shared by both FastAPI and the CLI.
87

98
## System overview
109

11-
```text
12-
FastAPI / CLI
13-
-> export service
14-
-> source fixture API
15-
-> mapping
16-
-> validation
17-
-> duplicate handling
18-
-> report writers
19-
-> SQLite run repository
10+
```mermaid
11+
flowchart LR
12+
entry["FastAPI / CLI"]
13+
service["export_service.py"]
14+
source["source_api.py"]
15+
mapping["mapping.py"]
16+
validation["validation.py"]
17+
reports["reports.py"]
18+
repository["repository.py"]
19+
outputs["CSV / JSON / summary"]
20+
sqlite["SQLite run history"]
21+
22+
entry --> service
23+
service --> source
24+
service --> mapping
25+
service --> validation
26+
service --> reports
27+
service --> repository
28+
reports --> outputs
29+
repository --> sqlite
2030
```
2131

2232
The export service is the core boundary. API and CLI code should only parse
@@ -31,7 +41,7 @@ inputs, call the service, and return the summary.
3141
- Keep output files easy to inspect in a spreadsheet.
3242
- Avoid abstractions that are not needed for the first export type.
3343

34-
## Package Layout
44+
## Package layout
3545

3646
```text
3747
src/clean_data_export_api/
@@ -88,7 +98,7 @@ Expected settings:
8898

8999
### `models.py`
90100

91-
Defines typed data models for the workflow.
101+
Defines typed data models.
92102

93103
Models include:
94104

@@ -104,7 +114,7 @@ Models include:
104114

105115
### `source_api.py`
106116

107-
Simulates a small vendor API over local fixture files.
117+
Simulates a small source API over local fixture files.
108118

109119
Responsibilities:
110120

@@ -114,8 +124,8 @@ Responsibilities:
114124
- return related `job_updates`.
115125
- expose imperfect data for validation and duplicate tests.
116126

117-
This module should feel like a documented API client, but it must not call a
118-
real external service.
127+
This module should behave like an API client. It must not call a real external
128+
service.
119129

120130
### `mapping.py`
121131

@@ -161,12 +171,12 @@ Workflow:
161171
10. Remove the run history row if report writing fails.
162172
11. Return summary.
163173

164-
This module coordinates the workflow but should delegate specialized logic to
165-
mapping, validation, reports, and repository modules.
174+
This module coordinates the workflow. Mapping, validation, report writing, and
175+
SQLite writes stay in their own modules.
166176

167177
### `reports.py`
168178

169-
Writes handoff files.
179+
Writes output files.
170180

171181
Expected files:
172182

@@ -175,7 +185,7 @@ Expected files:
175185
- `rejected_jobs.csv`
176186
- `run_summary.md`
177187

178-
Report writing should be deterministic so tests can compare expected output.
188+
Report writing must be deterministic so tests can compare output bytes.
179189

180190
### `repository.py`
181191

@@ -339,5 +349,5 @@ Fields:
339349
Expected bad source data should become rejected rows.
340350

341351
Unexpected runtime failures, such as unreadable fixture files, invalid output
342-
paths, or SQLite write failures, should return clear errors with enough context
343-
to debug the local run.
352+
paths, or SQLite write failures, should keep enough context to debug the local
353+
run.

‎docs/DELIVERY.md‎

Lines changed: 17 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,10 @@
22

33
## Purpose
44

5-
This document defines the handoff package for the finished delivery.
5+
This document defines the files a reviewer should expect after a demo export.
66

7-
The delivery shows more than working code. It shows how a buyer would receive a
8-
clean export workflow: output files, rejected rows, a run summary, and enough
9-
notes to rerun or extend the work.
7+
The package includes the generated outputs, rejected rows, a run summary, and
8+
the notes needed to rerun or extend the workflow.
109

1110
## Delivery package
1211

@@ -34,7 +33,7 @@ outputs/
3433

3534
### `clean_jobs.csv`
3635

37-
Spreadsheet-ready export of accepted job records.
36+
Spreadsheet export of accepted job records.
3837

3938
Required columns:
4039

@@ -50,12 +49,12 @@ Required columns:
5049

5150
JSON version of the accepted job records.
5251

53-
The JSON file should contain the same records and fields as `clean_jobs.csv`.
54-
It exists for technical handoff, API-style review, or downstream processing.
52+
The JSON file contains the same records and fields as `clean_jobs.csv`. It is
53+
useful when the export needs to feed another tool.
5554

5655
### `rejected_jobs.csv`
5756

58-
Rejected-row export for records that were not written to clean outputs.
57+
Rows that were not written to clean outputs.
5958

6059
Required columns:
6160

@@ -74,12 +73,11 @@ Required reason codes:
7473
- `invalid_date`
7574
- `duplicate_record`
7675

77-
`raw_record` should preserve enough source data to understand what failed
78-
without requiring a debugger.
76+
`raw_record` should preserve enough source data to understand what failed.
7977

8078
### `run_summary.md`
8179

82-
Human-readable summary of the export run.
80+
Readable summary of one export run.
8381

8482
Required sections:
8583

@@ -92,9 +90,9 @@ Required sections:
9290
- known limits.
9391
- suggested next steps.
9492

95-
## Buyer-facing handoff standard
93+
## Review standard
9694

97-
The delivery should answer these questions without requiring a meeting:
95+
The delivery should answer these questions from the files alone:
9896

9997
- What does this workflow do?
10098
- What source data does it read?
@@ -106,9 +104,9 @@ The delivery should answer these questions without requiring a meeting:
106104
- How can the workflow be rerun?
107105
- What can be extended later?
108106

109-
## Technical handoff standard
107+
## Technical review standard
110108

111-
The repository should also make the implementation easy to review:
109+
The repository should also make the implementation easy to inspect:
112110

113111
- API and CLI entry points should call the same export service.
114112
- mapping and validation should be testable without FastAPI or Typer.
@@ -129,17 +127,16 @@ When implementation changes:
129127
criteria change.
130128
- update `docs/PRD.md` if scope or acceptance criteria change.
131129

132-
Do not add a new long-form document unless existing docs become too large to
133-
keep clear.
130+
Do not add another long document unless the current set becomes hard to read.
134131

135-
## Next-step options
132+
## Next steps worth considering
136133

137134
Realistic follow-up work may include:
138135

139136
- connect the workflow to a real documented API.
140137
- write directly to Google Sheets.
141138
- add scheduled runs.
142-
- add buyer-specific validation rules.
139+
- add project-specific validation rules.
143140
- support larger files.
144141
- deploy the workflow to a server.
145142
- add another export type.
@@ -156,7 +153,7 @@ Avoid promising:
156153

157154
The delivery is ready when:
158155

159-
- the README explains the buyer problem, current status, and run path.
156+
- the README explains the problem, implemented scope, and run path.
160157
- source fixtures include valid, invalid, and duplicate records.
161158
- the output files exist and match this contract.
162159
- invalid and duplicate records are visible in rejected output.

0 commit comments

Comments
 (0)