Clarify data validation philosophy: lenient import, strict export - #236
Merged
ryan-mahoney merged 1 commit intoJan 25, 2026
Merged
Conversation
30 tasks
Copilot
AI
changed the title
[WIP] WIP address feedback on data validation in batched import
Clarify data validation philosophy: lenient import, strict export
Jan 25, 2026
ryan-mahoney
marked this pull request as ready for review
January 25, 2026 21:28
ryan-mahoney
merged commit Jan 25, 2026
e5a7c94
into
feat/gtfs/221-load-schedule-tables
1 check passed
ryan-mahoney
added a commit
that referenced
this pull request
Jan 25, 2026
* feat(gtfs): add schedule tables for calendars, trips, and stop times Add support for GTFS schedule data including: - Calendar and CalendarDate schemas and migrations - Trip schema and migration with route pattern support - StopTime schema and migration for schedule sequences - Import functionality for all schedule tables - Documentation for calendar, trip, and stop_time extensions This enables importing and managing service calendars, trip schedules, and stop time sequences which are essential for route planning and schedule management in the GTFS planner. * feat(gtfs): implement batched import for schedule tables - Replace individual inserts with `Repo.insert_all` batch processing - Add `BatchProcessor` and `RowParser` for improved performance - Implement real-time progress reporting via PubSub - Update `ImportLive` to run imports asynchronously with progress UI - Add support for importing trips, stop_times, calendars, and more Closes #227 * Update lib/gtfs_planner/gtfs/import/row_parser.ex Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update lib/gtfs_planner/gtfs/import.ex Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix PubSub subscription timing for real-time import progress (#233) * Initial plan * fix: subscribe to PubSub topic before async import task starts - Modified import_files to accept optional topic parameter - Generate topic and subscribe in LiveView before starting async task - Remove redundant subscription in handle_info callback - This ensures all progress messages are received during import Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Remove dead Multi-based error handler from import LiveView (#234) * Initial plan * Remove dead Multi-based error handling code Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Fix error handling for BatchProcessor map errors in ImportLive (#235) * Initial plan * Add map error handling for BatchProcessor errors Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: extract common error assignment logic to helper function Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Fix: prevent stack overflow in error message extraction Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: improve code readability with pattern matching Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: use descriptive variable names for clarity Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> Co-authored-by: Ryan Mahoney <ryan.mahoney@outlook.com> * Initial plan (#236) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com>
themightychris
pushed a commit
that referenced
this pull request
Apr 12, 2026
* feat(gtfs): add schedule tables for calendars, trips, and stop times Add support for GTFS schedule data including: - Calendar and CalendarDate schemas and migrations - Trip schema and migration with route pattern support - StopTime schema and migration for schedule sequences - Import functionality for all schedule tables - Documentation for calendar, trip, and stop_time extensions This enables importing and managing service calendars, trip schedules, and stop time sequences which are essential for route planning and schedule management in the GTFS planner. * feat(gtfs): implement batched import for schedule tables - Replace individual inserts with `Repo.insert_all` batch processing - Add `BatchProcessor` and `RowParser` for improved performance - Implement real-time progress reporting via PubSub - Update `ImportLive` to run imports asynchronously with progress UI - Add support for importing trips, stop_times, calendars, and more Closes #227 * Update lib/gtfs_planner/gtfs/import/row_parser.ex Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update lib/gtfs_planner/gtfs/import.ex Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix PubSub subscription timing for real-time import progress (#233) * Initial plan * fix: subscribe to PubSub topic before async import task starts - Modified import_files to accept optional topic parameter - Generate topic and subscribe in LiveView before starting async task - Remove redundant subscription in handle_info callback - This ensures all progress messages are received during import Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Remove dead Multi-based error handler from import LiveView (#234) * Initial plan * Remove dead Multi-based error handling code Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Fix error handling for BatchProcessor map errors in ImportLive (#235) * Initial plan * Add map error handling for BatchProcessor errors Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: extract common error assignment logic to helper function Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Fix: prevent stack overflow in error message extraction Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: improve code readability with pattern matching Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> * Refactor: use descriptive variable names for clarity Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com> Co-authored-by: Ryan Mahoney <ryan.mahoney@outlook.com> * Initial plan (#236) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: ryan-mahoney <988609+ryan-mahoney@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addressed review feedback on PR #228 regarding
direction_idvalidation in route patterns. The codebase intentionally accepts non-standard data during import and validates at export time.Context
The bot reviewer suggested making
direction_idrequired during import (usingextract_required), but this conflicts with the design philosophy:Changes
No code changes required. Current implementation correctly allows nil values:
Schema validation via
RoutePattern.changeset/2enforces requirements when creating records for export, not during raw import.💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.