Feat: Optimize database performance and fix dataset regeneration UI #93
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Backend Performance Improvements: - Added completed_rows column to track generation progress - Optimized database queries for faster UI loading - Enhanced use_case column population from backend API Frontend Regeneration Fixes: - Fixed Examples component to render immediately during regeneration - Eliminated unnecessary API calls during dataset regeneration (performance boost) - Resolved Form.useWatch timing issues by using form.getFieldsValue(true) pattern - Added regeneration detection across all DataGenerator components - Prevented API calls for topics, examples, prompts, and model params during regeneration Key Technical Changes: - Examples.tsx: Fixed data source logic using same pattern as Summary.tsx - Configure.tsx: Added regeneration detection to prevent inference_type override - Prompt.tsx: Conditional API fetching based on regeneration state - Parameters.tsx: Skip model params API during regeneration - UseCaseSelector.tsx: Prevent use cases API call during regeneration - API hooks: Enhanced with conditional URL generation and loading states Successfully tested: ✅ Examples render immediately during regeneration (no loading spinner) ✅ All form fields populate correctly from existing dataset data ✅ No unnecessary API calls during regeneration ✅ Original dataset creation functionality preserved ✅ Database performance improvements for swift UI loading Maintains backward compatibility while significantly improving user experience.
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes backend database performance and fixes frontend dataset regeneration issues. The key focus is improving the user experience during dataset regeneration by eliminating unnecessary API calls and ensuring proper data rendering.
- Added
completed_rowscolumn to track dataset generation progress and created a lightweight query for better performance - Fixed dataset regeneration UI by using existing form data instead of triggering API calls for topics, examples, prompts, and model parameters
- Enhanced use case handling by replacing hardcoded translations with dynamic API-based lookup system
Reviewed Changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| app/main.py | Switched to lightweight database query for generation history |
| app/core/database.py | Added optimized query method with completed_rows tracking |
| app/client/src/pages/DataGenerator/* | Fixed regeneration detection and conditional API fetching across all components |
| app/client/src/pages/Home/* | Added completed_rows column display and migrated to dynamic use case mapping |
| app/client/src/api/* | Enhanced hooks with conditional fetching and added use case mapping functionality |
Comments suppressed due to low confidence (1)
app/client/src/pages/DataGenerator/Prompt.tsx:252
- Fixed the typo from 'JSAON' to 'JSON' correctly.
label={
- Fixed test_generation_history to mock get_paginated_generate_metadata_light instead of get_paginated_generate_metadata - The API endpoint uses get_paginated_generate_metadata_light for better performance - All integration tests now pass successfully
Co-authored-by: Copilot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backend Performance Improvements:
Frontend new components added/modified
Frontend Regeneration Fixes:
Key Technical Changes:
Successfully tested:
✅ Examples render immediately during regeneration (no loading spinner) ✅ All form fields populate correctly from existing dataset data ✅ No unnecessary API calls during regeneration
✅ Original dataset creation functionality preserved ✅ Database performance improvements for swift UI loading.
✅ Backend testing and integration testing.
Maintains backward compatibility while significantly improving user experience.