Skip to content

Conversation

@kswen
Copy link

@kswen kswen commented May 28, 2025

What problem does this PR solve?

Add a "total" field to all List APIs (HTTP & Python SDK) to resolve pagination inaccuracies when integrating with third-party systems.

1. HTTP API

All list endpoints (datasets, documents, chunks, chat assistants, agent sessions, chat sessions) now return a response with the following top-level fields:

  • total: Total number of items matching the query.
  • page: Current page number.
  • page_size: Number of items per page.
  • The list of objects (e.g., datasets, docs, chunks, chats, agents, sessions).
  • Backend service layer updated to ensure correct calculation of total before pagination is applied.

2. Python SDK Updates

All list_* methods in the SDK (e.g., list_datasets, list_documents, list_chunks, list_chats, list_agents, list_sessions) now return a dictionary with:

  • total, page, page_size, and the relevant list.
  • All relevant SDK modules (ragflow.py, modules/dataset.py, modules/document.py, modules/chat.py, modules/agent.py) updated to match the new API structure.

3. Test Suite Updates

  • HTTP API tests: All tests for list endpoints now assert the presence and type of total, page, and page_size in the response, and check the list length via the new structure.
  • Python SDK tests: All SDK tests for list methods updated to check for the new dict structure and validate the types of the new fields.

4. Documentation Updates

HTTP API Reference: All relevant list endpoint sections updated to show the new response structure, including example JSON responses and field descriptions.

Python API Reference:

  • Migration note added at the top, explaining the new return structure for all list methods.
  • All method signatures, return value descriptions, and code examples updated to show the new dict structure.
  • Example code blocks now demonstrate how to access the list and pagination fields.
  • Minor corrections and improvements to code examples and parameter descriptions for clarity and accuracy.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 💞 feature Feature request, pull request that fullfill a new feature. 📖 documentation Improvements or additions to documentation labels May 28, 2025
@yingfeng yingfeng added the ci Continue Integration label May 28, 2025
@yingfeng
Copy link
Member

Thanks, please make sure to pass API test in CI

kswen added 2 commits May 29, 2025 10:27
…he response dictionary using the correct key, matching the new API response structure.
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 29, 2025
@kswen
Copy link
Author

kswen commented May 29, 2025

@yingfeng Hey, the failed test cases have been updated; please give it another CI shot, thanks!

@kswen
Copy link
Author

kswen commented May 29, 2025

All relevant test cases have been updated again.
waiting for give it another ci shot

@kswen
Copy link
Author

kswen commented May 29, 2025

@yingfeng
If resources are available, please run another CI build. I can fix any remaining issues ASAP

@KevinHuSh KevinHuSh removed the ci Continue Integration label May 30, 2025
@KevinHuSh
Copy link
Collaborator

Appreciations!
By our evaluations, this feature is not going to be merged into main branch.

@KevinHuSh KevinHuSh closed this Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continue Integration 📖 documentation Improvements or additions to documentation 💞 feature Feature request, pull request that fullfill a new feature. size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants