Skip to content

Add a "total" field to All List APIs (HTTP & Python SDK) #7911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

kswen
Copy link

@kswen kswen commented May 28, 2025

What problem does this PR solve?

Add a "total" field to all List APIs (HTTP & Python SDK) to resolve pagination inaccuracies when integrating with third-party systems.

1. HTTP API

All list endpoints (datasets, documents, chunks, chat assistants, agent sessions, chat sessions) now return a response with the following top-level fields:

  • total: Total number of items matching the query.
  • page: Current page number.
  • page_size: Number of items per page.
  • The list of objects (e.g., datasets, docs, chunks, chats, agents, sessions).
  • Backend service layer updated to ensure correct calculation of total before pagination is applied.

2. Python SDK Updates

All list_* methods in the SDK (e.g., list_datasets, list_documents, list_chunks, list_chats, list_agents, list_sessions) now return a dictionary with:

  • total, page, page_size, and the relevant list.
  • All relevant SDK modules (ragflow.py, modules/dataset.py, modules/document.py, modules/chat.py, modules/agent.py) updated to match the new API structure.

3. Test Suite Updates

  • HTTP API tests: All tests for list endpoints now assert the presence and type of total, page, and page_size in the response, and check the list length via the new structure.
  • Python SDK tests: All SDK tests for list methods updated to check for the new dict structure and validate the types of the new fields.

4. Documentation Updates

HTTP API Reference: All relevant list endpoint sections updated to show the new response structure, including example JSON responses and field descriptions.

Python API Reference:

  • Migration note added at the top, explaining the new return structure for all list methods.
  • All method signatures, return value descriptions, and code examples updated to show the new dict structure.
  • Example code blocks now demonstrate how to access the list and pagination fields.
  • Minor corrections and improvements to code examples and parameter descriptions for clarity and accuracy.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 💞 feature Feature request, pull request that fullfill a new feature. 📖 documentation Improvements or additions to documentation labels May 28, 2025
@yingfeng yingfeng added the ci Continue Integration label May 28, 2025
@yingfeng
Copy link
Member

Thanks, please make sure to pass API test in CI

kswen added 2 commits May 29, 2025 10:27
…he response dictionary using the correct key, matching the new API response structure.
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 29, 2025
@kswen
Copy link
Author

kswen commented May 29, 2025

@yingfeng Hey, the failed test cases have been updated; please give it another CI shot, thanks!

@kswen
Copy link
Author

kswen commented May 29, 2025

All relevant test cases have been updated again.
waiting for give it another ci shot

@kswen
Copy link
Author

kswen commented May 29, 2025

@yingfeng
If resources are available, please run another CI build. I can fix any remaining issues ASAP

@KevinHuSh KevinHuSh removed the ci Continue Integration label May 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📖 documentation Improvements or additions to documentation 💞 feature Feature request, pull request that fullfill a new feature. size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants