Skip to content

feat(output): add origin_top_logprobs field in vllm custom api - #458

Open
ivanbao9783 wants to merge 1 commit into
AISBench:masterfrom
ivanbao9783:feat/framework_out_logprob
Open

feat(output): add origin_top_logprobs field in vllm custom api#458
ivanbao9783 wants to merge 1 commit into
AISBench:masterfrom
ivanbao9783:feat/framework_out_logprob

Conversation

@ivanbao9783

@ivanbao9783 ivanbao9783 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献,我们非常重视。以下说明将使您的拉取请求更健康,更易于获得反馈。如果您不理解某些项目,请不要担心,只需提交拉取请求并从维护人员那里寻求帮助即可。

PR Type / PR类型

  • Feature(功能新增)
  • Bugfix(Bug 修复)
  • Docs(文档更新)
  • CI/CD(持续集成/持续部署)
  • Refactor(代码重构)
  • Perf(性能优化)
  • Dependency(依赖项更新)
  • Test-Cases(测试用例更新)
  • Other(其他)

Related Issue | 关联 Issue
Fixes #457

🔍 Motivation / 变更动机

Z客户现场诉求:
当前AISBench精度测试仅显示分数结果,难以反应真实的模型能力,且无法提供足够的信息支撑精度问题定位,需要补充单case的input/output token(推理效率&成本估算)、logprob(置信度&幻觉维度)

📝 Modification / 修改内容

  1. 精度测试单 case 增加 input/output token 信息落盘到 prediction.json
  2. vllm 接口配置 + 非 stream 模型 场景下,支持通过参数 获取logprob信息并落盘 prediction.json

【精度评测(落盘文件 *_predictions.jsonl)】

model_cfg stream logprobs参数 token信息 logprob信息
vllm_api_general False True Yes Yes
vllm_api_general False False Yes No
vllm_api_general_chat False True Yes Yes
vllm_api_general_chat False False Yes No
vllm_api_general_stream True True Yes No
vllm_api_general_stream True False Yes No
vllm_api_stream_chat True True Yes No
vllm_api_stream_chat True False Yes No
vllm_api_stream_chat_multiturn True True Yes No

【性能评测(落盘文件 → *_details.jsonl)】

model_cfg stream logprobs参数 token信息 logprob信息
vllm_api_general False True Yes Yes
vllm_api_general False False Yes No
vllm_api_general_chat False True Yes Yes
vllm_api_general_chat False False Yes No
vllm_api_general_stream True True Yes No
vllm_api_general_stream True False Yes No
vllm_api_stream_chat True True Yes No
vllm_api_stream_chat True False Yes No
vllm_api_stream_chat_multiturn True True Yes No

📐 Associated Test Results / 关联测试结果

image

⚠️ BC-breaking (Optional) / 向后不兼容变更(可选)

Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改?如果是,请描述它如何破坏兼容性,以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。

⚠️ Performance degradation (Optional) / 性能下降(可选)

If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降,请描述性能下降的影响和预期的性能改进。

🌟 Use cases (Optional) / 使用案例(可选)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能,最好在此处列出一些用例并更新文档。

✅ Checklist / 检查列表

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖,导致 Bug 的情况应在单元测试中添加。
  • The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是,请添加更多单元测试以确保正确性。
  • All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档(API 文档、文档字符串、示例教程)已更新以反映这些更改。

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects. / 如果此拉取请求对下游或其他相关项目有潜在影响,应在那些项目中测试此 PR。
  • CLA has been signed and all committers have signed the CLA in this PR. / CLA 已签署,且本 PR 中的所有提交者均已签署 CLA。

👥 Collaboration Info / 协作信息

  • Suggested Reviewers / 建议审核人: @xxx
  • Relevant Module Owners / 相关模块负责人: @xxx
  • Other Collaboration Notes / 其他协作说明:

🌟 Useful CI Command / 实用的CI命令

Command / 命令 Introduction / 介绍
/gemini review Performs a code review for the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 执行代码审核。
/gemini summary Provides a summary of the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 提供摘要。
/gemini help Displays a list of available commands of Gemini. / 显示 Gemini 可用命令的列表。
/readthedocs build Triggers a build of the documentation for the current pull request in its current state by Read the Docs. / 触发当前拉取请求在当前状态下由 Read the Docs 构建文档。
/pr_check 手动重新触发 PR 合入质量检查工作流(PR Quality Check),并在 PR 上评论即可生效。 / Manually re-runs the PR Quality Check workflow by commenting on the pull request.

@ivanbao9783
ivanbao9783 temporarily deployed to smoke-test-approval August 12, 2026 09:25 — with GitHub Actions Inactive
@github-actions github-actions Bot added feature test-cases no_review PR 未包含任何检视意见,合入前必须包含检视意见 labels Aug 12, 2026
@github-actions

Copy link
Copy Markdown

🚫 PR 合入质量检查未通过,PR 已被禁止合入。

未通过项:no_review

  • 无检视意见:请至少邀请一位 reviewer 提交检视意见(Approve / Request changes / Comment 均可)。

请按上述提示整改后,重新推送(push)或评论本 PR 即可触发再次检查。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature no_review PR 未包含任何检视意见,合入前必须包含检视意见 test-cases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[需求] AISBench精度测评维度补充

1 participant