feat: Parse API response errors when max_completions_tokens used and provide helpful messages to the user

[lkomali]

This PR improves UX in case of API response errors when max_completions_tokens used.

Summary by CodeRabbit

• New Features

  • Added intelligent API error diagnostics that automatically detect backend errors and display detailed console panels with problem descriptions, root causes, investigation steps, and recommended fixes.

• Tests

  • Added comprehensive tests for error detection and console diagnostics rendering.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This pull request introduces API error detection and console reporting capabilities for AIPerf. It adds a new enum member to ConsoleExporterType, creates a new error detection module with three classes (ErrorInsight, MaxCompletionTokensDetector, ConsoleApiErrorExporter), updates public exports, and includes comprehensive unit tests.

Changes

Cohort / File(s)	Summary
Enum Extension src/aiperf/common/enums/data_exporter_enums.py	Added API_ERRORS = "api_errors" member to ConsoleExporterType enum.
New Error Detection Module src/aiperf/exporters/console_api_error_exporter.py	Introduced three new classes: ErrorInsight (data model), MaxCompletionTokensDetector (static analyzer for max_completion_tokens backend errors), and ConsoleApiErrorExporter (console exporter implementing ConsoleExporterProtocol). Includes error detection logic, Rich panel rendering, and human-readable text formatting.
Public API Updates src/aiperf/exporters/__init__.py	Added imports and exports for ConsoleApiErrorExporter, ErrorInsight, and MaxCompletionTokensDetector to public API.
Unit Tests tests/unit/exporters/test_console_api_error_exporter.py	New test suite validating detector behavior (max_completion_tokens pattern matching, None returns for unrelated errors), exporter rendering (Rich panel creation, content validation), and edge cases (empty error summaries).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• Detector logic: Verify the MaxCompletionTokensDetector.detect() method correctly identifies the specific error pattern and extracts all relevant fields.
• Formatting logic: Review _format_text() method for correctness of error message construction and Rich markup syntax.
• Test coverage: Confirm unit tests adequately validate detector and exporter behavior, including edge cases like missing error_details and empty summaries.
• Integration points: Ensure the new exporter integrates correctly with ConsoleExporterProtocol and the detector chain architecture.

Poem

🐰 A new error detector hops into place,
Max tokens now caught with diagnostic grace,
Rich panels tell tales of what went awry,
So developers debug with a twinkle and sigh!
AIPerf's got wisdom—now users can see,
The path to fix backend, as clear as can be! ✨

Pre-merge checks

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding detection and helpful messaging for API errors when max_completion_tokens is used, which aligns with the new ConsoleApiErrorExporter and detector implementation.

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

• Provide your own instructions using the high_level_summary_instructions setting.
• Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
• Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

1. 📝 Description — Summarize the main change in 50–60 words, explaining what was done.
2. 📓 References — List relevant issues, discussions, documentation, or related PRs.
3. 📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.
4. 📊 Contributor Summary — Include a Markdown table showing contributions:
   | Contributor | Lines Added | Lines Removed | Files Changed |
5. ✔️ Additional Notes — Add any extra reviewer context.
   Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[lkomali]

@CodeRabbit start review

[codecov]

Codecov Report

❌ Patch coverage is 94.82759% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/aiperf/exporters/console_api_error_exporter.py	94.64%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

@lkomali: I'll start reviewing the changes in this PR now.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (3)

tests/unit/exporters/test_console_api_error_exporter.py (1)

75-122: Consider moving asyncio imports to the top.

The tests provide excellent coverage of the exporter's panel rendering and skip behavior. The inline import asyncio statements (lines 88, 118) could be moved to the top of the file for consistency with standard Python practices.

Apply this diff to move the import to the top:

 import json
 from unittest.mock import MagicMock
 
+import asyncio
 import pytest
 from rich.console import Console

Then remove the inline imports:

-    import asyncio
-
     asyncio.run(exporter.export(mock_console))

src/aiperf/exporters/console_api_error_exporter.py (2)

19-35: Consider using @DataClass for ErrorInsight.

The current implementation is functional, but using @dataclass would provide automatic __repr__, __eq__, and reduce boilerplate.

Apply this diff to refactor as a dataclass:

+from dataclasses import dataclass
+
+@dataclass
 class ErrorInsight:
     """Model to describe a detected API error insight."""
-
-    def __init__(
-        self,
-        title: str,
-        problem: str,
-        causes: list[str],
-        investigation: list[str],
-        fixes: list[str],
-    ):
-        self.title = title
-        self.problem = problem
-        self.causes = causes
-        self.investigation = investigation
-        self.fixes = fixes
+    
+    title: str
+    problem: str
+    causes: list[str]
+    investigation: list[str]
+    fixes: list[str]

---

49-58: Consider more specific exception handling.

Lines 51-52 use contextlib.suppress(Exception) which silently catches all exceptions during JSON parsing. While this prevents crashes, it could hide legitimate errors like encoding issues or malformed JSON that might indicate problems upstream.

Consider catching specific exceptions:

-            parsed = None
-            with contextlib.suppress(Exception):
-                parsed = json.loads(raw_msg)
+            parsed = None
+            try:
+                parsed = json.loads(raw_msg)
+            except (json.JSONDecodeError, TypeError):
+                # Expected if raw_msg isn't valid JSON
+                pass

Additionally, line 58's str() conversion is redundant since raw_msg is already a string from line 49 (err.message or "").

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3ac7729 and dca9e1d.

📒 Files selected for processing (4)

• src/aiperf/common/enums/data_exporter_enums.py (1 hunks)
• src/aiperf/exporters/__init__.py (2 hunks)
• src/aiperf/exporters/console_api_error_exporter.py (1 hunks)
• tests/unit/exporters/test_console_api_error_exporter.py (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (3)

tests/unit/exporters/test_console_api_error_exporter.py (2)

src/aiperf/exporters/console_api_error_exporter.py (4)

• ConsoleApiErrorExporter (91-138)
• MaxCompletionTokensDetector (37-86)
• detect (39-86)
• export (102-118)

tests/unit/exporters/test_metrics_base_exporter.py (1)

• exporter_config (67-76)

src/aiperf/exporters/__init__.py (1)

src/aiperf/exporters/console_api_error_exporter.py (3)

• ConsoleApiErrorExporter (91-138)
• ErrorInsight (19-34)
• MaxCompletionTokensDetector (37-86)

src/aiperf/exporters/console_api_error_exporter.py (4)

src/aiperf/common/enums/data_exporter_enums.py (1)

• ConsoleExporterType (7-14)

src/aiperf/common/factories.py (1)

• ConsoleExporterFactory (402-418)

src/aiperf/common/protocols.py (1)

• ConsoleExporterProtocol (312-320)

tests/unit/exporters/test_metrics_base_exporter.py (1)

• exporter_config (67-76)

🪛 Ruff (0.14.5)

src/aiperf/exporters/console_api_error_exporter.py

94-96: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)

• GitHub Check: integration-tests (macos-latest, 3.11)
• GitHub Check: integration-tests (macos-latest, 3.12)
• GitHub Check: integration-tests (macos-latest, 3.13)
• GitHub Check: integration-tests (ubuntu-latest, 3.12)
• GitHub Check: integration-tests (ubuntu-latest, 3.11)
• GitHub Check: integration-tests (macos-latest, 3.10)
• GitHub Check: integration-tests (ubuntu-latest, 3.13)
• GitHub Check: integration-tests (ubuntu-latest, 3.10)
• GitHub Check: build (macos-latest, 3.13)
• GitHub Check: build (macos-latest, 3.10)
• GitHub Check: build (ubuntu-latest, 3.13)

🔇 Additional comments (7)

src/aiperf/common/enums/data_exporter_enums.py (1)

8-8: LGTM!

The new API_ERRORS enum member is well-positioned and follows the existing naming conventions.

src/aiperf/exporters/__init__.py (1)

11-15: LGTM!

The new exports are correctly structured. Since this file is auto-generated by mkinit, the changes properly reflect the new module additions.

tests/unit/exporters/test_console_api_error_exporter.py (3)

16-44: LGTM!

The mock classes and fixtures provide clear test infrastructure that matches the expected error structure from the API.

---

47-58: LGTM!

Good coverage of the detection logic, verifying both the insight creation and key content presence.

---

61-72: LGTM!

Excellent coverage of edge cases where detection should not trigger.

src/aiperf/exporters/console_api_error_exporter.py (2)

102-118: LGTM!

The export logic is well-structured: it iterates through detectors, renders insights as Rich panels when detected, and flushes output appropriately.

---

120-138: LGTM!

The formatting logic produces clear, well-structured diagnostic output with proper Rich markup for emphasis and readability.

[ajcasagrande]

Well done! A few nits, but it looks great, thanks!

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@1bcb3dd985cd5bb5472eedd85ef19d6d8edfd8c6

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@1bcb3dd985cd5bb5472eedd85ef19d6d8edfd8c6

---

Last updated for commit: 1bcb3dd • Browse code