New HUD system for Wingman AI with independent HUD server and new HUD skill utilizing it. by SawPsyder · Pull Request #343 · ShipBit/wingman-ai

SawPsyder · 2026-02-02T19:33:28Z

This completely adds the HUD http server, the HUD renderer and HUD skill to Wingman AI as full core integrations.

This is marked as draft as it still needs several polish runs.
But the functionality is at 95% now - only missing HUD hide/show tools & HUD server start/stop as system setting change reaction.

TODO list:

Add HUD show/hide options as tool options
Polish Run skill
Polish Run http client
Polish run http server & manager
Polish readme files
Fully integrate server stop/start via settings change
Toggle to only allow on Windows (?)

Shackless · 2026-02-03T09:26:27Z

Aww, bummer this only works on Windows. We have other places in the code already where we check the OS and disable/skip certain features when not on Windows. Please do this for HUD, too. Check the OS very early in the chain and skip everything if not running on Windows. I'd expect no traces, server instances or initialized variables at all when running on MacOS. I'll make sure to hide everything HUD related in the MacOS UI, too. Please be very defensive with this, especially where HUD plugs into Core. Always expect it not to be available and initialized at all.

SawPsyder · 2026-02-03T10:33:47Z

Rendering the HUD is OS-level stuff.
So yeah - this one is for Windows and I have no way to develop and test a Mac or Linux version.

api/interface.py

templates/configs/settings.yaml

hud_server/platform/__init__.py

hud_server/rendering/__init__.py

hud_server/http_client.py

api/interface.py

Copilot

Pull request overview

This PR adds a comprehensive HUD (Heads-Up Display) server system to Wingman AI, providing HTTP-based control of transparent overlay windows for displaying messages, progress bars, chat, and persistent information. The implementation includes a FastAPI server, PIL-based overlay renderer (Windows only), layout management system, and both async/sync HTTP clients.

Changes:

Full HUD server module with HTTP API, state management, and rendering pipeline
Integration into wingman_core.py with lifecycle management and settings reactivity
HUD skill for controlling overlays from LLM tool calls
Configuration system with validation and runtime updates
Comprehensive test suite with visual and stress tests

Reviewed changes

Copilot reviewed 39 out of 42 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
wingman_core.py	Added HUD server lifecycle management, settings subscription, and platform check
wingmen/open_ai_wingman.py	Simplified tool response update logic (potential bug)
templates/configs/settings.yaml	Added hud_server configuration section
services/settings_service.py	Added hud_server settings event publishing
services/mcp_client.py	Enhanced SSE connection resilience with auto-reconnect
services/migrations/migration_200_to_210.py	Added hud_server settings migration
api/interface.py	Added HudServerSettings model to API interface
hud_server/*	Complete new module with server, client, renderer, layout manager
skills/hud/*	New HUD skill for overlay control

hud_server/tests/test_unicode_stress.py

hud_server/tests/run_tests.py

wingmen/open_ai_wingman.py

Copilot

Pull request overview

Copilot reviewed 39 out of 42 changed files in this pull request and generated 4 comments.

wingman_core.py

services/mcp_client.py

- Remove excessive debug logging from `hud_server`, `hud_manager`, and `skills/hud` to reduce console noise. - Replace `sys.stderr.write` with `_report_exception` in `overlay.py` for consistent error handling. - Reduce log noise for 404 errors in `hud_server` exception handler. - Clean up exception handling in `http_client` to suppress unnecessary retry logs.

…tion - Create `hud_server/constants.py` to centralize configuration, timeouts, limits, and API paths. - Update `HudServer` and `HudHttpClient` to utilize centralized constants for consistent behavior. - Enhance `HudHttpClientSync` with improved thread safety, locking, and event loop management during startup. - Add Pydantic field validation, constraints, and default values in `hud_server/models.py`. - Consolidate documentation into `hud_server/README.md` and `hud_server/API.md`, removing scattered README files. - Improve logging and error handling during server and overlay lifecycle events.

…ated documentation

… game test

…channel support (#10) * Initial plan * Implement render caching for HUD performance optimization Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Address code review feedback for HUD caching Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Mark deprecated legacy methods in overlay.py, document unified window system Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Add deprecation warnings to legacy methods in overlay.py Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Improve deprecated methods documentation clarity Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Optimize HUD rendering by removing redundant z-order updates for windows * Remove legacy HUD code and fix cache issues causing ghosting Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Add docstring to _init_fonts and complete code review Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Fix layout manager releasing slot before fade-out completes Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Refactor chat window handling and improve HUD rendering performance * Remove unused legacy chat window code and fix chat window integration Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Enhance Snake game with gradient snake body, combo system, and golden apples * Enable alpha channel support in HUD background colors Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Address code review feedback for alpha channel support Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Add emoji support and scroll fade effect to chat overlay rendering --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> Co-authored-by: Jan Winkler <jan@jmwinkler.de>

* Initial plan * Append consecutive same-sender chat messages instead of creating new entries When a message is added to a chat window from the same sender as the last message, the text is appended to the last message (separated by a space) instead of creating a new message entry with a repeated sender name. Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Update chat tests for same-sender message merging behavior - test_chat_markdown: alternate senders so each markdown feature renders separately - test_chat_auto_hide: use different sender for second message to prevent merging - test_chat_overflow: unique sender per message to prevent merging during overflow test - Add test_chat_message_merging: dedicated test for consecutive same-sender merging Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com>

* Initial plan * Add chat message update feature with message IDs - Add `id` field (UUID) to ChatMessage dataclass - Change send_chat_message to return message ID (or merged message's ID) - Add update_chat_message method to HudManager - Add ChatMessageResponse and ChatMessageUpdateRequest models - Add PUT /chat/message endpoint for updating messages by ID - Update POST /chat/message to return message_id in response - Update HTTP client (async + sync) with update_chat_message method - Update TestSession with update_chat_message helper - Update overlay handler to track message IDs and handle updates - Update state serialization/deserialization to include message IDs - Add test_chat_message_update test case - Update README documentation with message update API Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Address code review: keep sender/text required in ChatMessage, fix comment Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com>

* Initial plan * Use reactive WinEvent hook for HUD foreground management instead of polling Replace the removed polling loop with an event-driven approach using SetWinEventHook(EVENT_SYSTEM_FOREGROUND). The callback sets a thread-safe threading.Event flag, and the main render loop checks/clears it each frame, re-applying HWND_TOPMOST only when a foreground change is actually detected. This is both more performant (zero cost when no changes occur) and more responsive (reacts instantly to foreground changes) than the old 50ms poll. Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> * Add log line when HUD foreground change is detected and repositioned Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SawPsyder <44277746+SawPsyder@users.noreply.github.com>

…t family and font size rendering

…s for xvasynth and pocket TTS

…input

… classes for improved clarity and maintainability

…te tracking

…pport

…HUD HTTP client

…ry with dummy tool responses. It was actually able to cause problems/corrupted message history under certain circumstances (like user calls during ongoing tool responses)

…y and consistency

…- even during fade transitions

…ontent

…argin, spacing, and screen selection

…uration management

…ttings update functionality for hud server

…econnection attempts

…select

Shackless force-pushed the feature/hud-server-and-skill branch from b0f535c to fabf1c4 Compare February 3, 2026 09:17

Shackless added the client-update-required This PR requires the Wingman AI client to be updated label Feb 3, 2026

Shackless force-pushed the feature/hud-server-and-skill branch from e224ea9 to 60363f8 Compare February 17, 2026 09:57

Shackless requested changes Feb 17, 2026

View reviewed changes

SawPsyder marked this pull request as ready for review February 17, 2026 18:05

Copilot AI review requested due to automatic review settings February 17, 2026 18:05

Copilot started reviewing on behalf of SawPsyder February 17, 2026 18:05 View session

Copilot AI reviewed Feb 17, 2026

View reviewed changes

hud_server/tests/test_unicode_stress.py Show resolved Hide resolved

hud_server/tests/run_tests.py Show resolved Hide resolved

wingmen/open_ai_wingman.py Show resolved Hide resolved

SawPsyder requested a review from Copilot February 17, 2026 18:37

Copilot started reviewing on behalf of SawPsyder February 17, 2026 18:37 View session

SawPsyder changed the title ~~wip hud - init of hud skill and server~~ New HUD system for Wingman AI with independent HUD server and new HUD skill utilizing it. Feb 17, 2026

Copilot AI reviewed Feb 17, 2026

View reviewed changes

wingman_core.py Show resolved Hide resolved

wingman_core.py Show resolved Hide resolved

wingman_core.py Show resolved Hide resolved

services/mcp_client.py Outdated Show resolved Hide resolved

Jan Winkler and others added 16 commits February 18, 2026 11:04

wip hud - init of hud skill and server

3501533

removing unneeded code

8e3cce2

Enhance layout manager with improved window positioning logic and upd…

082c536

…ated documentation

Add fixed height option for HUD cells and implement interactive Snake…

822244c

… game test

Add type-safe enums and property classes for HUD elements and fix fon…

84289f6

…t family and font size rendering

Refactor HUD server shutdown process to remove unnecessary await call…

acf0b2d

…s for xvasynth and pocket TTS

Add persistent max height configuration for info panels and validate …

dee0907

…input

Refactor HUD message properties to use type-safe enums and structured…

9eae37d

… classes for improved clarity and maintainability

Enhance SSE connection management with automatic reconnection and sta…

0d3cb5d

…te tracking

Add HUD server settings management and platform checks for Windows su…

6cf27cb

…pport

Jan Winkler and others added 13 commits February 18, 2026 11:04

Implement retry logic for HTTP requests with timeout handling in the …

a1756cc

…HUD HTTP client

remove the user-call-message-block movement as it's no longer necessa…

fb53664

…ry with dummy tool responses. It was actually able to cause problems/corrupted message history under certain circumstances (like user calls during ongoing tool responses)

Refactor HUD element management to use WindowType for improved clarit…

0c26ae9

…y and consistency

Refactor window positioning logic to ensure immediate layout updates …

995f380

…- even during fade transitions

Enhance chat window overflow handling with fade effects for clipped c…

273dd34

…ontent

Add multi-monitor support for HUD rendering with screen selection

aa4446c

Add dynamic HUD settings update functionality for framerate, layout m…

4d0138c

…argin, spacing, and screen selection

Refactor HUD server settings and validation logic for improved config…

2f6fd05

…uration management

Update README to reflect changes in HUD group creation and dynamic se…

3176f38

…ttings update functionality for hud server

Refactor tool response update method docstring for clarity

8d7a745

Enhance error handling during HUD item clearance and SSE connection r…

5455295

…econnection attempts

fix win32 trying to init on non-Windows systems causing a crash

71f99b1

add new custom property type "color"

a12ef41

Shackless force-pushed the feature/hud-server-and-skill branch from 78129af to a12ef41 Compare February 18, 2026 11:05

Rearrange HUD configuration properties and made font settings single …

31f55aa

…select

Shackless merged commit 0638d33 into ShipBit:develop Feb 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New HUD system for Wingman AI with independent HUD server and new HUD skill utilizing it.#343

New HUD system for Wingman AI with independent HUD server and new HUD skill utilizing it.#343
Shackless merged 30 commits intoShipBit:developfrom
SawPsyder:feature/hud-server-and-skill

SawPsyder commented Feb 2, 2026 •

edited

Loading

Uh oh!

Shackless commented Feb 3, 2026 •

edited

Loading

Uh oh!

SawPsyder commented Feb 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

SawPsyder commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shackless commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SawPsyder commented Feb 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SawPsyder commented Feb 2, 2026 •

edited

Loading

Shackless commented Feb 3, 2026 •

edited

Loading