libobs-d3d11: Correctly handle duplicator failure

[ThrowTop]

Description

This change corrects error handling in the D3D11 desktop duplicator update path.
Previously, certain DXGI failures during AcquireNextFrame were not treated as fatal, causing the duplicator to remain in an invalid state and repeatedly return black frames.

The function now correctly returns false when frame acquisition fails, allowing higher-level logic to recreate the duplicator when required.

After applying this change and running it continuously for over four months across multiple systems, the persistent black screen issue has not reoccurred.
The error: gs_duplicator_update_frame: Failed to update frame (887A0001)
Occurred 3 times since applyting my patch and the duplicator has correctly recovered every time aka no permament black screen.

---

Motivation and Context

There have been long-standing reports of intermittent black screens when using DXGI desktop duplication capture in fullscreen games. Typically described as occurring very rarely and without a reliable reproduction path.

In practice, this issue manifests less than once per month and is therefore difficult to diagnose or reproduce on demand. When it does occur, capture remains black indefinitely until OBS or the capture pipeline is restarted (For example when clicking on the display capture source).

My testing shows that certain failure cases returned from AcquireNextFrame were not being handled as fatal errors when they should be. This leaves the duplicator in an invalid state instead of signaling failure and allowing higher-level logic to recreate the duplicator.

---

How Has This Been Tested?

• Windows 11 24h2 and 25h2
• DXGI Desktop Duplication capture
• Multi-monitor setup
• Verified capture recovers correctly after failure

---

Types of changes

• Bug fix (non-breaking change which fixes an issue)