Browser plugin 5.0.1: waitForExpectation's hardcoded 1s per-attempt timeout now takes effect, silently re-running page scripts

[studioycm]

Environment

• pestphp/pest v5.1.0
• pestphp/pest-plugin-browser v5.0.1 (upgraded from v4.3.1)
• Playwright 1.62.x, PHP 8.4, macOS

Summary

After upgrading the browser plugin from 4.3.1 to 5.0.1, browser tests whose page scripts take longer than one second began failing with measurements taken from a page that had been mutated by its own earlier, partially-completed executions.

The cause is not new code — it is a pre-existing hardcoded 1s per-attempt timeout in waitForExpectation() that only became effective in 5.0.1, because 5.0.1 fixed how the timeout is transmitted to Playwright.

Root cause

Execution::waitForExpectation() caps every attempt at a hardcoded 1000 ms, then makes one final uncapped call after the loop:

// src/Execution.php:138-160
public function waitForExpectation(callable $callback): mixed
{
    $timeout = Playwright::timeout();
    // ...
    while (microtime(true) < $end) {
        try {
            return Playwright::usingTimeout(1_000, $callback);   // <-- hardcoded
        } catch (ExpectationFailedException) {
            //
        }
        $this->resetAssertions($originalCount);
        self::instance()->tick();
    }

    return $callback();   // <-- final attempt, uncapped
}

AwaitableWebpage::__call() routes calls through that retry loop (unless the method is in $nonAwaitableMethods, or the configured timeout is <= 1000):

// src/Api/AwaitableWebpage.php:48-58
if (in_array($name, $this->nonAwaitableMethods, true) || Playwright::timeout() <= 1000) {
    $result = $webpage->{$name}(...$arguments);
} else {
    $result = Execution::instance()->waitForExpectation(
        fn () => $webpage->{$name}(...$arguments),
    );
}

So visit(...)->script(...) — and anything else going through AwaitableWebpage — is executed under a 1s cap, repeatedly.

Why 4.3.1 was unaffected

The cap existed before, but never reached Playwright. 5.0.1's client now sends the action timeout in both params and metadata, with the reason documented in the code itself:

// src/Playwright/Client.php:84-89
// Playwright reads the action timeout from the metadata since 1.62, and read it
// from the params before that. Both are validated with a schema that silently
// drops unknown keys, so sending it twice also covers an older install.
'params' => ['timeout' => $timeout, ...$params],
'metadata' => ['timeout' => $timeout, ...$meta],

That change is correct on its own terms. Its side effect is that the hardcoded 1s per-attempt cap in waitForExpectation() is now enforced by the server, where under 4.3.1 + Playwright ≥ 1.62 it was silently ignored.

Observed behaviour

For a page script that legitimately takes longer than 1s:

1. The attempt is aborted PHP-side at 1s while its page-side effects continue to land.
2. The loop retries. With a 30s configured timeout we measured 31 attempts (confirmed with an in-page attempt counter incremented on window).
3. The final, uncapped call then returns a measurement of a page that has been mutated by up to 30 partially-completed predecessors.

The failure does not look like a timeout. It looks like wrong data — which is considerably harder to diagnose.

Counterintuitive consequence

Because the per-attempt cap is a constant and the loop bound is the configured timeout, raising the timeout does not give a slow script more time — it grants it more 1s-truncated retries. Conversely, setting the timeout to <= 1000 disables the retry wrapper entirely (per the AwaitableWebpage guard above) and the script runs once, uncapped. Both directions are the opposite of what a user would expect.

Minimal reproduction

it('reproduces the 1s per-attempt cap', function (): void {
    $result = visit('/')->script(<<<'JS'
        () => {
            window.__attempts = (window.__attempts ?? 0) + 1;
            const start = Date.now();
            while (Date.now() - start < 1500) { /* > 1s of work */ }
            return window.__attempts;
        }
    JS);

    expect($result)->toBe(1); // observed: a much higher number
});

Workaround (for anyone hitting this)

Call ->page()->evaluate(...) instead of ->script(...). Webpage::script() is literally a passthrough:

// src/Api/Webpage.php:85-88
public function script(string $content): mixed
{
    return $this->page->evaluate($content);
}

so the two are semantically identical, but going through page() bypasses the AwaitableWebpage retry wrapper: one attempt, full timeout budget. Converting our call sites resolved the failures (one affected file went from 7 failures in 311s to 14/14 in 113s; the full browser suite from mixed failures to 58/58 in 164s).

Suggested direction

1. Derive the per-attempt timeout from the configured timeout rather than hardcoding 1000 ms (or make it configurable).
2. Consider whether user-supplied scripts should be routed through a retry loop at all — retrying a callback that mutates page state cannot be safe in general. If retries stay, documenting that expectations passed to waitForExpectation() must be idempotent would help.

Happy to test a patch against a real suite.

[Pawel-IT]

Confirmed, same root cause. A checkbox whose click handler busy-waits 1500ms shows a single $page->click() reaching the page 5 times on v5.0.1 — counted in-page, since the final checkbox state can land "correct" on an odd retry count. Repro test here: pestphp/pest-plugin-browser#249 (comment)

Since issues are disabled on the plugin repo and won't surface from here: the two open fixes are pestphp/pest-plugin-browser#227 and pestphp/pest-plugin-browser#249.