federation_sender repeatedly wakes up destinations ('Destination example.org has outstanding catch-up, waking up') making no progress

[reivilibre]

Description

If I am in a room and then leave that room,
it's possible to enter a state where some the federation sender will perpetually wake up that destination to send it new events for that room,
without making any progress since we are no longer in that room.

(reporting on behalf of others + matrix.org)

This logic selects destinations to wake up:

        WITH pdu_destinations AS (
            SELECT DISTINCT destination FROM destination_rooms
            LEFT JOIN destinations USING (destination)
            WHERE
                destination > ?
                AND destination_rooms.stream_ordering > COALESCE(destinations.last_successful_stream_ordering, 0)
                AND (
                    destinations.retry_last_ts IS NULL OR
                    destinations.retry_last_ts + destinations.retry_interval < ?
                )
            ORDER BY destination
            LIMIT 25
        ), to_device_destinations AS (
            SELECT DISTINCT destination FROM device_federation_outbox
            LEFT JOIN destinations USING (destination)
            WHERE
               destination > ?
               AND (
                    destinations.retry_last_ts IS NULL OR
                    destinations.retry_last_ts + destinations.retry_interval < ?
               )
            ORDER BY destination
            LIMIT 25
        )

        SELECT destination FROM pdu_destinations
        UNION SELECT destination FROM to_device_destinations
            ORDER BY destination
            LIMIT 25

so it's enough to have destination_rooms rows for a room you're not in, to always have a destination to wake up that you can't make progress with.

Steps to reproduce

untested

• join federated room
• send events, with some servers not online (so it enters catch-up mode) — those servers should not be in another room that you're in
• leave room
• do not purge the room
• view logs

Homeserver

matrix.org, others

Synapse Version

1.130.0

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres

Workers

Multiple workers

Platform

Configuration

No response

Relevant log output

part of `wake_destinations_needing_catchup-0`

Anything else that would be useful to know?

No response