KeyError on recent background updates #16054
Description
This issue has been migrated from #16054.
Description
We're seeing a KeyError on the recent updates introduced by matrix-org/synapse@11e15d7:
KeyError: 'current_state_events_stream_ordering_idx'
File "synapse/storage/background_updates.py", line 405, in run_background_updates
result = await self.do_next_background_update(sleep)
File "synapse/storage/background_updates.py", line 528, in do_next_background_update
update_info = self._background_update_handlers[self._current_background_update]
Specifically appears to be missing in BackgroundUpdates._background_update_handlers, but adding some logging indicates it is being registered as a background update:
{"message":"Registering background index update: current_state_events_stream_ordering_idx","namespace":"synapse.storage.background_updates","level":"info","time":1690998470.67,"request":"main","server_name":"beeper.com"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.9,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.98,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
If I run locally with sqlite there is no error, but I can't see any codepath that would change the behaviour for postgres 🤔