GCS backend initial state written without re-confirming it does not exist.

[steeling]

I think there's a race condition in the GCS state mgr (and maybe others). It's a simple fix, but hard to write a unit test for... maybe some E2E test with some artificial waits would be better.

Anyways, the bug is here:

backend/remote-state/gcs/backend_state.go:134

The code initially checks if the state existed prior to grabbing the state lock, then grabs the lock, the creates the state (without checking if it exists). My question is: is it even possible to grab a state lock on a state that doesn't exist? If so, there's a race condition where the state doesn't exist, then something comes in, creates it and releases the lock, then it's overwritten with empty state.

[jbardin]

Thanks @steeling, it appears that was overlooked in this implementation. You can see in the s3 backend, which is operationally similar, there is a second refresh to ensure no state was written between the fast-path and taking the lock. https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_state.go#L176

[steeling]

FYI I also have a bug fix against the azure provider #26561, which breaks force-unlock in non-default workspaces.

With these 2 bugs, I brough up some changes to the interfaces to simplify things, would be interested to experiment with that given a green light from hashicorp. #26572