Consider LOCAL_QUORUM mode for locking

[dalaro]

This is a request from a long-time multi-DC Titan user.

The deployment is active-passive between two DCs. Considering that they intend to run only one DC at a time, they're interested in reducing locking from QUORUM to LOCAL_QUORUM, in spite of the operational risk. We could create an option for this with lots of scary warnings in its documentation, but I don't see any way to mitigate the operational risk.

@mbroecheler WDYT?

[mbroecheler]

I have strong reservations against allowing LOCAL_QUORUM for locking since the reason somebody would use locking is to ensure consistency across the entire cluster.
The only use case where this would be useful is if you know that all writes will only ever happen in one DC and you can guarantee that. However, this makes it incredibly easy to shoot yourself in the foot since there is no way for Titan to verify this guarantee, i.e. Titan might be happily operating while inconsistencies arise.
Moreover, this would likely only every happen if somebody moves from single DC to multi DC with simple replication. However, they'll likely go on to full master-master DC setup soon thereafter (as a natural progression) which requires redesigning the application to be lock-free. So, this would (in most cases) only delay that part while being a liability.

Hence, I am against it but keeping this open for further discussion.