Repository navigation
feat(singleton): configurable lock timing, poll jitter and singleton placement - #279
Open
NiklasArbin wants to merge 54 commits into
Open
NiklasArbin wants to merge 54 commits into
NiklasArbin wants to merge 54 commits into
Conversation
Adds SingletonOptions and ConfigureSingletons for the poll interval, lock duration and renewal interval, and adds random jitter to the wait between lock attempts so instances that started together do not poll in lock-step. Defaults for the three durations are unchanged.
Core adds SingletonOptions and ConfigureSingletons (minor). Host uses them and adds poll jitter, and its package depends on the new Core.
SingletonAssignment.Compute picks a preferred host for every singleton lock with rendezvous hashing and a load cap that keeps heavy locks apart and spreads the rest over the hosts that still have room. Every host computes the same answer from the same inputs. Nothing uses it yet.
Each host keeps a lease on its own member blob, lists its group to find the live members and feeds them to SingletonAssignment. A member is alive while its blob is leased, so liveness does not depend on the clocks of the hosts. The advice is dropped when the group has not been read for StaleAfter. The lock manager can now tell whether a lock is held without taking it, and records the holding host and acquisition time on the lock blob.
… hand-over The receiver asks the placement before taking a lock and hands a held lock over once the preferred host has settled, draining the running message first when the transport can. A host that is not the preferred holder takes a lock only after seeing it free for the takeover grace. KnightBusHost joins and leaves the placement group, and a handler stopped by its cancellation token is no longer logged as an error. Opt in through UseBlobStorageSingletonPlacement.
Adds the singleton placement page and a pointer from the singleton processing page. Releases KnightBus.Core 18.5.0, KnightBus.Host 18.4.0, KnightBus.Azure.Storage 18.4.0 and KnightBus.Azure.ServiceBus 24.3.0, with changelog entries.
NiklasArbin
changed the title
A member blob was treated as abandoned from LastModified, which lease operations do not change, so a host that had been up for 10 minutes was deleted by its peers as soon as its lease lapsed and never rejoined. Unleased time is now tracked per blob, and the host re-uploads and re-leases its member blob when the renewal has stopped. Lock blob names were built with Path.Combine, which uses a backslash on Windows, so the uploaded and leased member blobs differed. They now always join with a forward slash. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Brings in the lease renewal fix (#278, KnightBus.Azure.Storage 18.3.1). Azure.Storage stays at 18.4.0 for the placement, and the changelog keeps both entries.
5 tasks done
…ons-and-poll-jitter
…in from cancelling the running message
5 tasks done
…bStorageLockManager
…f a DrainTimeout option
…able placement with the blob lock manager
5 tasks done
…sions and what to look at, and test the opt-out
…mulated time jumps
…osts the same everywhere SingletonPlacementOptions is gone, so Host and Azure.Storage no longer use internals of Core across package boundaries. The lock blob metadata records the same host id as the logs and the placement members.
…ady stops locks moving back and forth
…sts stay the same
…nTimerScope's constructor as it was
SingletonAssignment and SingletonHostInfo move into KnightBus.Azure.Storage as internals, the only place that uses them. ISingletonPlacement takes the locks and weights in StartAsync instead of a Register call during receiver creation, so the host no longer depends on the order in which the two happen.
…polling after an error The watcher thread, generation counter and polling and hand-over flags are replaced by one record of the held lock that only the loop changes. An error while checking the lock is logged instead of ending the loop, and a lock is released when the receiver it guards fails to start.
…have stopped A hand-over released the lock right after cancelling the receiver, so the next holder could start while a handler was still running. Each singleton now counts the messages it is handling, and the lock is kept until they have stopped, for at most the processor's MessageLockTimeout.
…tempt The member lease was forgotten before rejoining, so a rejoin that failed, for example during a storage outage, left the host out of the member list until it restarted.
…blob names With a lock directory that ends in '/', the member listing used a prefix with a double slash and found no members, so every host thought it was alone, and a leaving host deleted the wrong blob.
…tempt at start fails A failure after the member lease was taken left the host counted by the others but without a refresh loop, so it never gave advice, rejoined or swept. Start now always runs the loop, which joins and refreshes again until it succeeds.
…d since it was listed A sweeping host could delete a member blob between its owner writing it and leasing it on a rejoin. The owner then leased an empty blob that the other hosts cannot read, so they left it out while its lease stayed healthy.
A lock that was handed over or released sat idle until the preferred host's next poll, up to the poll interval plus jitter. The host that should hold a lock and does not now looks for it every 5 seconds; every other host keeps the poll interval.
…nd smaller fixes from review - A hand-over keeps the lock until the cancelled messages have stopped, however long that takes, instead of giving up after the message lock timeout. - A message that reaches the pipeline after the receiver was cancelled is abandoned instead of processed. - Leaving the placement runs alongside the shutdown drain and gives up with the stop token. - A cancelled start stops renewing the member lease. - PollJitter must be a finite number. - The blob lock manager rejects lease durations outside 15 to 60 seconds with a clear message. - ISingletonPlacement.Advise returns the preferred host id; SingletonPlacementAdvice is gone.
…lean up after failed starts - Lock blobs are named with Path.Combine again, as on master, so hosts on the old and new version take the same blob during a rolling upgrade when the directory ends with '/'. - An error from a cancellation callback no longer stops a hand-over halfway; one that has stopped the receiver is finished on the next poll whatever the advice says. - A host whose start fails leaves the placement. - After losing its lock a receiver waits for its last messages before taking the lock again. - The member lease's linked cancellation source is disposed after each rejoin. - The docs state what exclusion actually rests on.
…ed first refresh apart from a failed join, and trim duplicate tests
…ten again, and release the lock cleanly when the receiver fails to start
…er store behind ISingletonMembership
…bs by their age, and drop a catch that could not run
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every singleton lock is an independent blob-lease race, so the instance that starts first takes all of them and keeps them while it renews. Instances that start later (a rolling deploy, a scale-out) hold nothing, so the whole singleton workload runs on one host while the others sit idle. When that workload is heavy, that host is also the one that runs out of memory.
This PR makes the lock timing configurable, adds jitter to the poll, and adds singleton placement, an opt-in mechanism that spreads the locks over the live instances. Docs:
docs/features/singleton-placement.md.Changes
Options and jitter (first two commits)
SingletonOptions/services.ConfigureSingletons(...): poll interval, lock duration, renewal interval. Defaults are the old hardcoded values (1 min, 1 min, 19 s). Registration throws ifLockDurationis not longer thanRenewalInterval.PollJitter) to each wait, so instances that started together no longer poll in lock-step. The wait is never shorter than before, so failover is not faster and can be up to 20% slower.PollJitter = 0restores the old timing.Placement (opt in: nothing changes unless
UseBlobStorageSingletonPlacementis registered)SingletonAssignment(Core): a pure function, rendezvous hashing with a load cap that keeps heavy processors apart. Every host computes the same answer from the same inputs.BlobSingletonPlacement(Azure.Storage): each host leases its own member blob and lists its group, so liveness comes from the storage service and not from host clocks. If the group cannot be read forStaleAfterit stops advising and hosts compete for locks as before.SingletonChannelReceiver: asks the placement before taking a lock, and hands a lock over once the preferred host has settled (StabilityWindow,HandoffInterval). A host that is not preferred takes a lock only after seeing it free forTakeoverGrace, which needsISingletonLockInspector(the blob lock manager implements it).DrainTimeout, then cancels it. That needsIDrainableChannelReceiver, implemented for the Azure Storage queue and Azure Service Bus receivers; other transports cancel at once.HostNameandAcquiredAtUtcin their metadata.LastModified(lease operations do not change it, so peers were deleting live members). Lock blob names always join with/, so Windows hosts see the same blobs.Placement does not make mutual exclusion weaker: the lease still decides who may process. A placement bug can only make the distribution worse.
Versions (per
CONTRIBUTING.md)The pre-release workflow's own detection logic, run against this branch, picks up exactly these four, and the packed nupkgs of Host, Azure.Storage and Azure.ServiceBus each declare a dependency on Core 18.5.0. Changelogs: root, Core, Azure.ServiceBus.
#278 (the lease-renewal fix,
KnightBus.Azure.Storage18.3.1) has been merged and is merged into this branch, so Azure.Storage goes 18.3.1 -> 18.4.0 here.Tests
TimeProvider).mkdocs build --strictpass; Core, Host, Schedule and Azure.Storage unit suites and the lock integration tests pass.Known, not from this change:
Should_not_exceed_max_concurrent_when_prefetch_is_highis timing-sensitive and fails now and then, and 8SagaStoreTestsin the Azure.Storage integration suite fail on a cleanmaster(SagaNotFoundException).Not covered
No run against real Azure, and no real crash with lease expiry (the minimum lease is 15 s). Redis, PostgreSQL and NATS receivers cannot drain yet. Schedules are not placed.