Manage speakers manually for real-time refinement & avoiding speaker swapping

[ethanzrd]

Managing speakers manually will allow to signal duplicate speaker detections to the client when using the real-time mode, and to mitigate speaker swappings when using the sequential mode.