added transactional outbox and dapr post #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I did this to avoid the user perceiving Dapr sidecars as a component that fails often. Signed-off-by: Kenny Pflug <[email protected]>
Signed-off-by: Kenny Pflug <[email protected]>
Signed-off-by: Kenny Pflug <[email protected]>
Signed-off-by: Kenny Pflug <[email protected]>
|
|
||
| A good place for the outbox processor would be in a framework like Dapr which runs as a separate sidecar process next to your service (replicas), but unfortunately, it does not support this pattern out of the box (there is a [GitHub issue](https://github.com/dapr/dapr/issues/4233) for it). However, I also want to shortly discuss how your system architecture can mitigate the Transactional Outbox pattern. | ||
|
|
||
| Consider a kubernetes cluster with Dapr support: one or more instances of a service want to publish events via the accompanied Dapr sidecar. The message broker is either hosted as a cloud service (e.g. Azure Service Bus) or hosted as another pod in the cluster. What would need to happen so that you lose events? Let us make a thought experiment: if the broker is down, as long as your sidecar is up and running and you have it configured for [resiliency](https://docs.dapr.io/operations/resiliency/resiliency-overview/), it will just publish all pending events once the broker is back online. |
There was a problem hiding this comment.
Consider a Kubernetes cluster having Dapr deployed: one or more instances of a service want to publish events leveraging their injected Dapr sidecar container. The message broker is either hosted as a cloud service (e.g. Azure Service Bus) or running side-by-side in the cluster.
|
|
||
|  | ||
|
|
||
| So for things to go awry, your sidecar must be down. But these are built with robustness in mind. Once a Dapr sidecar is up and running, it will likely not fail. It is a technical microservice which has no business logic. If you have Kubernetes set up correctly with health checks and resource constraints, it will just deal with it. I do not want to rule out that Dapr Pub-Sub might contain bugs, but if you execute integration and load tests during development, you will probably not run into them in production. You as a software architect must decide if the additional investment to implement the Transactional Outbox pattern is worth the effort. |
There was a problem hiding this comment.
Kubernetes setup correctly... den Teil find ich etwas holprig. Kubernetes führt die Health-Probes immer aus wenn sie am Container konfiguriert sind. Wenn die Health-Probes fehlschlagen wir der Container von Kubernetes neugestartet.
Der Teil mit den Integration und Load tests erschließt sich mir nicht komplett was willst du damit sagen? Was ist "them" in production die tests oder pub-sub components? 😄
|
|
||
| So for things to go awry, your sidecar must be down. But these are built with robustness in mind. Once a Dapr sidecar is up and running, it will likely not fail. It is a technical microservice which has no business logic. If you have Kubernetes set up correctly with health checks and resource constraints, it will just deal with it. I do not want to rule out that Dapr Pub-Sub might contain bugs, but if you execute integration and load tests during development, you will probably not run into them in production. You as a software architect must decide if the additional investment to implement the Transactional Outbox pattern is worth the effort. | ||
|
|
||
| My two cents: if you run on a properly configured Kubernetes cluster, you can get away without the Transactional Outbox pattern. If you use Dapr in other contexts, for example simple docker-compose on Raspberry Pi(s), I would tend to implement a Transactional Outbox pattern - especially if I must not lose any events. |
There was a problem hiding this comment.
Auch hier würde ich eher schreiben wenn "If you run your workloads on Kubernetes and have them configured properly (resource constraints, health probes, etc.)"
| ``` | ||
|
|
||
| In the piece of code above, we can see an outbox processor whose `ProcessAsync` loads a batch of outbox items from the database and then transforms and sends each item to the message broker. If an error occurs, the processor restarts from the beginning of the loop until all outbox items are sent. It can be triggered periodically (e.g. every few seconds) or other components can trigger it, e.g. when they have created and saved an outbox item. | ||
|
|
There was a problem hiding this comment.
A proper vehicle for running the OutboxProcessor could be a Kubernetes DaemonSet which will process all outbox items created on the given worker-node.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.