Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement inter-topic dependency management #45

Open
stefan-cardnell-rh opened this issue Nov 29, 2024 · 0 comments
Open

Implement inter-topic dependency management #45

stefan-cardnell-rh opened this issue Nov 29, 2024 · 0 comments

Comments

@stefan-cardnell-rh
Copy link
Contributor

stefan-cardnell-rh commented Nov 29, 2024

Add an optional functionality for inter-topic dependency management, to resolve the following issues:

  1. If an Event 1 for entity A fails in a topic and is sent to the non-blocking retry or dead letter queue, it may be desired that Event 2 for the same entity A should also not be processed until Event 1 succeeds (Event 2 should be put in a holding queue). Essentially, the order of events for the same entity needs to be respected.
  2. It could be that messages in some topics should not be processed before messages in other topics. These dependent messages should be put in a holding queue until their dependency in the other topic has been processed.

"Entity" can be determined by the message key normally, but ideally it should be overideable.

Sources for implementation:

  1. https://www.confluent.io/blog/error-handling-patterns-in-kafka/#pattern-4
  2. https://medium.com/altitudehq/kafka-retries-and-maintain-order-of-retry-events-313482044351 - see "Maintain order of retry events"
  3. https://www.youtube.com/watch?v=FO2ptQNQKhM&ab_channel=DanielTammadge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant