Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requeuer and Message Delay #469

Merged
merged 45 commits into from
Oct 25, 2024
Merged

Requeuer and Message Delay #469

merged 45 commits into from
Oct 25, 2024

Conversation

m110
Copy link
Member

@m110 m110 commented Aug 20, 2024

  • Added the Requeuer component.
    • It works as a simpler version of the Forwarder, routing messages from one topic to another (a dynamic one).
    • Can be used to move messages that failed to process back to the original topic, so they don't block other messages.
    • Can be used together with the poison middleware and the delay component to delay the forwarding.
  • Added the delay package. It contains helpers for setting delay metadata on messages.
    • Does not do anything by itself. A Pub/Sub needs to support it explicitly. For now, that's the delayed postgres Pub/Sub implemented in Add PostgreSQLQueue and DelayedPostgreSQL Pub/Sub watermill-sql#34
    • Use case 1: publishing a message after a given delay or at given time (see the example).
    • Use case 2: automatically moving messages out of the poison queue to the original topic after a delay (used together with the Requeuer component).
  • Added the pq CLI tool for working with poison queues.

components/requeue/requeue.go Outdated Show resolved Hide resolved
components/requeue/requeue.go Outdated Show resolved Hide resolved
Copy link

@ccoVeille ccoVeille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm glad you liked my feedbacks

@m110 m110 changed the title Requeue PoC Requeuer component Sep 6, 2024
Copy link

codecov bot commented Oct 10, 2024

Codecov Report

Attention: Patch coverage is 73.28767% with 39 lines in your changes missing coverage. Please review.

Project coverage is 73.42%. Comparing base (f48da3c) to head (d9a7baa).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
components/requeuer/requeuer.go 55.55% 22 Missing and 10 partials ⚠️
components/delay/publisher.go 86.84% 4 Missing and 1 partial ⚠️
components/delay/delay.go 88.23% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #469      +/-   ##
==========================================
- Coverage   73.55%   73.42%   -0.13%     
==========================================
  Files          61       65       +4     
  Lines        3278     3424     +146     
==========================================
+ Hits         2411     2514     +103     
- Misses        715      746      +31     
- Partials      152      164      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@m110 m110 changed the title Requeuer component Requeuer and Message Delay Oct 11, 2024
components/delay/delay.go Show resolved Hide resolved
components/delay/delay.go Show resolved Hide resolved
components/delay/publisher.go Outdated Show resolved Hide resolved
tools/pq/backend/postgres.go Show resolved Hide resolved
@m110 m110 marked this pull request as ready for review October 24, 2024 10:06

## Usage

Set the `DATABASE_URL` environment variable to your PostgreSQL connection string.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be nice to provide example one - they are 10 different formats and it's useful to have it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -0,0 +1,32 @@
# pq
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably it would be nice to link here from poison queue godoc?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which one, though? The SQL one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add more docs in a follow-up PR.

@m110 m110 merged commit c32fdc4 into master Oct 25, 2024
9 of 11 checks passed
m110 added a commit to ThreeDotsLabs/watermill-sql that referenced this pull request Oct 25, 2024
* Added `PostgreSQLQueueSchema` and `PostgreSQLQueueOffsetsAdapter`. 
    * This schema is similar to the original one, except it allows filtering messages with a custom `WHERE` clause.
    * It doesn't support customer groups.
    * It allows deleting messages from the table after acking (optional).
* Added `NewDelayedPostgreSQLPublisher` and `NewDelayedPostgresSQLSubscriber` 
    * They work on top of the conditional schema and use the `delay` component implemented in ThreeDotsLabs/watermill#469
    * The idea is to receive messages with a delay or at a given time in a simple way.
* Reworked the `SchemaAdapter` and `OffsetsAdapter` interfaces **(Breaking change -> will bump the major to v4)**.
    * We had a few instances where some details have been missing from one of the interface methods. It makes it difficult to extend this library with new features, as each will require a major version bump, and we want to avoid it if possible.
    * To mitigate this, we're moving to a `params` struct passed to the methods instead of raw arguments. It adds some verbosity but allows extending the params if needed with no breaking changes to the library.
    * The methods now also return errors.
    * Migration: if you don't use a custom schema/offsets adapter, you don't need to do anything. If you do, change the methods to implement the new interfaces. Use values from the `params` argument and return errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants