Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fanout queue semantics not correct #45

Open
jeteon opened this issue Feb 15, 2018 · 2 comments
Open

Fanout queue semantics not correct #45

jeteon opened this issue Feb 15, 2018 · 2 comments

Comments

@jeteon
Copy link
Contributor

jeteon commented Feb 15, 2018

In the documentation here, fanout queue usage is described as:

Siberite allows you to insert new message into multiple queues at once by using the following syntax set <queue>+<another_queue>+<third_queue> ...

However, in the Kestrel documentation the intended behaviour (and what I expected) is:

If a queue name has a + in it (like "orders+audit"), it's treated as a fanout queue, using the format +. These queues belong to a parent queue -- in this example, the "orders" queue. Every item written into a parent queue will also be written into each of its children.

Which is to say that the "+" syntax is used to make is so the child queue receives all SETs that the parent queue receives after that. This is what I need since I wanted a producer to create a single queue and then the consumers to be able to create independent queues that follow the main producer queue. With this implementation, it would mean the producer has to know about every consumer and send each of them updates, which is rather inconvenient.

I understand that such a change may be breaking. Do you know of any uses that depend on the "broadcast" behaviour?

If this won't be fixed to be consistent with the Kestrel protocol, would you consider adding syntax for this, perhaps like?:

FOLLOW <child> <parent>

Or perhaps, would you consider making use of another unlikely filename character so that the syntax below complies with the use of "+" in Kestrel?:

SET parent>child 0 0 0
@bogdanovich
Copy link
Owner

bogdanovich commented Feb 15, 2018

Yes, looks like siberite's fanout queues implementation is different from kestrel's.
In kestrel, one write to parent queue becomes N writes where N is the number of child queues. And this configuration is stored somewhere.

Could you clarify a bit your use case?

I wanted a producer to create a single queue and then the consumers to be able to create independent queues that follow the main producer queue.

What do you mean by follow main producer queue?
For me it looks like you want consumer groups where every consumer group has separate offset of parent queue? If so, this is already implemented as consumer group feature:

set work 0 0 10
1234567890
STORED

get work.cg1
VALUE work 0 10
1234567890
END

get work.cg2
VALUE work 0 10
1234567890
END

... 
get work.cgN
VALUE work 0 10
1234567890
END

@jeteon
Copy link
Contributor Author

jeteon commented Feb 15, 2018

Yeah, that is the workaround that I'm currently using. The reason I was looking forward to using the fanout queue feature is that it's less error-prone. With the consumer group cursors, it works pretty much the way I want it to except if I (or someone else on the team) adds a new consumer but forgets to add a cursor (.something), that code would seem to work just fine but mess the scheme up. I liked the idea of the original queue being available for future consumers to replay from the beginning (or however much of it I decided to retain).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants