[BUG] Asynq task didn't got processed by the worker even after many hours of getting picked up

[gautam-tiwari]

Describe the bug

• I have build an application in golang with uses asynq.
• I have around 80-100 workers and 4 queues.
• For the past few weeks I have been encountering an issue in which the asynq worker picks up the tasks(as the state of the task is active) but it doesn't get processed.
• Even when the workers are free it doesn't gets picked up.

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):

1. Implement 4 different queues and about 80-100 workers.
2. Set the priority of the queues as 5, 7, 10 and 10.
3. Tasks should be such that they would take an average of 5 minutes per task.
4. Enqueue a lot of tasks(around 5000) related to the least priority queue.
5. Enqueue lesser tasks(around 1000) releated to high priority queue.
6. Then observe that a few tasks of the low priority queue won't even get picked up.

Expected behavior

• Workers should pick up the task even it is of low priority after a certain amount of time.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS:Linux
• Version of asynq package v0.24.1

Additional context

• One of the other possiblities would be that the workers picked up the task processed it but the redis key didn't got updated.

[of89013]

我估计我的错误也是这个情况

[gautam-tiwari]

Hi @of89013 How is that so?

[gautam-tiwari]

There also exists one issue where the "msg" subkey inside the redis asynq key map gets lost somehow. I'll attach a ss for reference.

Invalid asynq key(missing msg field)

Valid asynq key

[pior]

Are you using Redis Cluster?

[cobain]

it should the same issue with #952