Don't use `block_in_place` for long running blocking tasks #281

jeromegn · 2024-12-30T18:09:29Z

Uses more threads
Uses spawn_blocking instead of block_in_place in corro-pg
Processes items faster and in smaller batches

…stead of block_in_place for long running PG connections

… we don't have capacity: queue

…hey go dead

…rker has all the capacity in the world. reduce apply timeout and apply queue

jeromegn and others added 8 commits December 30, 2024 11:42

allow more worker threads on the tokio runtime, use spawn_blocking in…

d2aacc1

…stead of block_in_place for long running PG connections

process items immediately if we have capacity instead of queueing, if…

ed196de

… we don't have capacity: queue

make sure we rollback any ongoing transaction

952a3e7

add more logs to understand when processing changes halts

c245876

increase timeout for write conn

53b2aca

set a short keepalive on accepted pg conns so we detect better when t…

e0bbd40

…hey go dead

don't apply immediately if we have queued items, only apply if job wo…

4652743

…rker has all the capacity in the world. reduce apply timeout and apply queue

make extra sure we got precedence correct here

25992ba

jeromegn merged commit 3b5b07e into main Jan 22, 2025
3 of 4 checks passed

jeromegn deleted the less-blocking branch January 22, 2025 19:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't use `block_in_place` for long running blocking tasks #281

Don't use `block_in_place` for long running blocking tasks #281

jeromegn commented Dec 30, 2024 •

edited

Loading

Don't use block_in_place for long running blocking tasks #281

Don't use block_in_place for long running blocking tasks #281

Conversation

jeromegn commented Dec 30, 2024 • edited Loading

Don't use `block_in_place` for long running blocking tasks #281

Don't use `block_in_place` for long running blocking tasks #281

jeromegn commented Dec 30, 2024 •

edited

Loading