Skip to content

Commit

Permalink
Increase default CONCURRENCY for replicator (#1550)
Browse files Browse the repository at this point in the history
Since most users seem to connect to a remote hub, increase the default
concurrency.
  • Loading branch information
sds authored Oct 31, 2023
1 parent ab6e76c commit e9a5aec
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 1 deletion.
4 changes: 4 additions & 0 deletions apps/replicator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,12 @@ Once the Docker images have finished downloading, you should start to see messag
...
```

If you are connected to a hub over the internet (rather than on the same machine or private network) the increased latency will make this take longer, and you will likely need to wait some time before an estimation of the time remaining will appear. This is expected.

You may see messages out of order—this is fine. If messages like above are appearing, replication is working as expected.

Note that the number of messages in the Postgres table will **not** match the number in the hub, because the replicator doesn't backfill "Remove" messages (like `CastRemove` and `ReactionRemove`) since these technically indicate an absence of content, not the presence.

### Connecting to Postgres

While it will take a few hours to fully sync all data from the hub, you can start to query data right away.
Expand Down
3 changes: 2 additions & 1 deletion scripts/replicator.sh
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,8 @@ write_env_file() {
fi

if ! key_exists "CONCURRENCY"; then
echo "CONCURRENCY=$(portable_nproc)" >> .env
echo "# Set this higher the further the hub is from the replicator"
echo "CONCURRENCY=$(expr 4 \* $(portable_nproc))" >> .env
fi

if ! key_exists "WORKER_TYPE"; then
Expand Down

0 comments on commit e9a5aec

Please sign in to comment.