Skip to content

Commit

Permalink
Merge pull request #19 from shutterstock/beta
Browse files Browse the repository at this point in the history
Promote beta release to general release
  • Loading branch information
brianphillips authored Jul 21, 2022
2 parents 6aee124 + 3fe7dd0 commit 1d80498
Show file tree
Hide file tree
Showing 10 changed files with 11,600 additions and 4,488 deletions.
2 changes: 1 addition & 1 deletion .commitlintrc.yml
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
extends:
- "@commitlint/config-conventional"
- "@vidavidorra/commitlint-config"
3 changes: 3 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ updates:
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
commit-message:
prefix: "chore(deps): "
include: scope
27 changes: 24 additions & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,15 @@ name: Node.js CI

on:
push:
branches: [ main ]
branches:
- main
- alpha
- beta
pull_request:
branches: [ main ]
workflow_dispatch:
branches:
- main
- alpha
- beta

jobs:
build:
Expand All @@ -26,6 +31,8 @@ jobs:
uses: actions/setup-node@v2
with:
node-version: ${{ matrix.node-version }}
cache: npm
- run: npm install -g npm
- run: npm ci
- run: npm run build
- run: npm run lint
Expand All @@ -36,3 +43,17 @@ jobs:
- name: Stop containers
if: always()
run: docker-compose -f "docker-compose.yml" down
publish:
needs:
- build
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- run: npm ci
- run: npm run build
- name: Semantic Release
uses: cycjimmy/semantic-release-action@v3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
8 changes: 8 additions & 0 deletions .github/workflows/commitlint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,12 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-node@v2
with:
node-version: '18'
- run: npm install
- uses: wagoid/commitlint-github-action@v5
with:
configFile: .commitlintrc.yml
env:
NODE_PATH: ${{ github.workspace }}/node_modules
6 changes: 6 additions & 0 deletions .releaserc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
branches:
- main
- name: beta
prerelease: true
- name: alpha
prerelease: true
89 changes: 6 additions & 83 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ Depending on how you want to retry failed messages, you must create topics to st
For example, if you want to wait 5 seconds for the first two retry attempts and 60 seconds for the third, create topics named `${consumerGroup}-retry-5s` and `${consumerGroup}-retry-60s`.
You can configure the total number of retry attempts with the `maxRetries` parameter.

Remember, each of the topics used for facilitating retries should also have appropriate retention configurations that don't conflict with your retry times.
For example, if your retry policy requires messages to be persisted for 24 hours before a retry is attempted, your retention policy should allow plenty of leeway for the retry message to be picked up.

For more information on configuring retries, see [Retry Topic Naming Strategies](#retry-topic-naming-strategies).

### Example Usage
Expand Down Expand Up @@ -205,36 +208,7 @@ Messages from the "retry" topics are then processed after a configurable delay.
If a message continues to fail beyond a configurable maximum number of attempts, the message is published to a "dead letter" topic for manual inspection.
Here's a sequence diagram demonstrating the process:

<!-- prettier-ignore -->
```mermaid
sequenceDiagram
autonumber
participant topicA as [Kafka]<br>Topic A
participant cg as ConsumerGroupX
participant retry as [Kafka]<br>ConsumerGroupX-Retry-N
participant dead as [Kafka]<br>ConsumerGroupX-DeadLetter
activate cg
loop
topicA->>cg: fetch batch of messages
retry->>cg: fetch batch of messages
end
loop each message
opt From Retry Queue?
note over cg: Pause consumption until message<br>is ready to process
end
opt Error During Processing?
alt Retry?
cg->>retry:
else Fatal Error or Max Retries Reached?
cg->>dead:
end
end
cg->>topicA: commit offset
cg->>retry: commit offset
end
deactivate cg
```
[![](https://mermaid.ink/img/pako:eNqFU9tO3DAQ_ZWRn9kfiNBWiAUeChUFHio1PMzak2A1vmCPESvEvzPOZUtTqZWi-HLOnDknjt-UDoZUozI9F_Kadhb7hK71AFg4-OL2lOoqYmKrbUTPwCFafQaY4edX7H7h4-k-bR_qJpytubqvvPPgc3GUrlIo8ceak4jTYSX3Z8XmrlI239aVhtD8u3AnjGtirinGVJrtCzKJsbocQoh1hDnUZrvVfQMdsX6CPdZ36MBRzthTnpij3f8TyZulAxAKPoOTSIgMlyk4GJPB90KFvkwQgA9iMLxQgtrjFksWv2OuyDZ4KJ7tsOjV0DaLKzQHSQExBS3IpDWbmPpdpBQS7EqyvofbiSbTY1sceLJz3AExIEnHxA0suzSIn0tkHGZJeW7wday1lGWUuGTWKvWwPokszo6TkTQdQyNxnbMs37TLxJ_w2cpf8KwiPX4fsDpR8iM4tEb-8LcKt4qfyFGrxIcy1GEZuFWtfxdqiUbKLozlkFTToYQ8UfUS3B-8Vg2nQgtpviUz6_0D4rQdnw)](https://mermaid.live/edit#pako:eNqFU9tO3DAQ_ZWRn9kfiNBWiAUeChUFHio1PMzak2A1vmCPESvEvzPOZUtTqZWi-HLOnDknjt-UDoZUozI9F_Kadhb7hK71AFg4-OL2lOoqYmKrbUTPwCFafQaY4edX7H7h4-k-bR_qJpytubqvvPPgc3GUrlIo8ceak4jTYSX3Z8XmrlI239aVhtD8u3AnjGtirinGVJrtCzKJsbocQoh1hDnUZrvVfQMdsX6CPdZ36MBRzthTnpij3f8TyZulAxAKPoOTSIgMlyk4GJPB90KFvkwQgA9iMLxQgtrjFksWv2OuyDZ4KJ7tsOjV0DaLKzQHSQExBS3IpDWbmPpdpBQS7EqyvofbiSbTY1sceLJz3AExIEnHxA0suzSIn0tkHGZJeW7wday1lGWUuGTWKvWwPokszo6TkTQdQyNxnbMs37TLxJ_w2cpf8KwiPX4fsDpR8iM4tEb-8LcKt4qfyFGrxIcy1GEZuFWtfxdqiUbKLozlkFTToYQ8UfUS3B-8Vg2nQgtpviUz6_0D4rQdnw)

### Retry Topic Naming Strategies

Expand All @@ -260,33 +234,7 @@ To use this retry strategy, use the `RetryTopicNaming.ATTEMPT_BASED` `retryTopic
Here's a diagram of the message flow using this retry strategy.
This example uses a maximum of three retries with configured wait times of 5 seconds on the first two retries and a 60-second delay on the final attempt:

<!-- prettier-ignore -->
```mermaid
sequenceDiagram
autonumber
participant topicA as [Kafka]<br>Topic A
participant cg as ConsumerGroupX
participant retry1 as [Kafka]<br>ConsumerGroupX-Retry-1
participant retry2 as [Kafka]<br>ConsumerGroupX-Retry-2
participant retry3 as [Kafka]<br>ConsumerGroupX-Retry-3
participant dead as [Kafka]<br>ConsumerGroupX-DeadLetter
topicA->>+cg:
note over cg: Error processing message<br>(initial attempt)
cg-)retry1:
note over retry1: delay 5 seconds
retry1 ->> cg:
note over cg: Error processing message<br>(retry #35;1)
cg -) retry2:
note over retry2: delay 5 seconds
retry2 ->> cg:
note over cg: Error processing message<br>(retry #35;2)
cg -) retry3:
note over retry3: delay 60 seconds
retry3 ->> cg:
note over cg: Error processing essage<br>(retry #35;3)
cg -X dead: Send to dead-letter topic
```
[![](https://mermaid.ink/img/pako:eNqtk0FLw0AQhf_KsF4sGrAJ9RClUGzxoCfroWA8THencTHZjZuNUEr_u7tJKmqaUsVT2OG9x8fkzYZxLYjFrKS3ihSnqcTUYJ4oAKysVlW-JONfAAUaK7ksUFmwupB8AljC0x2uXvH5emnGj34Ik66ap155o1VZ5WRuja6KRVdlyJr18Efmd1Pw4DXBsMccHmMOe8zRMeaoaxaE4rB16hT3ZK1fZONv1heMx2c8jaGZKW0J9DsZ8LOZMdpAYTSnspQqhdx9MSUffyqVtBIzQJeZF3bQBPA0GDQ77Ebu5oIyXMMISuJaibKRtYt3OPAnnNoPJ9HoaviJAsGg_Sc9MOEhmPA_YMIuTNQDE-1gLi_20ES_o9kLE32BWdSdiWFOSrgq1K8gqwvSNIOdM9eeHKVwl7nxxoTZF8opYY6BCVphldmEJWrrpFUh0NJMSKsNi1eYlXTO_PHO14qz2JqKdqL2ulvV9gPvpFhh)](https://mermaid.live/edit#pako:eNqtk0FLw0AQhf_KsF4sGrAJ9RClUGzxoCfroWA8THencTHZjZuNUEr_u7tJKmqaUsVT2OG9x8fkzYZxLYjFrKS3ihSnqcTUYJ4oAKysVlW-JONfAAUaK7ksUFmwupB8AljC0x2uXvH5emnGj34Ik66ap155o1VZ5WRuja6KRVdlyJr18Efmd1Pw4DXBsMccHmMOe8zRMeaoaxaE4rB16hT3ZK1fZONv1heMx2c8jaGZKW0J9DsZ8LOZMdpAYTSnspQqhdx9MSUffyqVtBIzQJeZF3bQBPA0GDQ77Ebu5oIyXMMISuJaibKRtYt3OPAnnNoPJ9HoaviJAsGg_Sc9MOEhmPA_YMIuTNQDE-1gLi_20ES_o9kLE32BWdSdiWFOSrgq1K8gqwvSNIOdM9eeHKVwl7nxxoTZF8opYY6BCVphldmEJWrrpFUh0NJMSKsNi1eYlXTO_PHO14qz2JqKdqL2ulvV9gPvpFhh)

#### Topics based on the configured delay

Expand All @@ -307,32 +255,7 @@ To use this retry strategy, use the `RetryTopicNaming.DELAY_BASED` `retryTopicNa

Here's a sequence diagram of the various topics needing when using a `DELAY_BASED` naming strategy with the `retryDelays` parameter set to `[5, 5, 60]`:

<!-- prettier-ignore -->
```mermaid
sequenceDiagram
autonumber
participant topicA as [Kafka]<br>Topic A
participant cg as ConsumerGroupX
participant retry5s as [Kafka]<br>ConsumerGroupX-Retry-5s
participant retry60s as [Kafka]<br>ConsumerGroupX-Retry-60s
participant dead as [Kafka]<br>ConsumerGroupX-DeadLetter
topicA->>+cg:
note over cg: Error processing message<br>(initial attempt)
cg-)retry5s:
note over retry5s: delay 5 seconds
retry5s ->> cg:
note over cg: Error processing message<br>(retry #35;1)
cg -) retry5s:
note over retry5s: delay 5 seconds
retry5s ->> cg:
note over cg: Error processing message<br>(retry #35;2)
cg -) retry60s:
note over retry60s: delay 60 seconds
retry60s ->> cg:
note over cg: Error processing essage<br>(retry #35;3)
cg -X dead: Send to dead-letter topic
```
[![](https://mermaid.ink/img/pako:eNrFkzFPwzAQhf_KySxUNFKhSoeAKlW0YoCJMlQiDFf7GiwSO9gOUlX1v2MnaQWkRcDCFPn87unz5d2GcS2IJczSa0WK01RiZrBIFQBWTquqWJIJJ4ASjZNclqgcOF1KPgG08HiLqxd8ulqa8UMowqSr5llQXmtlq4LMjdFVueiqDDmzju0X089d0X0QRbE90j4a_Kjfy7oGglB83zz1ijtyLkyk6W_mEI3HZzxLoKkp7Qj0GxkItZkx2kBpNCdrpcqg8F_MKNifSiWdxBzQexal6zUGPIt67TC6nvsLQTmuIQZLXCvRvmc3Q08EfyKqDeBkGF-e72kg6sH_81x0efxvPAJU3zREo8EBpJCTXzEdRBp-QFrU-UlgTkr4WNSnKK_D0qSE9ZlPUoFS-HXbhMaUuWcqKGWegQlaYZW7lKVq66VVKdDRTEinDUtWmFvqs7CR87XiLHGmop2oXdlWtX0H7LVJmA)](https://mermaid.live/edit#pako:eNrFkzFPwzAQhf_KySxUNFKhSoeAKlW0YoCJMlQiDFf7GiwSO9gOUlX1v2MnaQWkRcDCFPn87unz5d2GcS2IJczSa0WK01RiZrBIFQBWTquqWJIJJ4ASjZNclqgcOF1KPgG08HiLqxd8ulqa8UMowqSr5llQXmtlq4LMjdFVueiqDDmzju0X089d0X0QRbE90j4a_Kjfy7oGglB83zz1ijtyLkyk6W_mEI3HZzxLoKkp7Qj0GxkItZkx2kBpNCdrpcqg8F_MKNifSiWdxBzQexal6zUGPIt67TC6nvsLQTmuIQZLXCvRvmc3Q08EfyKqDeBkGF-e72kg6sH_81x0efxvPAJU3zREo8EBpJCTXzEdRBp-QFrU-UlgTkr4WNSnKK_D0qSE9ZlPUoFS-HXbhMaUuWcqKGWegQlaYZW7lKVq66VVKdDRTEinDUtWmFvqs7CR87XiLHGmop2oXdlWtX0H7LVJmA)

## API

Expand Down
Loading

0 comments on commit 1d80498

Please sign in to comment.