Async Symfony Messenger #681

k-37 · 2024-10-27T12:03:12Z

This is proposal to fix issue #539 based on my interpretation of conversation in the issue.

docs/symfony-messenger.md

7-zete-7 · 2024-10-28T07:56:49Z

Great work, @k-37!

Do you think it makes sense to create a new entrypoint for the php-worker service, so that the --time-limit, --limit and --memory-limit parameters can be set via environment variables, and transports (and additional parameters) via the command parameter?

   php-worker:
     # ...
     environment:
       - RUN_MIGRATIONS=false
       - CONSUMER_TIME_LIMIT=60
       - CONSUMER_LIMIT=10
       - CONSUMER_MEMORY_LIMIT=128M
     command: ['async']

dimonx3 · 2024-11-02T20:52:01Z

Thanks @k-37 , @7-zete-7 .
But I have one question.
How to run php-worker-async container in a prod environment?
Please provide an example of how to add the php-worker-async service to the compose.prod.yaml file.

Co-authored-by: Stanislau Kviatkouski <[email protected]>

k-37 · 2024-11-03T10:44:14Z

Do you think it makes sense to create a new entrypoint for the php-worker service, so that the --time-limit, --limit and --memory-limit parameters can be set via environment variables, and transports (and additional parameters) via the command parameter?

Thank you for your review @7-zete-7. Since we are documenting needed changes to existing project files my intention was to provide minimal needed modifications of existing files, to make it as simple as possible. What do you think?

Improved `compose.prod.yaml`

k-37 · 2024-11-03T11:00:55Z

How to run php-worker-async container in a prod environment? Please provide an example of how to add the php-worker-async service to the compose.prod.yaml file.

Thank you @dimonx3 for taking interest in this pull request. I have improved instructions for compose.prod.yaml.

docs/symfony-messenger.md

7-zete-7 · 2024-11-04T07:33:56Z

Do you think it makes sense to create a new entrypoint for the php-worker service, so that the --time-limit, --limit and --memory-limit parameters can be set via environment variables, and transports (and additional parameters) via the command parameter?

Thank you for your review @7-zete-7. Since we are documenting needed changes to existing project files my intention was to provide minimal needed modifications of existing files, to make it as simple as possible. What do you think?

The idea of adding a separate entrypoint came from how cleverly the default entrypoint was bypassed (using /app/bin/console instead of bin/console).

symfony-docker/frankenphp/docker-entrypoint.sh

Line 4 in dc170d9

    
           if [ "$1" = 'frankenphp' ] || [ "$1" = 'php' ] || [ "$1" = 'bin/console' ]; then

It seems to me that such a trick may not be completely clear. It seems that it is necessary to either describe it or move the launch to a separate entrypoint so that this behavior is more obvious.

7-zete-7 · 2024-11-04T08:11:18Z

Perhaps we should move the check for the RUN_MIGRATIONS variable to line 51 instead of 29? So that the connection check remains and only migrations are excluded.

symfony-docker/frankenphp/docker-entrypoint.sh

Lines 29 to 54 in dc170d9

    
           if grep -q ^DATABASE_URL= .env; then 
        
           	echo "Waiting for database to be ready..." 
        
           	ATTEMPTS_LEFT_TO_REACH_DATABASE=60 
        
           	until [ $ATTEMPTS_LEFT_TO_REACH_DATABASE -eq 0 ] || DATABASE_ERROR=$(php bin/console dbal:run-sql -q "SELECT 1" 2>&1); do 
        
           		if [ $? -eq 255 ]; then 
        
           			# If the Doctrine command exits with 255, an unrecoverable error occurred 
        
           			ATTEMPTS_LEFT_TO_REACH_DATABASE=0 
        
           			break 
        
           		fi 
        
           		sleep 1 
        
           		ATTEMPTS_LEFT_TO_REACH_DATABASE=$((ATTEMPTS_LEFT_TO_REACH_DATABASE - 1)) 
        
           		echo "Still waiting for database to be ready... Or maybe the database is not reachable. $ATTEMPTS_LEFT_TO_REACH_DATABASE attempts left." 
        
           	done 
        
           	if [ $ATTEMPTS_LEFT_TO_REACH_DATABASE -eq 0 ]; then 
        
           		echo "The database is not up or not reachable:" 
        
           		echo "$DATABASE_ERROR" 
        
           		exit 1 
        
           	else 
        
           		echo "The database is now ready and reachable" 
        
           	fi 
        
           	if [ "$( find ./migrations -iname '*.php' -print -quit )" ]; then 
        
           		php bin/console doctrine:migrations:migrate --no-interaction --all-or-nothing 
        
           	fi 
        
           fi

k-37 · 2024-11-11T07:18:58Z

@7-zete-7 thank you for your suggestions.

I must admit that I don't quite understand the issue with entrypoint and what improvements should be made.

I'm split on the suggestion to move migration check.

Is it OK to leave both decisions to project maintainers who are more knowledgeable about the project than me?

7-zete-7 · 2024-11-11T07:52:55Z

I must admit that I don't quite understand the issue with entrypoint and what improvements should be made.

The default entrypoint will only work if the first element in command is frankenphp, php, or bin/console.

symfony-docker/frankenphp/docker-entrypoint.sh

Line 4 in dc170d9

    
           if [ "$1" = 'frankenphp' ] || [ "$1" = 'php' ] || [ "$1" = 'bin/console' ]; then

This PR uses /app/bin/console as the first element in command, which skips entire code of then statement of this if.

7-zete-7 · 2024-11-11T07:59:08Z

I'm split on the suggestion to move migration check.

In my opinion, when using the RUN_MIGRATIONS parameter, it is better to leave the check for a connection to the database, and only disable the application of migrations.

symfony-docker/frankenphp/docker-entrypoint.sh

Lines 51 to 53 in dc170d9

    
           if [ "$( find ./migrations -iname '*.php' -print -quit )" ]; then 
        
           	php bin/console doctrine:migrations:migrate --no-interaction --all-or-nothing 
        
           fi

Is it OK to leave both decisions to project maintainers who are more knowledgeable about the project than me?

Can you help to choose the appropriate option, @maxhelias?

k-37 · 2024-11-11T16:02:31Z

The default entrypoint will only work if the first element in command is frankenphp, php, or bin/console.

@7-zete-7 thank you for the clarification. What do you think about using:

command: ['php', '-f', '/app/bin/console', 'messenger:consume', 'async', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M']

instead of:

command: ['/app/bin/console', 'messenger:consume', 'async', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M']

7-zete-7 · 2024-11-11T17:15:01Z

The default entrypoint will only work if the first element in command is frankenphp, php, or bin/console.

@7-zete-7 thank you for the clarification. What do you think about using:
command: ['php', '-f', '/app/bin/console', 'messenger:consume', 'async', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M']
instead of:
command: ['/app/bin/console', 'messenger:consume', 'async', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M']

Thanks for the great examples @k-37!

Of these two options, I would prefer the second one.

Although it would make sense to just use bin/console instead of /app/bin/console.

command: ['bin/console', 'messenger:consume', 'async', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M']

k-37 · 2024-11-11T18:13:45Z

bin/console it is then. Thank you @7-zete-7.

Regarding your point about RUN_MIGRATIONS. Since php-worker-async has:

depends_on:
  php:
    condition: service_healthy

it is started after php service which waits for database and runs migrations. php-worker-async will start slightly faster without additional database check which is currently used only before running migrations.

My preference would be to leave it as it is. But I'm fine with whatever maintainers like better.

References: - dunglas/symfony-docker#681

docs/symfony-messenger.md

k-37 · 2024-11-20T21:14:57Z

Thank you @daFish for taking interest in this PR.

I use ^DATABASE_URL= here without quotes around it and it works as expected. I just rebuilt my project to test it.

Here is relevant line in .env file.

Here is compose.yaml and compose.override.yaml.

Do you see any other difference which might be behind your issue?

daFish · 2024-11-22T09:51:36Z

@k-37 I wanted to recreate the behaviour but wasn't able to. 😅 I think my comment is no longer relevant.

maxhelias · 2024-11-25T13:36:43Z

Hi @k-37, @7-zete-7,
Thanks for tackle this.
I finally took the time for a first reading. I haven't tested it yes, but it's look a great job.

I'm not really in favor of a new entrypoint to stay as lite as possible. I find that the time-limit, limit & memory-limit options are not necessary in env variable, the command can be easily modified (as we are copy/paste from the doc). WDYT @dunglas ?

For the docker-entrypoint.sh modification, the RUN_MIGRATIONS, can be apply directly IMO with the default value.

I'll try this soon for a better feedback 😉

docs/symfony-messenger.md

Referenced from dunglas/symfony-docker#681

moshkov-konstantin · 2024-12-10T10:32:01Z

@dunglas Maybe merge?

LaurentSanson · 2024-12-26T13:39:29Z

Hi all,

Any update on this PR?

Not sure if it's related but my php service is not healthy anymore with the added code (on the CI)

k-37 · 2024-12-27T08:40:58Z

Hi @LaurentSanson,

Could you please give more details about your service and environment? I would like to reproduce the problem.

If you could provide simplified non working example that would be great.

LaurentSanson · 2024-12-27T08:49:18Z

Hey @k-37 !

Sure, here is some of my actual code :

compose.yaml

services:
    php:
        image: ${IMAGES_PREFIX:-}app-php
        restart: unless-stopped
        environment:
            SERVER_NAME: ${SERVER_NAME:-localhost}, php:80
            MERCURE_PUBLISHER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeThisMercureHubJWTSecretKey!}
            MERCURE_SUBSCRIBER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeThisMercureHubJWTSecretKey!}
            DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@database:5432/${POSTGRES_DB}?serverVersion=${POSTGRES_VERSION}&charset=${POSTGRES_CHARSET:-utf8}
        volumes:
            - caddy_data:/data
            - caddy_config:/config
        ports:
            # HTTP
            -   target: 80
                published: 80
                protocol: tcp
            # HTTPS
            -   target: 443
                published: 443
                protocol: tcp
            # HTTP/3
            -   target: 443
                published: 443
                protocol: udp
    
    php-worker:
        image: ${IMAGES_PREFIX:-}app-php-worker
        restart: unless-stopped
        environment:
            - RUN_MIGRATIONS=false
        healthcheck:
            disable: true
        depends_on:
            - php
    
    # Mercure is installed as a Caddy module, prevent the Flex recipe from installing another service
    ###> symfony/mercure-bundle ###
    ###< symfony/mercure-bundle ###
    
    ###> doctrine/doctrine-bundle ###
    database:
        image: postgres:${POSTGRES_VERSION:-16}-alpine
        environment:
            POSTGRES_DB: ${POSTGRES_DB:-app}
            # You should definitely change the password in production
            POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-!ChangeMe!}
            POSTGRES_USER: ${POSTGRES_USER:-app}
        healthcheck:
            test: [ "CMD", "pg_isready", "-d", "${POSTGRES_DB:-app}", "-U", "${POSTGRES_USER:-app}" ]
            timeout: 5s
            retries: 5
            start_period: 60s
        volumes:
            - database_data:/var/lib/postgresql/data:rw
            # You may use a bind-mounted host directory instead, so that it is harder to accidentally remove the volume and lose all your data!
            # - ./docker/db/data:/var/lib/postgresql/data:rw
    ###< doctrine/doctrine-bundle ###

    ###> sensiolabs/gotenberg-bundle ###
    gotenberg:
        image: 'gotenberg/gotenberg:8'
    ###< sensiolabs/gotenberg-bundle ###

volumes:
    caddy_data:
    caddy_config:
    ###> symfony/mercure-bundle ###
    ###< symfony/mercure-bundle ###
    
    ###> doctrine/doctrine-bundle ###
    database_data:
    ###< doctrine/doctrine-bundle ###

compose.override.yaml

services:
    php:
        build:
            context: .
            target: frankenphp_dev
        volumes:
            - ./:/app
            - ./frankenphp/Caddyfile:/etc/caddy/Caddyfile:ro
            - ./frankenphp/conf.d/20-app.dev.ini:/usr/local/etc/php/app.conf.d/20-app.dev.ini:ro
            # If you develop on Mac or Windows you can remove the vendor/ directory
            #  from the bind-mount for better performance by enabling the next line:
            #- /app/vendor
        environment:
            MERCURE_EXTRA_DIRECTIVES: demo
            # See https://xdebug.org/docs/all_settings#mode
            XDEBUG_MODE: "${XDEBUG_MODE:-off}"
        extra_hosts:
            # Ensure that host.docker.internal is correctly defined on Linux
            - host.docker.internal:host-gateway
        tty: true
    
    php-worker:
        profiles:
            - donotstart
    php-worker-async:
        scale: 2
        extends:
            file: compose.yaml
            service: php-worker
        image: ${IMAGES_PREFIX:-}app-php-worker-async
        build:
            context: .
            target: frankenphp_dev
        command: [ 'bin/console', 'messenger:consume', 'scheduler_form_analysis', '-vv', '--time-limit=60', '--limit=10', '--memory-limit=128M' ]
        volumes:
            - ./:/app
            - /app/var/
        depends_on:
            php:
                condition: service_healthy
    
    ###> symfony/mercure-bundle ###
    ###< symfony/mercure-bundle ###
    
    ###> doctrine/doctrine-bundle ###
    database:
        ports:
            - "5432:5432"
    ###< doctrine/doctrine-bundle ###
    
    ###> symfony/mailer ###
    maildev:
        image: maildev/maildev
        container_name: maildev-container
        command: bin/maildev --web 1080 --smtp 25 --verbose
        healthcheck:
            test: 'wget -O - http://127.0.0.1:1080/healthz || exit 1'
        ports:
            - "1080:1080"
    ###< symfony/mailer ###

    ###> sensiolabs/gotenberg-bundle ###
    gotenberg:
        ports:
            - "3000"
    ###< sensiolabs/gotenberg-bundle ###

And here is the error on the CI :

 Network testmyform_default  Creating
 Network testmyform_default  Created
 Volume "testmyform_caddy_data"  Creating
 Volume "testmyform_caddy_data"  Created
 Volume "testmyform_caddy_config"  Creating
 Volume "testmyform_caddy_config"  Created
 Volume "testmyform_database_data"  Creating
 Volume "testmyform_database_data"  Created
 Container maildev-container  Creating
 Container testmyform-database-1  Creating
 Container testmyform-php-1  Creating
 Container testmyform-gotenberg-1  Creating
 Container testmyform-gotenberg-1  Created
 Container testmyform-database-1  Created
 Container maildev-container  Created
 Container testmyform-php-1  Created
 Container testmyform-php-worker-async-2  Creating
 Container testmyform-php-worker-async-1  Creating
 Container testmyform-php-worker-async-1  Created
 Container testmyform-php-worker-async-2  Created
 Container maildev-container  Starting
 Container testmyform-php-1  Starting
 Container testmyform-database-1  Starting
 Container testmyform-gotenberg-1  Starting
 Container testmyform-gotenberg-1  Started
 Container testmyform-database-1  Started
 Container maildev-container  Started
 Container testmyform-php-1  Started
 Container testmyform-php-1  Waiting
 Container testmyform-php-1  Error
dependency failed to start: container testmyform-php-1 is unhealthy
Error: Process completed with exit code 1.

I hope this will help, but once again not 100% sure it's related to the added code but kind of

k-37 · 2024-12-27T11:11:24Z

@LaurentSanson,

Can you SSH into your CI server and get output of the command:

docker logs testmyform-php-1

Do you have any problems when running your application with Docker outside of CI, i.e. locally?

LaurentSanson · 2024-12-27T11:53:15Z

Locally I forgot to add the --wait option... I've just added it and rebuilt all the containers, here is the error I have :

container xxx-php-worker-async-1 has no healthcheck configured

In the compose.yaml my editor is giving me an error on the healthcheck for the php-worker =>

Missing required key(s): 'test'


Declare a check that's run to determine whether or not containers for this service are "healthy".
 
healthcheck:   test: ["CMD", "curl", "-f", "http:// localhost"]   interval: 1m30s   timeout: 10s   retries: 3   start_period: 40s

PS : if I change

        healthcheck:
            disable: true

By

        healthcheck:
            test: ["NONE"]

I've got this error : container xxx-php-worker-async-2 has no healthcheck configured

k-37 · 2024-12-27T12:50:59Z

That looks like a separate issue maybe caused by --wait option, which waits for services to be running|healthy.

In compose.yaml you have:

php-worker:
    ...
    healthcheck:
        disable: true

Health check is disabled for php-worker because worker will restart itself periodically.

Your original error is about php container not about php-worker.

Try starting your application locally with these commands and report eventual errors:

docker compose down --remove-orphans
docker compose build --pull --no-cache
docker compose up # Note that we are not using `--detach` option, we want to see logs in the terminal

I have working example of application using async Symfony messenger. Your compose files look pretty similar to mine. Container testmyform-php-1 Error might be something specific to your application or environment.

LaurentSanson · 2024-12-27T13:01:57Z

Yes if the containers are restarting periodically, we can't use the --wait option anymore I guess... that's quite a shame but that'll do for now

I have working example of application using async Symfony messenger. Your compose files look pretty similar to mine. Container testmyform-php-1 Error might be something specific to your application or environment.

Yes I found an error due to migrations, only related to my mistakes, my bad.

k-37 added 2 commits October 27, 2024 12:45

Update README.md

817470a

Create symfony-messenger.md

e50c128

7-zete-7 reviewed Oct 28, 2024

View reviewed changes

docs/symfony-messenger.md Outdated Show resolved Hide resolved

7-zete-7 reviewed Oct 28, 2024

View reviewed changes

docs/symfony-messenger.md Outdated Show resolved Hide resolved

7-zete-7 reviewed Oct 28, 2024

View reviewed changes

docs/symfony-messenger.md Outdated Show resolved Hide resolved

k-37 and others added 2 commits November 3, 2024 11:22

Apply suggestions from code review

abf96f4

Co-authored-by: Stanislau Kviatkouski <[email protected]>

Update symfony-messenger.md

0d82aba

k-37 added 2 commits November 3, 2024 11:53

Update symfony-messenger.md

2639b9c

Improved `compose.prod.yaml`

Update symfony-messenger.md

aa80aa2

dimonx3 reviewed Nov 3, 2024

View reviewed changes

docs/symfony-messenger.md Outdated Show resolved Hide resolved

k-37 added 4 commits November 3, 2024 12:10

Update symfony-messenger.md

a1bfdf2

Update symfony-messenger.md

0fb2915

Update symfony-messenger.md

71a4536

Update symfony-messenger.md

f058913

Update symfony-messenger.md

eeb912d

k-37 added a commit to k-37/dolphin-is-symfony-ddd-experiment that referenced this pull request Nov 12, 2024

Improve messenger async consumer

011eb00

References: - dunglas/symfony-docker#681

daFish reviewed Nov 20, 2024

View reviewed changes

docs/symfony-messenger.md Show resolved Hide resolved

maxhelias added the documentation Documentation needs adjustment label Nov 25, 2024

maxhelias mentioned this pull request Nov 25, 2024

Symfony Messenger #539

Open

norkunas reviewed Nov 25, 2024

View reviewed changes

docs/symfony-messenger.md Show resolved Hide resolved

norkunas reviewed Nov 25, 2024

View reviewed changes

docs/symfony-messenger.md Show resolved Hide resolved

Update symfony-messenger.md

1678c4c

pan93412 added a commit to database-playground/app-sf that referenced this pull request Dec 8, 2024

feat(docker): Add app-sf-worker for asynchronous message handling

957aede

Referenced from dunglas/symfony-docker#681

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async Symfony Messenger #681

Async Symfony Messenger #681

k-37 commented Oct 27, 2024

7-zete-7 commented Oct 28, 2024

dimonx3 commented Nov 2, 2024 •

edited

Loading

k-37 commented Nov 3, 2024

k-37 commented Nov 3, 2024

7-zete-7 commented Nov 4, 2024

7-zete-7 commented Nov 4, 2024

k-37 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

k-37 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

k-37 commented Nov 11, 2024

k-37 commented Nov 20, 2024

daFish commented Nov 22, 2024

maxhelias commented Nov 25, 2024

moshkov-konstantin commented Dec 10, 2024

LaurentSanson commented Dec 26, 2024 •

edited

Loading

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024 •

edited

Loading

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024

Async Symfony Messenger #681

Are you sure you want to change the base?

Async Symfony Messenger #681

Conversation

k-37 commented Oct 27, 2024

7-zete-7 commented Oct 28, 2024

dimonx3 commented Nov 2, 2024 • edited Loading

k-37 commented Nov 3, 2024

k-37 commented Nov 3, 2024

7-zete-7 commented Nov 4, 2024

7-zete-7 commented Nov 4, 2024

k-37 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

k-37 commented Nov 11, 2024

7-zete-7 commented Nov 11, 2024

k-37 commented Nov 11, 2024

k-37 commented Nov 20, 2024

daFish commented Nov 22, 2024

maxhelias commented Nov 25, 2024

moshkov-konstantin commented Dec 10, 2024

LaurentSanson commented Dec 26, 2024 • edited Loading

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024 • edited Loading

k-37 commented Dec 27, 2024

LaurentSanson commented Dec 27, 2024

dimonx3 commented Nov 2, 2024 •

edited

Loading

LaurentSanson commented Dec 26, 2024 •

edited

Loading

LaurentSanson commented Dec 27, 2024 •

edited

Loading