[Proposal] Scenario-based deployments

[shiftedreality]

Terms

• ECE-Tools: A project by Magento for the deployment of Magento on Cloud infrastructure (https://github.com/magento/ece-tools)
• Magento Cloud Docker: A project for emulation Cloud infrastructure locally (https://github.com/magento/magento-cloud-docker)

Magento Cloud ecosystem

Current state

Desired state

The desired state - is to have flexible Magento deployment tools which will be used by Community and will be an easily extendable system.

Introduction

ECE-Tools got a good adoption by the community as a robust and predictable too for Magento deployment. It also includes a bunch of useful tools and Cloud Docker environment.  From the beginning of ECE-Tools it was focused on only one type of deployment  (de-facto Platform.sh deployment scenario). This was satisfying our needs until ideas with universal Magento deployment solution were proposed.

With time ECE-Tools became a universal Tool, developers were building their custom deployments around. ECE-Tools will lack the next features:

• Custom deploy scenarios
• Possibility to extend scenarios
• Possibility to remove not needed steps of default scenarios

The benefits will be next:

• Very robust and flexible system to fit almost any type of deployment scenarios
• All-in-one solution to deploy Magento
• Support of Magento Open Source (Previously CE)
• Extended debug and logging features
• An API endpoints with backward compatibility guarantees
• Possibilities to introduce own implementations of common API's

Design

The design proposal fits the current Magento Cloud infrastructure.

Terms

• Scenario - an XML configuration files which describes sequence and configuration parameters to deploy Magento

• Step - an XML structure which describes a programmatic step to be executed

New ECE-Tools implementation will be targeted to deliver maximum flexibility to end-users. ECE-Tools will provide an easy to use API and predefined scenarios and steps to fit Magento's Cloud infrastructure.

Scenarios

Definition

Magento will will provide default scenarios to deploy Magento in **XML **format. Scenarios can be combined and replaced to satisfy customer's needs.

hooks:
    build: |
        set -e
        php ./vendor/bin/ece-tools run scenario/generate.xml
        php ./vendor/bin/ece-tools run scenario/transfer.xml
    deploy: |
        php ./vendor/bin/ece-tools run scenario/deploy.xml
    post_deploy: |
        php ./vendor/bin/ece-tools run scenario/post-deploy.xml

Composite step

<step name="set-production-mode" type="Magento\MagentoCloud\Step\SetProductionMode">
    <argument name="steps">
        <item name="stepA" type="Magento\MagentoCloud\Step\SetProductionMode\A" priority="100" />
        <item name="stepB" type="Magento\MagentoCloud\Step\SetProductionMode\B" priority="200" />
    </argument>
</step>

Priority

• Priority is determined from the step location in scenario

Default Scenarios

Magento MUST include default scenarios for supported Cloud deployment pipelines.

Steps

StepInterface definition

StepInterface is generally lightweight processing class

namespace VendorName\VendorModule\Step;

use Magento\MagentoCloud\Step\SetProductionMode as OriginalSetProductionMode;

class SetProductionMode implements StepInterface
{
    /**
     * @var OriginalSetProductionMode
     */
    private $step;

    public function __construct(OriginalSetProductionMode $step)
    {
        $this->step = $step;
    }

    public function run(): void
    {
        // Do something
        $this->step->run();
        // Do something
    }
}

StepException definition

Each step may throw only 1 type of exception - StepException

class StepException extends \Exception
{
}

Steps MUST NOT be aware of the existence of other steps and be fully replaceable.

Extensibility

By using the XML format of scenarios, an ECE-Tools will have the possibility to extend default scenarios by 3rd-party extensions.

Extending default deploy scenario

In this example, the custom scenario is merged on top of the default scenario.

hooks:
  deploy: |
    php ./vendor/bin/ece-tools run scenario/deploy.xml vendor/vendor-name/module-name/deploy.xml

Merging rules

• Scenarios are prioritized from their positions. Where right has higher priority than left. (B → A)
• If Scenario B has a step StepA with the same name as step StepA in Scenario A, step from Scenario B will be used

Removing default steps

Custom scenario can remove steps from default scenarios:

<scenario>
    <step name="set-production-mode"  skip="true" />
</scenario>

Replacing steps

Custom scenarios can replace default steps and provide custom implementation.

<scenario>
    <step name="set-production-mode" type="VendorName\VendorModule\Step\SetProductionMode" />
</scenario>

SI MUST be able to call the original step:

Customizing steps

We'll provide the possibility to customize some of the default steps, like Shell execution:

<scenario>
    <step name="run-unit-tests" name="Magento\MagentoCloud\Shell\ShellRunner">
        <argument name="command">phpunit -c vendor/test.xml</argument>
    </step>
</scenario>

Validations

ECE-Tools MUST provide an XSD schema for scenarios validation with IDE and static tools.

Backward compatibility

• Current hooks MUST BE compatible with new scenario-based flows
• Current commands MUST BE compatible with new scenario-based flows

API introduction

ECE-Tools will provide an API interfaces which MUST follow Semantic Versioning standards.

• All API interfaces MUST be marked with @api annotation
• Changes to API interfaces MUST be approved by Architect

API interfaces MAY include but not limited to:

• Public interfaces such as ShellInterface
• Default scenarios names
• Default steps names

Logging

• Each deployment MUST begin with the info of executed/merged scenarios:

[2019-07-24 20:00:00] INFO: Starting scenario(s) "scenario/deploy.xml", "vendor/vendor-name/module-name/deploy.xml"
[2019-07-24 20:00:00] ...
[2019-07-24 21:00:00] INFO: Scenario(s) ended

• The logging mechanism MUST reflect the history of executed steps. Steps are logged with INFO level verbosity. Names of steps are extracted from class name from  UpperCamelCase. Where:
  • SetProductionModeStep: "Set Production Mode"
  • DeployStaticContentStep: "Deploy Static Content" 

[2019-07-24 20:00:00] INFO: Run "Set Production Mode" step
[2019-07-24 20:00:00] DEBUG: Run command "bin/magento deploy:mode:set production"
[2019-07-24 20:00:00] INFO: Run "Deploy Static Content" step

New Commands

ece-tools run

This command executes the desired scenario.

Arguments

• scenario - an array of scenarios in XML format to be execute. Value is the relative path to the scenario file

Options

• verbosity - a Symfony Console verbosity modifier

Implementation phases

Phase 1

Introduce DI component

Laravel DI is a fast and reliable system, but it is too messy due to lack of XML configuration support.

• Laravel must provide one of XML-based DI components:
  • Symfony Dependency Injection
  • Magento Dependency Injection

This will be essentially needed for 3rd-party developers for developing custom scenarios. Basic configuration is presented below (Using Symfony DI):

<container>
    <services>
        <defaults autowire="true" autoconfigure="true" public="true"/>
        <prototype namespace="Magento\CloudTools\" resource="../src/*" />
    </services>
</container>

Introduce scenarios processor and parser mechanism

• Implement XSD schema
• Add default scenarios

Refactor existing code to fit the new scenario-based deployments

• Refactor all Processor classes into Step classes and fit to new scenarios
• Refactor all unit tests

Introduce scenario merger mechanism

• Add diff option to see the difference between default and actual scenarios

Introduce an API endpoints

• All Steps are covered with @api annotation

• Important interfaces are covered with @api annotation

Add coverage of new functionality to Functional Testing Framework

• Verify that current tests are passing
• Add scenario with custom steps

Document extensibility mechanism and API endpoints

• An API interfaces and Steps must be marked with an @api annotation

Phase 2

Introduce extensibility mechanism based on Dependency Injection

• Add merged of DI configurations
• Add modules registrations for r-rd party modules

Introduce priority to Steps

• Provide more flexibility to injecting steps

Dependencies

• [Proposal] Using ECE Tools with Magento Open Source #512

[piotrekkaminski]

i understand the AWS/Azure versions - but not sure why you think for custom deployments this is better than someone already using Chef/Puppet etc

[pinkeen]

@piotrekkaminski This does not replace the CI deployment process but rather augments it as I see it.

Instead of having your own scripts (or whatever you're using specific to your stack) that perform various Magento setup tasks (like SCD, DB upgrade, etc.) you can use ece-tools. We might even integrate it into our own AWS-based hosting someday. I think one of the coolest benefits would be project portability across CI/hosting solutions - this way any project using ece-tools could be deployed to any compatible hosting or local development solution without too many build/deploy configuration adjustments (if hosting-specific stuff is abstracted away properly). Additionally, this gives way more flexibility with existing setups like MGC Docker or MGC itself.

[shiftedreality]

API:

• CLI
  -- All commands with @api are considered as API
  -- Adding new required argument is API
  -- Removing argument is API
  -- Removing or renaming option is API

• Interfaces and Classes
  -- All interfaces and classes with @api
  -- Protected/public methods in classes with @api
  -- Renaming class is API change
  -- Renaming protected/public methods is API change
  -- Adding required argument
  -- Removing required argument
  -- Changing argument type
  -- Changing return type
  -- Adding return type explicitly

[YPyltiai]

This was completed.