📖 Airflow 3.0 - Initial Infrastructure Buildout for Airflow-Dev

[jhpyke]

User Story

As a maintainer of Airflow
I need a modern platform that can be easily maintained and updated
So that I can continue providing a high quality service.

Value / Purpose

Following on from #6096, we should start building out the infrastructure of our new Airflow Environments. I believe that the minimum amount of infra required should be:

• An S3 bucket for holding DAGs and Config files (in Analytical-Platform-Compute)
• A requirements file for Airflow 2.10.3
• A VPC (may be shared/existing)
• A Security Group with Inbound/Outbound rules (may be shared/existing)
• A MWAA environment (mw1.medium)
• 5 cloudwatch log groups (Task, Web Server, Scheduler, Worker, DAG Processing)
• A Node Execution Role that is empowered enough to fetch images from required image repos.

Once we've built the MWAA environment, that then unlocks a number of strands of work (making images accessible, designing the DAG Factory/IAM Builder setup, Image Scanning and Monitoring etc.) as there will be the core infrastructure in place for other areas to reference.

Useful Contacts

@jhpyke, @jacobwoffenden

User Types

Airflow Users

Hypothesis

If we build a bare minimum MWAA environment targeting current airflow version
Then we will be able to build out the rest of the required infra.

Proposal

We should provision the resources as laid out in the agreed architecture diagram, as terraform in this repo. Once we have an environment we can upload DAGs to, that facilitates the effective work on and testing of many subsequent tickets.

Additional Information

No response

Definition of Done

• We have a MWAA environment that is running in Analytical-Platform-Compute
• We are able to execute a DAG within that environment
• A Kubernetes pod is able to be scheduled from that environment
• We are able to retrieve all relevant logs from cloudwatch

[jhpyke]

Much of the work captured in this ticket is in practice already completed, in the form of this PoC (Zephyr). Only needs mild cleanup to ensure functionality is fully in line with architectural diagram.