“We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubernetes jobs, managing infra monitoring with Datadog, etc., etc.”
DAGs can be published to S3 for cutting down on like half of these dependencies. And the nice thing about MWAA is log & stats publishing over cloudwatch, which should flow into any existing amazon integrated tooling.
For our team setting up terraform for iam & mwaa, some deploy pipelines to s3, and connecting some config bits to wire up splunk logs / monitoring pieces was not that much work. Initiating a separated vendor relationship & pricing out data ingress/egress costs would blow that work out of the water but maybe it’s a difference in company size/placement.
DAGs can be published to S3 for cutting down on like half of these dependencies. And the nice thing about MWAA is log & stats publishing over cloudwatch, which should flow into any existing amazon integrated tooling.
For our team setting up terraform for iam & mwaa, some deploy pipelines to s3, and connecting some config bits to wire up splunk logs / monitoring pieces was not that much work. Initiating a separated vendor relationship & pricing out data ingress/egress costs would blow that work out of the water but maybe it’s a difference in company size/placement.