Deploy Apache Airflow on AWS Elastic Kubernetes Service (EKS)

Data orchestration looks simple right up to the point where platform pressure arrives: schedules collide, Kubernetes workers sit pending, retries overload a warehouse or third-party API, and several teams want one more DAG promoted before the release window closes.
Apache Airflow remains a pragmatic choice for batch and event-adjacent workflow orchestration because it gives engineers mature scheduling, dependency modeling, retries, backfills, pools, task-level observability, and a broad provider ecosystem. The production challenge is not getting one DAG to run. It is keeping the scheduler responsive, the metadata database healthy, logs durable, secrets controlled, and worker capacity elastic as adoption spreads across data, analytics, ML, finance, and platform teams.
Deploying Apache Airflow on AWS Elastic Kubernetes Service (EKS) is a strong pattern when you want Airflow to behave like shared cloud-native infrastructure instead of a carefully nursed server. EKS lets you run task workloads as Kubernetes pods, isolate teams or environments with namespaces and node groups, scale cluster capacity around bursty execution, and integrate cleanly with AWS IAM, networking, storage, logging, and monitoring services. It also makes the operational tradeoffs more explicit: resource requests matter, IAM boundaries are visible, image and dependency management need discipline, and noisy DAGs are easier to contain than they are on a single VM or long-lived worker fleet.
This guide walks through preparing an EKS environment, deploying Airflow with Helm, and wiring in the production-adjacent pieces teams usually need after the first successful install. Along the way, it calls out the decisions that affect day-two operations: executor choice, node sizing, IAM model, metadata database dependency, remote log storage, upgrade safety, and the checks to run when DAGs stop scheduling or worker pods stop starting.

‍

Validate prerequisites and EKS environment assumptions before installing Airflow

Before installing Apache Airflow, confirm that both your local tooling and AWS landing zone are ready. You need AWS CLI v2, eksctl, kubectl, Helm, and permissions to create or administer the EKS cluster, managed node groups, IAM roles, security groups, and supporting VPC resources. In many organizations, the real blockers are not Helm values or Airflow settings; they are account boundaries, subnet routing, DNS behavior, egress controls, private endpoint access, certificate handling, or security group rules discovered after the chart has already been deployed.
If your team uses AWS IAM Identity Center, named CLI profiles, separate workload accounts, private clusters, or pre-approved network designs, verify these items up front:

• The target AWS account, region, VPC, private and public subnet strategy, and outbound access path for pods.
• The Kubernetes version, managed add-ons, and cluster policies expected by your platform team.
• The IAM model for Airflow pods, such as IRSA or EKS Pod Identity, instead of broad node-level permissions.
• The planned exposure model for the Airflow webserver, whether an internal load balancer, ingress, VPN-only access, zero-trust proxy, or another controlled path.
• Where metadata, logs, DAG code, secrets, and container images will live, and which teams own those dependencies.

This walkthrough uses eksctl to create the EKS cluster because it is quick, repeatable, and useful for a hands-on implementation. In a production platform, the cluster may already be managed through Terraform, CloudFormation, Crossplane, or an internal developer platform. If that is your environment, keep the approved provisioning path and skip only the cluster-creation step; creating a parallel cluster for a tutorial usually creates drift, duplicate IAM roles, and unclear ownership.
Before deploying Airflow, verify that kubectl and Helm can reach the intended cluster, the namespace strategy is agreed, IAM permissions are mapped to the correct Kubernetes service accounts, and the node groups have enough spare capacity for both always-on Airflow services and bursty task execution. At minimum, plan for the webserver, scheduler, triggerer, DAG processor, metadata database connectivity, log shipping, and worker pods. Also check whether the cluster already has the add-ons and integrations Airflow will rely on, such as CoreDNS, metrics collection, image pull access, a default storage class if you need persistent volumes, and a supported way to expose the web UI internally. A clean Helm install is not the finish line; the real test is whether the first failed task leaves useful logs, the first burst of DAG runs gets scheduled, and the first permission error is contained to the right workload identity.

Configure AWS identity, eksctl, and kubectl access to the target cluster

1. Install or update AWS CLI v2, then confirm it is authenticated against the AWS account, region, and profile where EKS will run. If your organization uses SSO, IAM Identity Center, role assumption, or multiple named profiles, sign in first and verify the active identity before touching the cluster. The identity should be able to create or manage the EKS control plane, node groups, IAM roles, security groups, and any required VPC resources; otherwise, later failures can look like Kubernetes or Helm issues when they are actually AWS authorization problems. Also make sure your kubectl context points at the intended cluster and region before applying anything. Accidentally deploying Airflow into a shared staging cluster, an old sandbox cluster, or the wrong workload account is a preventable platform incident, not a harmless setup mistake.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install

Please refer to the full AWS installation guide for other operating systems and architectures.

Once Installed, we have to configure the AWS cli on the local machine. Refer to this AWS guide about configuring the CLI locally.

2. Install the eksctl CLI:  (skip to step 3 if you already have eksctl installed)

curl --location "https://github.com/weaveworks/eksctl/releases/download/0.104.0/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin

You can also refer to the eksctl installation guide.

Create the AWS EKS (Elastic Kubernetes Service) Cluster

Create an EKS cluster, or skip this step if you already have a cluster that meets the networking, IAM, and capacity requirements for Airflow.

You can create an EKS Cluster directly from the AWS management console or 

using the eksctl cluster command.

Run the below command to create an EKS cluster in a public subnet in the Oregon region.

eksctl create cluster --name airflow-cluster --region us-west-2 --nodegroup-name standard-workers --node-type t3.medium --nodes 3 --nodes-min 1 --nodes-max 4 --managed

You can find a detailed blog on setting up an EKS Cluster.

Connect to the EKS Cluster from your local machine

1. Install kubectl in your local machine using

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

kubectl version

Please refer to the AWS kubectl & eksctl configuration guide for other operating systems and architectures.

2. After setting up your cluster, you must access it from your local machine. The below command will update the “kubeconfig” file.

aws eks --region us-west-2 update-kubeconfig --name airflow-cluster

‍Setup Helm Locally

Run the below command to install Helm on your local machine.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Please refer to the Installing Helm guide for other operating systems and architectures.

Support Dynamic Volume Provisioning for Persistent Storage using EBS

For an elastic scalable service, dynamic volume provisioning is preferred. Persistent storage must be configured and registered. 

• If your cluster does not already support dynamic persistent volumes, follow this guide to configure the Amazon EBS CSI Driver add-on and dynamic volume provisioning before installing Airflow.

‍

Some background about Airflow

What is Airflow?

Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines and other workflow automation. Teams commonly use it for ETL and ELT jobs, but the same model works for ML workflows, reporting jobs, infrastructure tasks, and cross-system automation. With Airflow, you can create, schedule, and monitor complex workflows, connect multiple data sources and services, and send success or failure notifications to tools such as Slack or email. Workflows are defined in Python as a Directed Acyclic Graph (DAG), which describes the tasks, dependencies, and execution order. Once Airflow is deployed, operators and data engineers use the web UI to inspect DAG runs, retry failed tasks, review logs, and manage pipeline execution.

Use cases of Airflow:

• Data ETL Automation: Streamline the extraction, transformation, and loading of data from various sources into storage systems.
• Data processing: Coordinate batch jobs for cleansing, aggregation, enrichment, validation, and loading data across warehouses, object storage, APIs, and downstream analytics systems.
• Data Migration: Manage data transfer between different systems or cloud platforms.
• Model Training: Automate the training of machine learning models on large datasets.
• Reporting: Generate and distribute reports and analytics dashboards automatically.
• Workflow Automation: Coordinate complex processes with multiple dependencies.
• IoT Data: Analyze and process data from IoT devices.
• Workflow Monitoring: Track workflow progress and receive alerts for issues.

Benefits of using Airflow in Kubernetes

Deploying Apache Airflow on a Kubernetes cluster offers several advantages over deploying it on a virtual machine:

• Scalability: Kubernetes allows you to scale your Airflow deployment horizontally by adding more pods to handle increased workloads automatically.
• Isolation: Enables running different tasks of the same pipeline on various cluster nodes by deploying each task as an isolated pod.
• Automation: Kubernetes-native features such as autoscaling, self-healing, pod scheduling, and rolling updates reduce manual intervention. For Airflow, that means workers can scale with workload demand, failed pods can be replaced automatically, and upgrades can be handled with less disruption when the deployment is configured carefully.
• Portability: Deploying on Kubernetes makes your Airflow setup more portable across different environments, whether on-premise or cloud.
• Integration: Kubernetes integrates seamlessly with various tools for monitoring, logging, and security, enhancing the overall management of your Airflow deployment.

‍

Airflow Architecture Diagram

1. The airflow components are the Executor, Scheduler, Web Server, and Airflow database. The Airflow worker and Triggerer are also involved.
2. As the diagram shows, the data engineer writes Airflow DAGs as Python files. Each DAG defines a workflow: the tasks to run, the dependencies between them, and the order in which Airflow should execute them. In a Kubernetes deployment, those DAG files are usually delivered through a Git-based sync process, a mounted volume, or a custom image, so treat them like application code: version them, review them, and promote them through environments deliberately.
3. The Scheduler picks up these DAGs and has the config to run the tasks specified in the DAGs.
4. In the above diagram, the Scheduler runs tasks using Kubernetes Executor and creates a separate pod for every task, which provides isolation. 
5. Airflow also stores pipeline metadata in an external database. The main configuration file used by the Web server, Scheduler, and workers is airflow.cfg. 
6. The Data Engineer can view the entire flow through the Airflow UI. Users can also check the logs, monitor the pipelines, and set alerts.

‍

Airflow Deployment Options

When deploying Apache Airflow, there are multiple approaches to consider, each with unique advantages and challenges. Let us see the different deployment examples:

• Amazon Managed Workflows for Apache Airflow (MWAA)

You should configure the service through the AWS Management Console. There, you can define your environment, set up necessary permissions, and integrate with other AWS services.

• Google Cloud Composer:

For Google Cloud Composer, create the environment from the Google Cloud Console or your infrastructure-as-code workflow, then connect it to services such as BigQuery and Google Cloud Storage. The managed service removes much of the Airflow control-plane burden, but you still need to manage DAG quality, permissions, networking, and environment sizing.

• Azure Data Factory with Airflow Integration:

Try to Configure Airflow through the Azure Portal. Integrate with other Azure services for efficient workflow automation.

• Self-hosted on AWS EC2:

We can launch and configure EC2 instances. We must install Airflow, set up the environment, configure databases, and set up the scheduler.

• Running on Kubernetes (e.g., AWS EKS):

We can create Kubernetes clusters, deploy Airflow using Helm charts or custom YAML files, and manage container orchestration and scaling.

These are the different options or ways to deploy Airflow, but we are focusing on Amazon Web Service EKS to deploy Airflow, so let us see this in the below section.

‍

Deploy Airflow on AWS EKS

Let us install Apache Airflow in the EKS cluster using the helm chart.

1. Create a new namespace.

kubectl create namespace airflow

2. Add the Helm chart repository.

helm repo add apache-airflow https://airflow.apache.org

3. Update your Helm repository.

helm repo update

4. Deploy Airflow using the remote Helm Chart

helm install airflow apache-airflow/airflow --namespace airflow   --debug

You will get the Airflow webserver and default Postgres connection credentials in the output. Copy them and save them somewhere.

5. Examine the deployments by getting the Pods

Kubectl get pods -n airflow

The Airflow instance is set up in EKS. All the airflow pods should be running.

Let’s prepare Airflow to run our first DAG

At this point, Airflow is deployed using the default configuration. Let's see how we can get the default values from the helm chart on our local machine, modify it, and update a new release.

1. Save the configuration values from the helm chart by running the below command.

helm show values apache-airflow/airflow > values.yaml

This command generates a file named

values.yaml

in your current directory, which you can modify and save as needed.

2. Check the release version of the helm chart by running the following command.

helm ls -n airflow

3. Let us add the ingress configuration to access the airflow instance over the internet. 
We need to deploy an ingress controller in the EKS cluster first. The commands below will install the NGINX ingress controller from the helm repository. 

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm install nginx-ingress ingress-nginx/ingress-nginx --namespace airflow-ingress --create-namespace --set controller.replicaCount=2kubectl get pods -n airflow-ingress

Note - All the pods should be running.

kubectl get service nginx-ingress-controller --namespace airflow-ingress

Look for the external IP in the output of the get service command.

After installing the ingress controller, add the required configuration in the values.yaml file and save the file. There is a section dedicated to the ingress configuration.

# Ingress configuration
ingress:
  enabled: true
  web:
    enabled: true
    annotations: {}
    path: "/"
    pathType: "ImplementationSpecific"
    host: 
    ingressClassName: "nginx"

After the changes to the values in the values.yaml file, we run the helm upgrade command to deploy the changes and create a new release version.

By default, the Helm Chart deploys its own Postgres instance, but using a managed Postgres instance is recommended instead.

You can modify the Helm Chart’s values.yaml file to add  configuration of the managed database and volumes

metadataConnection:
             user: postgres
             pass: postgres
             protocol: postgresql
             host: 
             port: 5432
             db: postgres
             sslmode: disable

Run the helm upgrade command to implement the changes done above.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

Check the release version after the above command is run successfully. You should observe that the revision has changed to 2.

‍

Accessing Airflow UI

We will use port-forwarding to access the Airflow UI in this tutorial. Run the below command and access “localhost:8080” on the browser.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

Use the default webserver credentials saved in the above section, “Installing Airflow Helm chart.”

At this point, Airflow is set up and is accessible. Hurray 😀

You can also access the UI over your domain, which is added in the ingress configuration in the above section.

‍

Create your first Airflow DAG (in Git)

No DAGs have been added to our Airflow deployment yet. Let us see how we can add them.

To Set up a private GitHub repository for DAG, you can create a new one using the Github website's UI.

You can also install the git command line interface on your local machine and run commands to initialize an empty git repo.

git init

Adding DAG configs to the git repo

Once the git repo is initialized, create a DAG file like “sample_dag.py” and push it to the remote branch.

git add .
git commit -m 'Adding first DAG'
git remote add origin
git push -u origin main

Integrate Airflow with a private Git repo

To integrate Airflow with a private Git repository, you will need credentials, i.e. username /password or an SSH key.

We will use the SSH key to connect to the git repo. Skip the first step below if the SSH Key already exists in your Github account.

1. [Skip if it already exists] Generate an SSH key in your local machine and add it to the GitHub account (If not already present). 

ssh-keygen -t ed25519 -C ""

2. Create a generic secret in the namespace where airflow is deployed. This secret contains your SSH key.

kubectl create secret generic airflow-ssh-git-secret --from-file=gitSshKey= -n airflow

3. Update the Git configuration in values.yaml file and run helm update command like in the above section.

gitSync:
    enabled: true
    repo: 
    branch: 
    rev: HEAD
    depth: 1
    maxFailures: 0
    subPath: ""
sshKeySecret: airflow-ssh-git-secret

Below is a “sample_dag.py” that demonstrates a simple workflow. 

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2024, 8, 8),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}
dag = DAG('hello_world', default_args=default_args, schedule_interval=timedelta(days=1))
t1 = BashOperator(
    task_id='say_hello',
    bash_command='echo "Hello World from Airflow!"',
    dag=dag,
)

Upon completion, you can see the DAGs in the UI interface. Airflow automatically detects new DAGs, but you can manually refresh the DAGs list in the Airflow UI by clicking the "Refresh" button on the DAGs page.

The UI has many options/settings to experiment with, such as code, graphs, audit logs, etc.

You can also check the EKS cluster’s activity and DAG dashboard from the Activity tab.

Run the Airflow job

DAGs can be scheduled to run or triggered manually from the UI interface. There is a run button on the rightmost side of the DAG table.

Also, it can be triggered from within the DAG.

‍

Make your Airflow on Kubernetes Production-Grade

Apache Airflow is a powerful tool for orchestrating workflows, but making it production-ready requires careful attention to several key areas. Below, we explore strategies to enhance security, performance, monitoring, and ensure high availability in your Airflow deployment.

‍

1. Improved Security

a. Role-Based Access Control (RBAC)

• Implementation: Enable RBAC in Airflow to ensure only authorized users can access specific features and data.
• Benefits: Limits access to critical areas and reduces the risk of unauthorized changes or data breaches.

Please refer to Access Control guide.

b. Secrets Management

• Implementation: Integrate with external secret management tools like AWS Secrets Manager, HashiCorp Vault, or Kubernetes secrets.
• Benefits: Securely store sensitive information like API keys and database passwords, keeping them out of your codebase.

Refer to this AWS document for Secrets management in EKS

Guide to use Kubernetes secrets in Airflow DAG

c. Network Security

• Implementation: Use network policies and security groups to restrict Airflow's web interface and API access.
• Benefits: Minimizes exposure to potential attacks by limiting network access to trusted sources only.

Refer to this guide to implement Network Security in EKS.

‍

2. Improved Performance

a. Optimized Resource Allocation

• Implementation: Right-size your Kubernetes pods and nodes based on the workload demand. Use Kubernetes Horizontal Pod Autoscaler (HPA) to scale Airflow resources dynamically and cluster autoscaler to scale nodes.
• Benefits: Ensures efficient use of resources, reduces costs, and prevents bottlenecks during peak loads.

Airflow uses Executors for Autoscaling pods. 

Refer to this generic guide on Implementing HPA and Cluster Autoscaler in EKS.
HPA will autoscale the different Airflow components, while the Cluster Autoscaler will make sure there are nodes to satisfy those requirements.

b. Task Parallelism

• Implementation: Configure Airflow to handle parallel task execution by optimizing the number of worker pods and setting appropriate concurrency limits.
• Benefits: Accelerates workflow execution by running multiple tasks simultaneously, improving overall performance.

Check out this guide for Implementing parallelism in Airflow.

c. Use of ARM Instances

• Implementation: Consider running workloads on ARM-based instances like AWS Graviton for cost efficiency.
• Benefits: ARM instances often provide a better cost-to-performance ratio, especially for compute-intensive tasks.

A quick guide to Creating an EKS cluster with ARM instances.

d. Use of HTTPS for ingress host

• Implementation: Consider having HTTPS for the Airflow URL using TLS/SSL certificates with the Ingress controller in Kubernetes.
• Benefits: HTTPS encrypts data to enhance the security of information being transferred. This is especially crucial when handling sensitive data, as encryption helps protect it from unauthorized access during transmission.

Refer to this guide to Install NGINX ingress and configure TLS.

3. Monitoring

a. Metrics Collection and Alerting

• Implementation: Expose Airflow metrics to Prometheus so the platform team can watch scheduler lag, task duration, worker saturation, queue depth, pod restarts, and database connectivity. Use Grafana dashboards for day-to-day visibility, and configure Prometheus Alertmanager to page on symptoms that actually break pipelines, such as a stuck scheduler, exhausted workers, repeated task failures, or missing DAG heartbeats.
• Benefits: It provides visibility into Airflow’s performance, allowing you to identify and address issues proactively and enabling quick response to potential problems, reducing downtime and maintaining workflow reliability.

Refer to the “How to set up Prometheus and Grafana with Airflow” guide.

b. Logs Collection

• Implementation: Set up centralized logging with tools like Elasticsearch, Logstash, Kibana (ELK stack or EFK stack), or Grafana Loki.
• Benefits: Simplifies troubleshooting by consolidating logs from all Airflow components into a single, searchable interface.

Refer to this guide on how to Setup Elastic, Fluentd, and Kibana on EKS.

‍

4. High Availability

a. Redundant Components

• Implementation: Deploy multiple replicas of Airflow’s web server, scheduler, and worker nodes to ensure redundancy.
• Benefits: Increases resilience by preventing single points of failure, ensuring that workflows continue even if one component goes down.

To deploy multiple pods in Apache Airflow using a Helm chart, follow these steps:

1. Set Replicas for the Scheduler:

In your values.yaml file set the scheduler.replicas to the desired number of replicas. For example:

scheduler:
  replicas: 2

2. Set Replicas for the Web Server:

Similarly, set the web.replicas to deploy multiple web server pods:

web:
  replicas: 2

3. Deploy the Helm Chart:

Apply the Helm chart with the updated values.yaml file:

helm upgrade --install airflow apache-airflow/airflow -f values.yaml

This configuration ensures that multiple scheduler and web server pods are deployed, contributing to the high availability of your Airflow setup.

Airflow helm chart’s value.yaml file can be found here.

b. Database High Availability

• Implementation: Use a highly available database solution like Amazon RDS with Multi-AZ deployment for Airflow’s metadata database.
• Benefits: Ensures continuous operation and data integrity even during a database failure.

Refer to Amazon RDS with the Multi-AZ deployment guide.

c. Backup and Disaster Recovery

• Implementation: Regularly backup Airflow’s database and configuration files. Implement a disaster recovery plan that includes rapid failover procedures.
• Benefits: Protects against data loss and enables quick recovery in case of catastrophic failures.

Read this document to set up automated backups in Amazon RDS.

Refer to this AWS page to learn about “Backup and Restore of EKS.”

Conclusion

Setting up Apache Airflow on Amazon EKS is a powerful way to manage your workflows at scale, but it requires careful planning and configuration to ensure it’s production-ready. Following this guide, you've deployed Airflow on EKS, created a simple DAG, connected Airflow with a private Git repository, and learned about different ways to implement security, performance, high availability, monitoring, and logging. With these optimizations, your Airflow deployment is now more efficient, cost-effective, and ready to handle the demands of real-world data orchestration.

Frequently Asked Questions

1. What is Apache Airflow?

• Apache Airflow is an open-source tool that helps in orchestrating and managing workflows through Directed Acyclic Graphs (DAGs). It automates complex processes like ETL (Extract, Transform, Load) jobs, machine learning pipelines, and more.

2. Why deploy Airflow on Amazon EKS?

• Deploying Airflow on Amazon EKS offers scalability, flexibility, and robust workflow management. EKS simplifies Kubernetes management, allowing you to focus on scaling and securing your Airflow environment.

3. What are the prerequisites for deploying Airflow on EKS?

• You need an AWS account, an EKS cluster, kubectl configured on your local environment, a dynamic storage class using EBS volumes, and Helm for package management.

4. How do I monitor Airflow on EKS?

• You can integrate Prometheus and Grafana for monitoring. Using Loki for log aggregation can also help in centralized log management and troubleshooting.

5. What Kubernetes add-ons are recommended for a production-grade Airflow setup?

• Essential add-ons include External Secret Operator for secure secrets management, Prometheus and Grafana for monitoring, and possibly Loki for logging.

6. Can Airflow be integrated with external databases like RDS?

• Yes, it’s common to configure Airflow to use an external PostgreSQL database hosted on Amazon RDS for production environments, providing reliability and scalability for your metadata storage.

7. How can I access the Airflow UI on EKS?

• You can access the Airflow UI by setting up a LoadBalancer service or using an Ingress Controller with a DNS pointing to your load balancer for easy access.

8. How do I manage DAGs in a production environment?

• For production, keep DAGs in a private Git repository and let the Airflow deployment pull them with the Helm chart’s `gitSync` sidecar instead of baking DAGs into the image for every change. Pin the sync target to a branch, tag, or commit strategy that matches your release process, store credentials in Kubernetes Secrets, and treat DAG updates like application releases: reviewed, tested, and easy to roll back when a bad DAG breaks scheduling.