Deploy Apache Airflow on AWS Elastic Kubernetes Service (EKS)

Airflow becomes a platform concern as soon as more than one team relies on it for production work. Morning schedules stack up, month-end jobs create worker-pod pressure, retries amplify load against warehouses and third-party APIs, and a high-priority DAG often lands during the maintenance window that was supposed to be quiet.
Apache Airflow remains a pragmatic orchestration choice for batch and event-adjacent workflows because it gives teams mature scheduling, dependency modeling, retries, backfills, pools, task-level visibility, and a deep provider ecosystem. The hard part is not getting a demo DAG to turn green. The hard part is keeping schedulers responsive, protecting the metadata database, keeping logs after short-lived pods exit, scoping secrets and AWS permissions, and scaling workers when several teams share the same control plane.
Deploying Apache Airflow on AWS Elastic Kubernetes Service (EKS) is a strong pattern when Airflow needs to behave like cloud-native infrastructure instead of a carefully maintained VM. EKS lets you run task workloads as Kubernetes pods, isolate environments with namespaces and node groups, scale around bursty execution, and integrate with AWS identity, networking, storage, logging, and observability. It also makes the tradeoffs harder to ignore: resource requests must reflect real DAG behavior, IAM boundaries need to be explicit, Python dependencies should be baked into versioned images, and noisy workflows need pools, queues, priority weights, and concurrency limits before they starve the rest of the platform.
In this guide you will prepare an EKS cluster, deploy Airflow with the official Helm chart, open the Airflow UI, sync DAGs from a private Git repository, run a first DAG, and then harden the setup for production. The emphasis is day-two operation: executor choice, workload identity, the metadata database, remote logs, and upgrade safety.

A note on versions: recent releases of the official Airflow Helm chart deploy Airflow 3.x by default. Airflow 3 replaced the webserver with an API server (which serves the UI), runs the DAG processor as its own component, and has workers talk to the API server instead of the metadata database. Some steps below were written for Airflow 2.x; where Airflow 3 differs, a note says so.

‍

Some background about Airflow

What is Airflow?

Apache Airflow is an open-source platform for authoring, scheduling, and operating workflows. Teams often start with ETL or ELT pipelines, then extend the same pattern to ML jobs, reporting runs, infrastructure automation, and cross-system maintenance tasks. Airflow lets engineers create, schedule, and monitor workflows across multiple services, with notifications through Slack, email, or another integration when a run succeeds, fails, or needs attention. Each workflow is written in Python as a Directed Acyclic Graph (DAG), which defines the tasks, dependencies, and allowed execution order. After deployment, the web UI becomes the operating console: engineers inspect DAG runs, retry failed tasks, read logs, pause risky pipelines, and find out whether a workflow is blocked by code, infrastructure, data quality, or an upstream dependency.

Use cases of Airflow:

• Data ETL Automation: Streamline the extraction, transformation, and loading of data from various sources into storage systems.
• Data processing: Coordinate batch jobs for cleansing, aggregation, enrichment, validation, and loading data across warehouses, object storage, APIs, and downstream analytics systems.
• Data Migration: Manage data transfer between different systems or cloud platforms.
• Model Training: Automate the training of machine learning models on large datasets.
• Reporting: Generate and distribute reports and analytics dashboards automatically.
• Workflow Automation: Coordinate complex processes with multiple dependencies.
• IoT Data: Analyze and process data from IoT devices.
• Workflow Monitoring: Track workflow progress and receive alerts for issues.

Benefits of using Airflow in Kubernetes

Deploying Apache Airflow on a Kubernetes cluster offers several advantages over deploying it on a virtual machine:

• Scalability: Kubernetes allows you to scale your Airflow deployment horizontally by adding more pods to handle increased workloads automatically.
• Isolation: Enables running different tasks of the same pipeline on various cluster nodes by deploying each task as an isolated pod.
• Automation: Kubernetes-native features such as autoscaling, self-healing, pod scheduling, and rolling updates reduce manual intervention. For Airflow, that means workers can scale with workload demand, failed pods can be replaced automatically, and upgrades can be handled with less disruption when the deployment is configured carefully.
• Portability: Deploying on Kubernetes makes your Airflow setup more portable across different environments, whether on-premise or cloud.
• Integration: Kubernetes works with common tools for monitoring, logging, and security, which makes the Airflow deployment easier to operate.

‍

Airflow Architecture Diagram

1. The core Airflow components are the scheduler (which runs the executor), the web server, the DAG processor, the triggerer, the workers, and the metadata database. In Airflow 3, the web server is replaced by the API server, which serves the UI and the REST API, and the DAG processor always runs as a separate component.
2. As the diagram shows, the data engineer writes Airflow DAGs as Python files. Each DAG defines a workflow: the tasks to run, the dependencies between them, and the order in which Airflow should execute them. In a Kubernetes deployment, those DAG files are usually delivered through a Git-based sync process, a mounted volume, or a custom image, so treat them like application code: version them, review them, and promote them through environments deliberately.
3. The Scheduler picks up these DAGs and has the config to run the tasks specified in the DAGs.
4. In the above diagram, the Scheduler runs tasks using Kubernetes Executor and creates a separate pod for every task, which provides isolation. The official Helm chart defaults to the CeleryExecutor instead, which keeps long-running worker pods that pull tasks from a Redis queue.
5. Airflow also stores pipeline metadata in an external database, typically PostgreSQL (on AWS, commonly Amazon RDS or Aurora PostgreSQL). The main configuration file used by the Airflow components is airflow.cfg. On Kubernetes, the Helm chart renders it from the config section of values.yaml, and any setting can also be overridden with AIRFLOW__SECTION__KEY environment variables.
6. The Data Engineer can view the entire flow through the Airflow UI. Users can also check the logs, monitor the pipelines, and set alerts.

‍

Airflow Deployment Options

There are several ways to run Apache Airflow, each with its own tradeoffs:

• Amazon Managed Workflows for Apache Airflow (MWAA)

MWAA is the managed Airflow service from AWS. You create an environment from the AWS Management Console or infrastructure as code, set up permissions, and integrate with other AWS services, while AWS runs the scheduler, workers, web server, and metadata database. MWAA supports Airflow 3.x, but you are limited to the Airflow versions and environment sizes AWS offers.

• Google Cloud Composer:

For Google Cloud Composer, create the environment from the Google Cloud Console or your infrastructure-as-code workflow, then connect it to services such as BigQuery and Google Cloud Storage. The managed service removes much of the Airflow control-plane burden, but you still need to manage DAG quality, permissions, networking, and environment sizing.

• Azure Data Factory with Airflow Integration:

Configure Airflow through the Azure Portal and integrate it with other Azure services.

• Self-hosted on AWS EC2:

You launch and configure EC2 instances, then install Airflow, set up the environment, configure the database, and run the scheduler yourself. This gives full control, but upgrades, scaling, and recovery are all on you.

• Running on Kubernetes (e.g., AWS EKS):

You create the Kubernetes cluster, deploy Airflow with the official Helm chart or custom manifests, and manage scaling and upgrades. You get more control than a managed service offers, in exchange for more operational work.

This guide focuses on running Airflow on AWS EKS. If you are still choosing between EKS and MWAA, see the FAQ at the end of this post.

‍

Prerequisites: validate EKS and production assumptions

Before installing Apache Airflow, make sure the operator workstation or deployment pipeline and the AWS landing zone are actually ready. You need AWS CLI v2, kubectl, Helm, and eksctl if eksctl is the tool you will use to create the cluster. You also need permissions to create or administer the EKS control plane, managed node groups, IAM roles, security groups, and the VPC components the cluster depends on. In practice, the first blockers are rarely in the Airflow Helm values. They are usually account boundaries, subnet routing, DNS, egress policy, private endpoint access, TLS, image registry access, or security group rules that only show up once pods start failing.
If your team uses AWS IAM Identity Center, named CLI profiles, separate workload accounts, private EKS clusters, or standard network blueprints, validate these items before installing anything:

• The target AWS account, region, cluster name, VPC, subnet strategy, and outbound access path for Airflow system pods and task pods.
• The Kubernetes version, managed add-ons, ingress or load balancer controller requirements, certificate path, DNS model, and any admission policies enforced by the platform team.
• The workload IAM model for Airflow, such as IAM Roles for Service Accounts (IRSA) or EKS Pod Identity, instead of broad node-level permissions. AWS now recommends EKS Pod Identity where possible: it needs the EKS Pod Identity Agent add-on, but no per-cluster OIDC provider or trust policy updates. IRSA still works and is the option for Kubernetes clusters outside EKS.
• The exposure model for the Airflow webserver, such as an internal load balancer, ingress, VPN-only access, zero-trust proxy, or another controlled access path.
• Where the metadata database, remote logs, DAG code, secrets, and container images will live, and which team owns each dependency.
• The approved source for Airflow images and DAG-related dependencies, whether that is Amazon ECR, another private registry, or an allowed external registry.
• The backup, retention, upgrade, and disaster recovery expectations for the metadata database, remote logs, DAG storage, and any persistent volumes.
• The expected task profile: CPU-heavy transforms, memory-heavy Python jobs, long-running sensors, KubernetesPodOperator workloads, deferrable operators, or short high-volume tasks that stress the scheduler and metadata database in different ways.

This walkthrough uses eksctl because it is a fast, repeatable way to create an EKS cluster for a hands-on Airflow deployment. In a production platform, the cluster is often owned by Terraform, CloudFormation, Crossplane, or an internal provisioning workflow. If that is your environment, use the approved path and treat the eksctl section as a reference shape, not a competing source of truth. A tutorial cluster created outside the normal process is a common source of drift: duplicate IAM roles, unmanaged node groups, missing tags, and public endpoints no one approved.
Before installing Airflow, verify the cluster from the operator’s point of view. kubectl and Helm should target the intended cluster and namespace. AWS permissions should be attached to Kubernetes service accounts through IRSA or EKS Pod Identity instead of inherited broadly from node roles. Node groups need enough steady-state capacity for the webserver, scheduler, triggerer, DAG processor, metadata database connectivity, and platform agents, plus headroom for task bursts. If you rely on Cluster Autoscaler, Karpenter, or another scaling mechanism, confirm it can add the right instance types quickly enough for worker pods with realistic CPU, memory, ephemeral storage, availability zone, and CPU architecture requirements. Check taints, tolerations, labels, and topology rules early. Many Airflow worker failures are really unschedulable pod failures caused by placement constraints that were never tested with production-sized tasks.
Also confirm the cluster services Airflow depends on: healthy CoreDNS, working metrics collection, image pull access, a default storage class if persistent volumes are required, network access to the metadata database, and an approved route to the web UI, preferably internal or otherwise restricted. If the deployment uses remote logging, validate the bucket, encryption, lifecycle policy, and write permissions before the first production DAG runs. A successful Helm release is not the finish line. The useful acceptance test is operational: failed tasks leave readable remote logs, a burst of DAG runs does not starve the cluster, worker pods start without image, networking, scheduling, or IAM errors, and a Kubernetes permission issue is isolated to the correct workload identity instead of hidden by an overly permissive node role. If those checks fail, pause before onboarding more DAGs; otherwise, the first busy business cycle becomes your load test.

Set up AWS identity, eksctl, and kubectl with clear workload boundaries

Install or update AWS CLI v2, then verify that it is authenticated against the exact AWS account, region, and profile where the EKS cluster will run. If your organization uses IAM Identity Center, SSO, role assumption, or named profiles, sign in first and run a caller-identity check before creating or changing resources. The provisioning identity needs enough access to manage the EKS control plane, node groups, IAM roles, security groups, and any required VPC resources. If it does not, the failure may not show up at creation time. It can surface later as a confusing Kubernetes scheduling issue, image pull error, or failed Helm release.
Keep three identities distinct in the runbook: the human or pipeline identity that provisions infrastructure, the Kubernetes identity that installs Airflow resources, and the runtime identity that Airflow pods use when they call AWS APIs. That separation makes S3 remote logs, Secrets Manager reads, ECR pulls, metadata database access, and DAG-level AWS permissions diagnosable. A scheduler pod that can parse DAG files or write logs should not automatically mean every task pod can read every secret in the account. Map permissions to service accounts and workloads deliberately, then document which Airflow components are allowed to use each role.
For Airflow on EKS, treat workload identity as part of the application design. The webserver, scheduler, workers, triggerer, DAG processor, and task pods rarely need identical AWS access. A platform team might allow the scheduler to write remote logs, let selected DAG tasks assume narrowly scoped roles, and keep the webserver away from data-plane permissions. That takes more work than attaching broad access to the node role, but it turns incidents into bounded failures instead of account-wide exposure.
Before applying manifests or running Helm, confirm that kubeconfig, kubectl, Helm, and the namespace context all point to the intended cluster. Deploying Airflow into a shared staging cluster, an old sandbox, or the wrong workload account is not a harmless setup mistake; it is a platform incident waiting to happen. Show the account, region, cluster, and namespace in the terminal prompt or CI output, and make the pipeline fail closed when they do not match the expected target.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install

Please refer to the full AWS installation guide for other operating systems and architectures.

Once Installed, we have to configure the AWS cli on the local machine. Refer to this AWS guide about configuring the CLI locally.

2. Install the eksctl CLI (skip this step if you already have eksctl installed):

curl --location "https://github.com/weaveworks/eksctl/releases/download/0.104.0/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin

Note: the command above pins an old eksctl release from the former weaveworks/eksctl repository that does not support current EKS versions. eksctl now lives under the eksctl-io GitHub organization. Follow the official eksctl installation instructions, which download the latest release, or run brew install aws/tap/eksctl on macOS.

You can also refer to the eksctl installation guide.

Create the AWS EKS (Elastic Kubernetes Service) Cluster

Create an EKS cluster, or skip this step if you already have a cluster that meets the networking, IAM, and capacity requirements for Airflow.

You can create an EKS cluster from the AWS Management Console, but this guide uses eksctl from the command line so the control plane, worker nodes, and supporting AWS resources are created consistently.

Run the following command to create an EKS cluster in a public subnet in the Oregon region. For a production environment, review the networking, private endpoint, and node group choices against your organization’s baseline before reusing this shape.

eksctl create cluster --name airflow-cluster --region us-west-2 --nodegroup-name standard-workers --node-type t3.medium --nodes 3 --nodes-min 1 --nodes-max 4 --managed

AWS documents the full process in its guide to setting up an EKS Cluster.

Connect to the EKS Cluster from your local machine

1. Install kubectl on your local machine:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

kubectl version

Please refer to the AWS kubectl & eksctl configuration guide for other operating systems and architectures.

2. After the cluster is created, configure local access by updating your kubeconfig file. This makes kubectl and Helm point at the new EKS cluster, so verify the active context before installing Airflow.

aws eks --region us-west-2 update-kubeconfig --name airflow-cluster

‍Setup Helm Locally

Run the below command to install Helm on your local machine.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Please refer to the Installing Helm guide for other operating systems and architectures.

Support Dynamic Volume Provisioning for Persistent Storage using EBS

Several chart components can request persistent volumes, such as the bundled PostgreSQL database, Redis, and Celery workers, so the cluster needs a default storage class that provisions volumes on demand. On EKS that is the Amazon EBS CSI driver, available as an EKS add-on. The driver needs IAM permissions, which you can grant through EKS Pod Identity or IRSA.

• If the cluster does not already support dynamic persistent volumes, configure the Amazon EBS CSI Driver add-on and dynamic volume provisioning before installing Airflow, or confirm that your platform team provides an equivalent default storage class.

‍

Deploy Airflow on AWS EKS

Install Apache Airflow in the EKS cluster using the official Apache Airflow Helm chart.

1. Create a new namespace.

kubectl create namespace airflow

2. Add the Helm chart repository.

helm repo add apache-airflow https://airflow.apache.org

3. Update your Helm repository.

helm repo update

4. Deploy Airflow using the remote Helm Chart

helm install airflow apache-airflow/airflow --namespace airflow   --debug

Without a --version flag, this installs the latest chart, which on recent chart releases means Airflow 3.x with the CeleryExecutor. For repeatable installs, pin the chart with --version and set the Airflow version explicitly (the chart's airflowVersion and defaultAirflowTag values), so a later helm repo update does not silently change what you deploy.

The command output can include Airflow webserver details, generated passwords, connection strings, Helm notes, or default database credentials. Treat that output as sensitive operational material. Move any values you need into your secret store, rotate the defaults before the environment is shared, and keep raw credentials out of tickets, chat, and CI logs.

5. Examine the deployments by getting the Pods

Kubectl get pods -n airflow

The Airflow instance is set up in EKS. All the Airflow pods should be running (type the command in lowercase, since kubectl is case-sensitive). On Airflow 3, the list shows an api-server pod instead of a webserver pod, plus a separate dag-processor pod.

Let’s prepare Airflow to run our first DAG

At this point, Airflow is deployed using the default configuration. Let's see how we can get the default values from the helm chart on our local machine, modify it, and update a new release.

1. Save the configuration values from the helm chart by running the below command.

helm show values apache-airflow/airflow > values.yaml

This command generates a file named

values.yaml

in your current directory, which you can modify and save as needed.

2. Check the release version of the helm chart by running the following command.

helm ls -n airflow

3. Add ingress only after deciding how the Airflow UI should be reached. 
For a production environment, avoid exposing Airflow directly to the public internet unless you have an explicit security requirement and controls such as SSO, TLS, IP allow lists, and audit logging. The example below installs the NGINX ingress controller from its Helm repository so the cluster can route HTTP traffic to the Airflow webserver. Use an internal load balancer or private ingress pattern if Airflow is meant for operators on a corporate network or VPN.

Note: the Kubernetes project retired the community ingress-nginx controller, and it has had no releases or security fixes since March 2026. It still works for a lab cluster, but for production on EKS consider the AWS Load Balancer Controller (for example, an internal ALB) or a Gateway API implementation.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm install nginx-ingress ingress-nginx/ingress-nginx --namespace airflow-ingress --create-namespace --set controller.replicaCount=2kubectl get pods -n airflow-ingress

Note - All the pods should be running.

kubectl get service nginx-ingress-controller --namespace airflow-ingress

Check the service or ingress output for the address AWS assigned to the Airflow webserver. A new load balancer can remain pending for several minutes, so rerun the check before changing chart values or reinstalling the release. If it stays pending, inspect the ingress or load balancer controller service, subnet tags, security groups, controller logs, and Kubernetes events. At that stage, the issue is usually EKS integration or AWS networking rather than Airflow itself.

After installing the ingress controller, add the required configuration in the values.yaml file and save the file. There is a section dedicated to the ingress configuration. On Airflow 3, configure the UI ingress under ingress.apiServer; the ingress.web block shown below applies to Airflow 2.x.

# Ingress configuration
ingress:
  enabled: true
  web:
    enabled: true
    annotations: {}
    path: "/"
    pathType: "ImplementationSpecific"
    host: <your domain URL>
    ingressClassName: "nginx"

After the changes to the values in the values.yaml file, we run the helm upgrade command to deploy the changes and create a new release version.

By default, the Helm chart deploys its own PostgreSQL instance. For anything beyond a test, use a managed database such as Amazon RDS for PostgreSQL instead.

In values.yaml, set postgresql.enabled to false and add the managed database connection under data.metadataConnection:

metadataConnection:
             user: postgres
             pass: postgres
             protocol: postgresql
             host: <YOUR_POSTGRES_HOST>
             port: 5432
             db: postgres
             sslmode: disable

The example uses plain-text credentials and sslmode: disable to keep it short. For RDS, store the connection string in a Kubernetes Secret referenced by data.metadataSecretName, use sslmode: require, and consider enabling the chart's PgBouncer, because Airflow opens many database connections.

Run the helm upgrade command to implement the changes done above.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

Check the release version after the above command is run successfully. You should observe that the revision has changed to 2.

Send task logs to S3

With the KubernetesExecutor, a task's local log disappears with its pod. Configure remote logging to Amazon S3 so logs survive pod restarts and stay readable in the UI. Set these options in the config.logging section of values.yaml (or as AIRFLOW__LOGGING__* environment variables): remote_logging: 'True', remote_base_log_folder: s3://<your-bucket>/airflow-logs, and remote_log_conn_id: aws_default. Leave the AWS connection without stored keys and grant S3 access to the Airflow service accounts through EKS Pod Identity or IRSA.

‍

Accessing Airflow UI

We will use port-forwarding to access the Airflow UI in this tutorial. Run the below command and access “localhost:8080” on the browser.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

The port-forward command depends on your Airflow version (both assume the Helm release is named airflow): kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow on Airflow 2.x, or kubectl port-forward svc/airflow-api-server 8080:8080 -n airflow on Airflow 3.x.

Log in with the default credentials from the Helm install output in the “Deploy Airflow on AWS EKS” section above, and change them before anyone else uses the environment.

At this point, Airflow is set up and is accessible. Hurray 😀

You can also access the UI over your domain, which is added in the ingress configuration in the above section.

‍

Create your first Airflow DAG (in Git)

No DAGs have been added to our Airflow deployment yet. Let us see how we can add them.

To set up a private GitHub repository for your DAGs, create one in the Github website's UI.

You can also install the Git command-line interface on your local machine and initialize an empty repository there. This is useful when you want a simple DAG-sync workflow, but keep the repository structure intentional: separate DAG code from local experiments, avoid committing credentials, and use branches or pull requests if multiple engineers will update workflows.

git init

Adding DAG configs to the git repo

Once the git repo is initialized, create a DAG file like “sample_dag.py” and push it to the remote branch.

git add .
git commit -m 'Adding first DAG'
git remote add origin
git push -u origin main

Integrate Airflow with a private Git repo

To integrate Airflow with a private Git repository, you need credentials: a username and access token, or an SSH key.

We will use the SSH key to connect to the git repo. Skip the first step below if the SSH Key already exists in your Github account.

1. [Skip if it already exists] Generate an SSH key in your local machine and add it to the GitHub account (If not already present). 

ssh-keygen -t ed25519 -C "<mailaddress>"

2. Create a generic secret in the same namespace where Airflow is deployed. This secret holds the SSH key used by Airflow or the DAG-sync mechanism to read from the repository, so keep it namespace-scoped, restrict repository permissions to the minimum required access, and avoid reusing a personal developer key with broader privileges.

kubectl create secret generic airflow-ssh-git-secret --from-file=gitSshKey=<path to SSH private key> -n airflow

3. Update the Git settings in the values.yaml file, then run the same Helm upgrade command shown in the previous section to apply the change.

gitSync:
    enabled: true
    repo: <your private Git repository URL>
    branch: <Branch-name>
    rev: HEAD
    depth: 1
    maxFailures: 0
    subPath: ""
sshKeySecret: airflow-ssh-git-secret

In the chart's values.yaml, gitSync sits under dags:, and sshKeySecret belongs inside the gitSync block (dags.gitSync.sshKeySecret). The chart's production guide also recommends setting dags.gitSync.knownHosts to GitHub's verified host key.

Below is a “sample_dag.py” that demonstrates a simple workflow.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2024, 8, 8),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}
dag = DAG('hello_world', default_args=default_args, schedule_interval=timedelta(days=1))
t1 = BashOperator(
    task_id='say_hello',
    bash_command='echo "Hello World from Airflow!"',
    dag=dag,
)

This sample uses Airflow 2.x syntax. Airflow 3 removed the airflow.operators.bash_operator import path and the schedule_interval argument, so on Airflow 3 import BashOperator from airflow.providers.standard.operators.bash and pass schedule=timedelta(days=1) instead. Running ruff check --select AIR301 on your DAG folder flags these issues.

Once the sync runs, the DAG appears in the UI. Airflow detects new DAG files automatically, but it scans the DAG folder on an interval, so a new DAG can take a few minutes to show up.

The UI has many options/settings to experiment with, such as code, graphs, audit logs, etc.

You can also check the EKS cluster’s activity and DAG dashboard from the Activity tab.

Run the Airflow job

DAGs can be scheduled to run or triggered manually from the UI interface. There is a run button on the rightmost side of the DAG table.

Also, it can be triggered from within the DAG.

‍

Make your Airflow on Kubernetes Production-Grade

Getting Airflow to run is the easy part. Making it production-ready takes work in four areas: security, performance, monitoring, and high availability.

‍

1. Improved Security

a. Role-Based Access Control (RBAC)

• Implementation: Enable RBAC in Airflow to ensure only authorized users can access specific features and data.
• Benefits: Limits access to critical areas and reduces the risk of unauthorized changes or data breaches.

Please refer to Access Control guide.

b. Secrets Management

• Implementation: Integrate with external secret management tools like AWS Secrets Manager, HashiCorp Vault, or Kubernetes secrets.
• Benefits: Securely store sensitive information like API keys and database passwords, keeping them out of your codebase.

Refer to this AWS document for Secrets management in EKS

Airflow can also read connections and variables straight from AWS Secrets Manager through the Amazon provider's secrets backend (airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend), so credentials never live in DAG code or values.yaml.

Guide to use Kubernetes secrets in Airflow DAG

c. Network Security

• Implementation: Use network policies and security groups to restrict Airflow's web interface and API access.
• Benefits: Minimizes exposure to potential attacks by limiting network access to trusted sources only.

Refer to this guide to implement Network Security in EKS.

‍

2. Improved Performance

a. Right-size Airflow components and worker pods

• Implementation: Right-size your Kubernetes pods and nodes based on the workload demand. Use Kubernetes Horizontal Pod Autoscaler (HPA) to scale Airflow resources dynamically and cluster autoscaler to scale nodes.
• Benefits: Ensures efficient use of resources, reduces costs, and prevents bottlenecks during peak loads.

Airflow uses Executors for Autoscaling pods. With the CeleryExecutor, the chart can autoscale worker pods with KEDA based on queued and running tasks. With the KubernetesExecutor, every task is its own pod, so capacity comes from the node autoscaler (Cluster Autoscaler or Karpenter).

Refer to these guides for implementing HPA on EKS and using the Cluster Autoscaler in EKS.
Use HPA carefully with Airflow components. It can help scale stateless or worker workloads when the metrics match the bottleneck, but scaling the scheduler or webserver blindly can create more noise than capacity. The Cluster Autoscaler solves a different problem: it adds or removes nodes when pending pods cannot be scheduled. In practice, tune both together so worker pods can scale out during DAG bursts without leaving the cluster permanently overprovisioned.

b. Task Parallelism

• Implementation: Configure Airflow to handle parallel task execution by optimizing the number of worker pods and setting appropriate concurrency limits.
• Benefits: Accelerates workflow execution by running multiple tasks simultaneously, improving overall performance.

Check out this guide for Implementing parallelism in Airflow.

c. Use of ARM Instances

• Implementation: Consider running workloads on ARM-based instances like AWS Graviton for cost efficiency.
• Benefits: ARM instances often provide a better cost-to-performance ratio, especially for compute-intensive tasks.

A quick guide to Creating an EKS cluster with ARM instances.

d. Use of HTTPS for ingress host

• Implementation: Consider having HTTPS for the Airflow URL using TLS/SSL certificates with the Ingress controller in Kubernetes.
• Benefits: HTTPS encrypts data to enhance the security of information being transferred. This is especially crucial when handling sensitive data, as encryption helps protect it from unauthorized access during transmission.

Refer to this guide to Install NGINX ingress and configure TLS.

3. Monitoring

a. Collect actionable metrics and alerts

• Implementation: Expose Airflow metrics to Prometheus so the platform team can watch scheduler lag, task duration, worker saturation, queue depth, pod restarts, and database connectivity. Use Grafana dashboards for day-to-day visibility, and configure Prometheus Alertmanager to page on symptoms that actually break pipelines, such as a stuck scheduler, exhausted workers, repeated task failures, or missing DAG heartbeats.
• Benefits: It provides visibility into Airflow’s performance, allowing you to identify and address issues proactively and enabling quick response to potential problems, reducing downtime and maintaining workflow reliability.

Refer to the “How to set up Prometheus and Grafana with Airflow” guide.

b. Log Collection

• Implementation: Set up centralized logging with tools like Elasticsearch, Logstash, Kibana (ELK stack or EFK stack), or Grafana Loki.
• Benefits: Simplifies troubleshooting by consolidating logs from all Airflow components into a single, searchable interface.

Refer to this guide on how to Setup Elastic, Fluentd, and Kibana on EKS.

‍

4. High Availability

a. Redundant Components

• Implementation: Deploy multiple replicas of Airflow’s web server, scheduler, and worker nodes to ensure redundancy.
• Benefits: Increases resilience by preventing single points of failure, ensuring that workflows continue even if one component goes down.

Use the Helm chart to run the Airflow components as separate Kubernetes pods, then verify that each component is scheduled, healthy, and using the expected service account:

1. Set Replicas for the Scheduler:

In your values.yaml file set the scheduler.replicas to the desired number of replicas. For example:

scheduler:
  replicas: 2

2. Set Replicas for the Web Server:

Similarly, set the replica count for the web server to deploy multiple web server pods. In the chart, the key is webserver.replicas on Airflow 2.x and apiServer.replicas on Airflow 3.x:

web:
  replicas: 2

3. Deploy the Helm Chart:

Apply the Helm chart with the updated values.yaml file:

helm upgrade --install airflow apache-airflow/airflow -f values.yaml

This configuration ensures that multiple scheduler and web server pods are deployed, contributing to the high availability of your Airflow setup.

Airflow helm chart’s value.yaml file can be found here.

b. Database High Availability

• Implementation: Use a highly available database solution like Amazon RDS with Multi-AZ deployment for Airflow’s metadata database.
• Benefits: Ensures continuous operation and data integrity even during a database failure.

Refer to Amazon RDS with the Multi-AZ deployment guide.

c. Backup and Disaster Recovery

• Implementation: Regularly backup Airflow’s database and configuration files. Implement a disaster recovery plan that includes rapid failover procedures.
• Benefits: Protects against data loss and enables quick recovery in case of catastrophic failures.

Read this document to set up automated backups in Amazon RDS.

Refer to this AWS page to learn about “Backup and Restore of EKS.”

Conclusion

Setting up Apache Airflow on Amazon EKS is a powerful way to manage your workflows at scale, but it requires careful planning and configuration to ensure it’s production-ready. Following this guide, you've deployed Airflow on EKS, created a simple DAG, connected Airflow with a private Git repository, and learned about different ways to implement security, performance, high availability, monitoring, and logging. With these changes in place, your Airflow deployment is better prepared for real-world data orchestration. If you are moving an existing Airflow 2 deployment to Airflow 3, review the version notes above and the upgrade question in the FAQ before you upgrade.

Frequently Asked Questions

1. What is Apache Airflow?

Apache Airflow is an open-source tool that helps in orchestrating and managing workflows through Directed Acyclic Graphs (DAGs). It automates complex processes like ETL (Extract, Transform, Load) jobs, machine learning pipelines, and more.

2. Why deploy Airflow on Amazon EKS?

Deploying Airflow on Amazon EKS offers scalability, flexibility, and robust workflow management. EKS simplifies Kubernetes management, allowing you to focus on scaling and securing your Airflow environment.

3. What are the prerequisites for deploying Airflow on EKS?

You need an AWS account, an EKS cluster, kubectl configured on your local environment, a dynamic storage class using EBS volumes, and Helm for package management.

4. How do I monitor Airflow on EKS?

You can integrate Prometheus and Grafana for monitoring. Using Loki for log aggregation can also help in centralized log management and troubleshooting.

5. What Kubernetes add-ons are recommended for a production-grade Airflow setup?

Essential add-ons include External Secret Operator for secure secrets management, Prometheus and Grafana for monitoring, and possibly Loki for logging.

6. Can Airflow be integrated with external databases like RDS?

Yes, it’s common to configure Airflow to use an external PostgreSQL database hosted on Amazon RDS for production environments, providing reliability and scalability for your metadata storage.

7. How can I access the Airflow UI on EKS?

You can access the Airflow UI by setting up a LoadBalancer service or using an Ingress Controller with a DNS pointing to your load balancer for easy access.

8. How do I manage DAGs in a production environment?

For production, keep DAGs in a private Git repository and let the Airflow deployment pull them with the Helm chart’s `gitSync` sidecar instead of baking DAGs into the image for every change. Pin the sync target to a branch, tag, or commit strategy that matches your release process, store credentials in Kubernetes Secrets, and treat DAG updates like application releases: reviewed, tested, and easy to roll back when a bad DAG breaks scheduling.

9. Should I use the KubernetesExecutor or the CeleryExecutor on EKS?

The CeleryExecutor, the Helm chart's default, keeps a pool of long-running worker pods that pull tasks from a Redis queue. Tasks start fast, which suits many short tasks. The KubernetesExecutor starts a pod per task: better isolation, per-task resources and images, and no idle workers, at the cost of pod startup time. Since Airflow 2.10 you can also configure multiple executors and choose one per DAG or task, for example Celery for small tasks and Kubernetes for heavy ones.

10. Should I use Amazon MWAA or self-host Airflow on EKS?

MWAA runs the scheduler, workers, web server, and metadata database for you and supports Airflow 3.x, so it removes most of the operational work. Self-hosting on EKS costs engineering time, but gives you control over Airflow and provider versions, the executor, custom images, node types such as Graviton, and networking. Compare MWAA pricing for your environment size against the real node, RDS, and engineering cost before deciding.

11. How should I manage Airflow connections and secrets on EKS?

Keep credentials out of DAG code and out of plain values.yaml. Store connections and variables in AWS Secrets Manager and let Airflow's Secrets Manager backend look them up at runtime, or sync them into Kubernetes Secrets with the External Secrets Operator and expose them as AIRFLOW_CONN_<CONN_ID> environment variables. Give the pods AWS access through EKS Pod Identity or IRSA instead of static access keys.

12. How do I scale Airflow workers on EKS?

With the CeleryExecutor, scale the number of worker pods (a fixed replica count, or the chart's KEDA autoscaling) and tune worker concurrency. With the KubernetesExecutor, each task is a pod, so scaling means making sure the cluster can add nodes quickly through Cluster Autoscaler or Karpenter. In both cases, use parallelism, max_active_tasks_per_dag, and pools to protect the metadata database and downstream systems.

13. How do I upgrade Airflow on EKS safely?

Back up the metadata database, pin the chart and Airflow versions, and upgrade staging before production. The chart runs the database migration as a job during helm upgrade, so watch that job first. For a move from Airflow 2 to Airflow 3, upgrade to the latest 2.x first, trim old metadata with airflow db clean, check DAGs with ruff check --select AIR301, and rename webserver values to apiServer, as described in the official Airflow 3 upgrade guide and the Helm chart upgrade notes.