Deploy Apache Airflow on AWS Elastic Kubernetes Service (EKS)

Airflow becomes a platform concern as soon as more than one team relies on it for production work. Morning schedules stack up, month-end jobs create worker-pod pressure, retries amplify load against warehouses and third-party APIs, and a high-priority DAG often lands during the maintenance window that was supposed to be quiet.
Apache Airflow remains a pragmatic orchestration choice for batch and event-adjacent workflows because it gives teams mature scheduling, dependency modeling, retries, backfills, pools, task-level visibility, and a deep provider ecosystem. The hard part is not getting a demo DAG to turn green. The hard part is keeping schedulers responsive, protecting the metadata database, preserving logs after short-lived pods exit, scoping secrets and AWS permissions correctly, and scaling workers when data engineering, analytics, ML, finance, and platform teams all share the same control plane.
Deploying Apache Airflow on AWS Elastic Kubernetes Service (EKS) is a strong pattern when Airflow needs to behave like cloud-native infrastructure instead of a carefully maintained VM. EKS lets you run task workloads as Kubernetes pods, isolate environments with namespaces and node groups, scale around bursty execution, and integrate with AWS identity, networking, storage, logging, and observability. It also makes the tradeoffs harder to ignore: resource requests must reflect real DAG behavior, IAM boundaries need to be explicit, Python dependencies should be baked into versioned images, and noisy workflows need pools, queues, priority weights, and concurrency limits before they starve the rest of the platform.
This guide walks through preparing an EKS environment, deploying Airflow with Helm, and wiring in the production-adjacent pieces teams usually need soon after the first successful install. The emphasis is day-two operation: executor choice, node sizing, workload identity, metadata database reliability, remote log storage, upgrade safety, and the checks to run when DAGs stop scheduling, worker pods fail to start, or Monday morning traffic looks nothing like the installation test.

‍

Validate EKS prerequisites and production operating assumptions before installing Airflow

Before installing Apache Airflow, make sure the operator workstation or deployment pipeline and the AWS landing zone are actually ready. You need AWS CLI v2, kubectl, Helm, and eksctl if eksctl is the tool you will use to create the cluster. You also need permissions to create or administer the EKS control plane, managed node groups, IAM roles, security groups, and the VPC components the cluster depends on. In real platform environments, the first blockers are rarely hidden in Airflow Helm values. They are usually account boundaries, subnet routing, DNS, egress policy, private endpoint access, TLS handling, image registry access, or security group rules that only become visible after pods begin failing.
If your team uses AWS IAM Identity Center, named CLI profiles, separate workload accounts, private EKS clusters, or standard network blueprints, validate these items before installing anything:

• The target AWS account, region, cluster name, VPC, subnet strategy, and outbound access path for Airflow system pods and task pods.
• The Kubernetes version, managed add-ons, ingress or load balancer controller requirements, certificate path, DNS model, and any admission policies enforced by the platform team.
• The workload IAM model for Airflow, such as IAM Roles for Service Accounts (IRSA) or EKS Pod Identity, instead of broad node-level permissions.
• The exposure model for the Airflow webserver, such as an internal load balancer, ingress, VPN-only access, zero-trust proxy, or another controlled access path.
• Where the metadata database, remote logs, DAG code, secrets, and container images will live, and which team owns each dependency.
• The approved source for Airflow images and DAG-related dependencies, whether that is Amazon ECR, another private registry, or an allowed external registry.
• The backup, retention, upgrade, and disaster recovery expectations for the metadata database, remote logs, DAG storage, and any persistent volumes.
• The expected task profile: CPU-heavy transforms, memory-heavy Python jobs, long-running sensors, KubernetesPodOperator workloads, deferrable operators, or short high-volume tasks that stress the scheduler and metadata database in different ways.

This walkthrough uses eksctl because it is a fast, repeatable way to create an EKS cluster for a hands-on Airflow deployment. In a production platform, the cluster is often owned by Terraform, CloudFormation, Crossplane, or an internal provisioning workflow. If that is your environment, use the approved path and treat the eksctl section as a reference shape, not a competing source of truth. A tutorial cluster created outside the normal control plane is a common source of drift: duplicate IAM roles, unmanaged node groups, missing tags, permissive security groups, public endpoints no one approved, and cleanup work that never lands in a backlog.
Before installing Airflow, verify the cluster from the operator’s point of view. kubectl and Helm should target the intended cluster and namespace. AWS permissions should be attached to Kubernetes service accounts through IRSA or EKS Pod Identity instead of inherited broadly from node roles. Node groups need enough steady-state capacity for the webserver, scheduler, triggerer, DAG processor, metadata database connectivity, and platform agents, plus headroom for task bursts. If you rely on Cluster Autoscaler, Karpenter, or another scaling mechanism, confirm it can add the right instance types quickly enough for worker pods with realistic CPU, memory, ephemeral storage, availability zone, and CPU architecture requirements. Check taints, tolerations, labels, and topology rules early. Many Airflow worker failures are really unschedulable pod failures caused by placement constraints that were never tested with production-sized tasks.
Also confirm the cluster services Airflow depends on: healthy CoreDNS, working metrics collection, image pull access, a default storage class if persistent volumes are required, network access to the metadata database, and an approved route to the web UI, preferably internal or otherwise restricted. If the deployment uses remote logging, validate the bucket, encryption, lifecycle policy, and write permissions before the first production DAG runs. A successful Helm release is not the finish line. The useful acceptance test is operational: failed tasks leave readable remote logs, a burst of DAG runs does not starve the cluster, worker pods start without image, networking, scheduling, or IAM errors, and a Kubernetes permission issue is isolated to the correct workload identity instead of hidden by an overly permissive node role. If those checks fail, pause before onboarding more DAGs; otherwise, the first busy business cycle becomes your load test.

Set up AWS identity, eksctl, and kubectl with clear workload boundaries

Install or update AWS CLI v2, then verify that it is authenticated against the exact AWS account, region, and profile where the EKS cluster will run. If your organization uses IAM Identity Center, SSO, role assumption, or named profiles, sign in first and run a caller-identity check before creating or changing resources. The provisioning identity needs enough access to manage the EKS control plane, node groups, IAM roles, security groups, and any required VPC resources. If it does not, the failure may not show up cleanly at creation time; it can reappear later as a confusing Kubernetes scheduling issue, image pull error, or failed Helm release.
Keep three identities distinct in the runbook: the human or pipeline identity that provisions infrastructure, the Kubernetes identity that installs Airflow resources, and the runtime identity that Airflow pods use when they call AWS APIs. That separation makes S3 remote logs, Secrets Manager reads, ECR pulls, metadata database access, and DAG-level AWS permissions diagnosable. A scheduler pod that can parse DAG files or write logs should not automatically mean every task pod can read every secret in the account. Map permissions to service accounts and workloads deliberately, then document which Airflow components are allowed to use each role.
For Airflow on EKS, treat workload identity as part of the application design. The webserver, scheduler, workers, triggerer, DAG processor, and task pods rarely need identical AWS access. A platform team might allow the scheduler to write remote logs, let selected DAG tasks assume narrowly scoped roles, and keep the webserver away from data-plane permissions. That takes more work than attaching broad access to the node role, but it turns incidents into bounded failures instead of account-wide exposure. It also makes reviews cleaner: when a DAG needs a new AWS permission, you evaluate the task role or service account that needs it instead of expanding a shared node role used by unrelated workloads.
Before applying manifests or running Helm, confirm that kubeconfig, kubectl, Helm, and the namespace context all point to the intended cluster. Deploying Airflow into a shared staging cluster, an old sandbox, or the wrong workload account is not a harmless setup mistake; it is a platform incident waiting to happen. Put the account, region, cluster name, and namespace in the terminal prompt, CI output, release ticket, or deployment checklist so the operator has to see the target before making changes. Where possible, make the pipeline fail closed when the expected account, cluster, or namespace does not match. Guardrails are cheaper than cleaning up a release installed in the wrong environment.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install

Please refer to the full AWS installation guide for other operating systems and architectures.

Once Installed, we have to configure the AWS cli on the local machine. Refer to this AWS guide about configuring the CLI locally.

2. Install the eksctl CLI:  (skip to step 3 if you already have eksctl installed)

curl --location "https://github.com/weaveworks/eksctl/releases/download/0.104.0/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin

You can also refer to the eksctl installation guide.

Create the AWS EKS (Elastic Kubernetes Service) Cluster

Create an EKS cluster, or skip this step if you already have a cluster that meets the networking, IAM, and capacity requirements for Airflow.

You can create an EKS Cluster directly from the AWS management console or 

Use eksctl to create the cluster from the command line so the control plane, worker nodes, and supporting AWS resources are created consistently.

Run the following command to create an EKS cluster in a public subnet in the Oregon region. For a production environment, review the networking, private endpoint, and node group choices against your organization’s baseline before reusing this shape.

eksctl create cluster --name airflow-cluster --region us-west-2 --nodegroup-name standard-workers --node-type t3.medium --nodes 3 --nodes-min 1 --nodes-max 4 --managed

You can find a detailed blog on setting up an EKS Cluster.

Connect to the EKS Cluster from your local machine

1. Install kubectl in your local machine using

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

kubectl version

Please refer to the AWS kubectl & eksctl configuration guide for other operating systems and architectures.

2. After the cluster is created, configure local access by updating your kubeconfig file. This makes kubectl and Helm point at the new EKS cluster, so verify the active context before installing Airflow.

aws eks --region us-west-2 update-kubeconfig --name airflow-cluster

‍Setup Helm Locally

Run the below command to install Helm on your local machine.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Please refer to the Installing Helm guide for other operating systems and architectures.

Support Dynamic Volume Provisioning for Persistent Storage using EBS

For an elastic scalable service, dynamic volume provisioning is preferred. Persistent storage must be configured and registered. 

• If the cluster does not already support dynamic persistent volumes, configure the Amazon EBS CSI Driver add-on and dynamic volume provisioning before installing Airflow, or confirm that your platform team provides an equivalent default storage class.

‍

Some background about Airflow

What is Airflow?

Apache Airflow is an open-source platform for authoring, scheduling, and operating workflows. Teams often start with ETL or ELT pipelines, then extend the same pattern to ML jobs, reporting runs, infrastructure automation, and cross-system maintenance tasks. Airflow lets engineers create, schedule, and monitor workflows across multiple services, with notifications through Slack, email, or another integration when a run succeeds, fails, or needs attention. Each workflow is written in Python as a Directed Acyclic Graph (DAG), which defines the tasks, dependencies, and allowed execution order. After deployment, the web UI becomes the operating console: engineers inspect DAG runs, retry failed tasks, read logs, pause risky pipelines, and determine whether a workflow is blocked by code, infrastructure, data quality, or an upstream dependency.

Use cases of Airflow:

• Data ETL Automation: Streamline the extraction, transformation, and loading of data from various sources into storage systems.
• Data processing: Coordinate batch jobs for cleansing, aggregation, enrichment, validation, and loading data across warehouses, object storage, APIs, and downstream analytics systems.
• Data Migration: Manage data transfer between different systems or cloud platforms.
• Model Training: Automate the training of machine learning models on large datasets.
• Reporting: Generate and distribute reports and analytics dashboards automatically.
• Workflow Automation: Coordinate complex processes with multiple dependencies.
• IoT Data: Analyze and process data from IoT devices.
• Workflow Monitoring: Track workflow progress and receive alerts for issues.

Benefits of using Airflow in Kubernetes

Deploying Apache Airflow on a Kubernetes cluster offers several advantages over deploying it on a virtual machine:

• Scalability: Kubernetes allows you to scale your Airflow deployment horizontally by adding more pods to handle increased workloads automatically.
• Isolation: Enables running different tasks of the same pipeline on various cluster nodes by deploying each task as an isolated pod.
• Automation: Kubernetes-native features such as autoscaling, self-healing, pod scheduling, and rolling updates reduce manual intervention. For Airflow, that means workers can scale with workload demand, failed pods can be replaced automatically, and upgrades can be handled with less disruption when the deployment is configured carefully.
• Portability: Deploying on Kubernetes makes your Airflow setup more portable across different environments, whether on-premise or cloud.
• Integration: Kubernetes integrates seamlessly with various tools for monitoring, logging, and security, enhancing the overall management of your Airflow deployment.

‍

Airflow Architecture Diagram

1. The airflow components are the Executor, Scheduler, Web Server, and Airflow database. The Airflow worker and Triggerer are also involved.
2. As the diagram shows, the data engineer writes Airflow DAGs as Python files. Each DAG defines a workflow: the tasks to run, the dependencies between them, and the order in which Airflow should execute them. In a Kubernetes deployment, those DAG files are usually delivered through a Git-based sync process, a mounted volume, or a custom image, so treat them like application code: version them, review them, and promote them through environments deliberately.
3. The Scheduler picks up these DAGs and has the config to run the tasks specified in the DAGs.
4. In the above diagram, the Scheduler runs tasks using Kubernetes Executor and creates a separate pod for every task, which provides isolation. 
5. Airflow also stores pipeline metadata in an external database. The main configuration file used by the Web server, Scheduler, and workers is airflow.cfg. 
6. The Data Engineer can view the entire flow through the Airflow UI. Users can also check the logs, monitor the pipelines, and set alerts.

‍

Airflow Deployment Options

When deploying Apache Airflow, there are multiple approaches to consider, each with unique advantages and challenges. Let us see the different deployment examples:

• Amazon Managed Workflows for Apache Airflow (MWAA)

You should configure the service through the AWS Management Console. There, you can define your environment, set up necessary permissions, and integrate with other AWS services.

• Google Cloud Composer:

For Google Cloud Composer, create the environment from the Google Cloud Console or your infrastructure-as-code workflow, then connect it to services such as BigQuery and Google Cloud Storage. The managed service removes much of the Airflow control-plane burden, but you still need to manage DAG quality, permissions, networking, and environment sizing.

• Azure Data Factory with Airflow Integration:

Try to Configure Airflow through the Azure Portal. Integrate with other Azure services for efficient workflow automation.

• Self-hosted on AWS EC2:

We can launch and configure EC2 instances. We must install Airflow, set up the environment, configure databases, and set up the scheduler.

• Running on Kubernetes (e.g., AWS EKS):

We can create Kubernetes clusters, deploy Airflow using Helm charts or custom YAML files, and manage container orchestration and scaling.

These are the different options or ways to deploy Airflow, but we are focusing on Amazon Web Service EKS to deploy Airflow, so let us see this in the below section.

‍

Deploy Airflow on AWS EKS

Let us install Apache Airflow in the EKS cluster using the helm chart.

1. Create a new namespace.

kubectl create namespace airflow

2. Add the Helm chart repository.

helm repo add apache-airflow https://airflow.apache.org

3. Update your Helm repository.

helm repo update

4. Deploy Airflow using the remote Helm Chart

helm install airflow apache-airflow/airflow --namespace airflow   --debug

The command output can include Airflow webserver details, generated passwords, connection strings, Helm notes, or default database credentials. Treat that output as sensitive operational material. Move any values you need into the approved secret store or password manager, rotate throwaway defaults before the environment is shared, and avoid pasting raw credentials into tickets, chat threads, screenshots, CI logs, or long-lived runbooks.

5. Examine the deployments by getting the Pods

Kubectl get pods -n airflow

The Airflow instance is set up in EKS. All the airflow pods should be running.

Let’s prepare Airflow to run our first DAG

At this point, Airflow is deployed using the default configuration. Let's see how we can get the default values from the helm chart on our local machine, modify it, and update a new release.

1. Save the configuration values from the helm chart by running the below command.

helm show values apache-airflow/airflow > values.yaml

This command generates a file named

values.yaml

in your current directory, which you can modify and save as needed.

2. Check the release version of the helm chart by running the following command.

helm ls -n airflow

3. Add ingress only after deciding how the Airflow UI should be reached. 
For a production environment, avoid exposing Airflow directly to the public internet unless you have an explicit security requirement and controls such as SSO, TLS, IP allow lists, and audit logging. The example below installs the NGINX ingress controller from its Helm repository so the cluster can route HTTP traffic to the Airflow webserver. Use an internal load balancer or private ingress pattern if Airflow is meant for operators on a corporate network or VPN. 

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm install nginx-ingress ingress-nginx/ingress-nginx --namespace airflow-ingress --create-namespace --set controller.replicaCount=2kubectl get pods -n airflow-ingress

Note - All the pods should be running.

kubectl get service nginx-ingress-controller --namespace airflow-ingress

Check the service or ingress output for the address AWS assigned to the Airflow webserver. A new load balancer can remain pending for several minutes, so rerun the check before changing chart values or reinstalling the release. If it stays pending, inspect the ingress or load balancer controller service, subnet tags, security groups, controller logs, and Kubernetes events. At that stage, the issue is usually EKS integration or AWS networking rather than Airflow itself.

After installing the ingress controller, add the required configuration in the values.yaml file and save the file. There is a section dedicated to the ingress configuration.

# Ingress configuration
ingress:
  enabled: true
  web:
    enabled: true
    annotations: {}
    path: "/"
    pathType: "ImplementationSpecific"
    host: 
    ingressClassName: "nginx"

After the changes to the values in the values.yaml file, we run the helm upgrade command to deploy the changes and create a new release version.

By default, the Helm Chart deploys its own Postgres instance, but using a managed Postgres instance is recommended instead.

You can modify the Helm Chart’s values.yaml file to add  configuration of the managed database and volumes

metadataConnection:
             user: postgres
             pass: postgres
             protocol: postgresql
             host: 
             port: 5432
             db: postgres
             sslmode: disable

Run the helm upgrade command to implement the changes done above.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

Check the release version after the above command is run successfully. You should observe that the revision has changed to 2.

‍

Accessing Airflow UI

We will use port-forwarding to access the Airflow UI in this tutorial. Run the below command and access “localhost:8080” on the browser.

helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

Use the default webserver credentials saved in the above section, “Installing Airflow Helm chart.”

At this point, Airflow is set up and is accessible. Hurray 😀

You can also access the UI over your domain, which is added in the ingress configuration in the above section.

‍

Create your first Airflow DAG (in Git)

No DAGs have been added to our Airflow deployment yet. Let us see how we can add them.

To Set up a private GitHub repository for DAG, you can create a new one using the Github website's UI.

You can also install the Git command-line interface on your local machine and initialize an empty repository there. This is useful when you want a simple DAG-sync workflow, but keep the repository structure intentional: separate DAG code from local experiments, avoid committing credentials, and use branches or pull requests if multiple engineers will update workflows.

git init

Adding DAG configs to the git repo

Once the git repo is initialized, create a DAG file like “sample_dag.py” and push it to the remote branch.

git add .
git commit -m 'Adding first DAG'
git remote add origin
git push -u origin main

Integrate Airflow with a private Git repo

To integrate Airflow with a private Git repository, you will need credentials, i.e. username /password or an SSH key.

We will use the SSH key to connect to the git repo. Skip the first step below if the SSH Key already exists in your Github account.

1. [Skip if it already exists] Generate an SSH key in your local machine and add it to the GitHub account (If not already present). 

ssh-keygen -t ed25519 -C ""

2. Create a generic secret in the same namespace where Airflow is deployed. This secret holds the SSH key used by Airflow or the DAG-sync mechanism to read from the repository, so keep it namespace-scoped, restrict repository permissions to the minimum required access, and avoid reusing a personal developer key with broader privileges.

kubectl create secret generic airflow-ssh-git-secret --from-file=gitSshKey= -n airflow

3. Update the Git settings in the values.yaml file, then run the same Helm upgrade command shown in the previous section to apply the change.

gitSync:
    enabled: true
    repo: 
    branch: 
    rev: HEAD
    depth: 1
    maxFailures: 0
    subPath: ""
sshKeySecret: airflow-ssh-git-secret

Below is a “sample_dag.py” that demonstrates a simple workflow. 

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2024, 8, 8),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}
dag = DAG('hello_world', default_args=default_args, schedule_interval=timedelta(days=1))
t1 = BashOperator(
    task_id='say_hello',
    bash_command='echo "Hello World from Airflow!"',
    dag=dag,
)

Upon completion, you can see the DAGs in the UI interface. Airflow automatically detects new DAGs, but you can manually refresh the DAGs list in the Airflow UI by clicking the "Refresh" button on the DAGs page.

The UI has many options/settings to experiment with, such as code, graphs, audit logs, etc.

You can also check the EKS cluster’s activity and DAG dashboard from the Activity tab.

Run the Airflow job

DAGs can be scheduled to run or triggered manually from the UI interface. There is a run button on the rightmost side of the DAG table.

Also, it can be triggered from within the DAG.

‍

Make your Airflow on Kubernetes Production-Grade

Apache Airflow is a powerful tool for orchestrating workflows, but making it production-ready requires careful attention to several key areas. Below, we explore strategies to enhance security, performance, monitoring, and ensure high availability in your Airflow deployment.

‍

1. Improved Security

a. Role-Based Access Control (RBAC)

• Implementation: Enable RBAC in Airflow to ensure only authorized users can access specific features and data.
• Benefits: Limits access to critical areas and reduces the risk of unauthorized changes or data breaches.

Please refer to Access Control guide.

b. Secrets Management

• Implementation: Integrate with external secret management tools like AWS Secrets Manager, HashiCorp Vault, or Kubernetes secrets.
• Benefits: Securely store sensitive information like API keys and database passwords, keeping them out of your codebase.

Refer to this AWS document for Secrets management in EKS

Guide to use Kubernetes secrets in Airflow DAG

c. Network Security

• Implementation: Use network policies and security groups to restrict Airflow's web interface and API access.
• Benefits: Minimizes exposure to potential attacks by limiting network access to trusted sources only.

Refer to this guide to implement Network Security in EKS.

‍

2. Improved Performance

a. Right-size Airflow components and worker pods

• Implementation: Right-size your Kubernetes pods and nodes based on the workload demand. Use Kubernetes Horizontal Pod Autoscaler (HPA) to scale Airflow resources dynamically and cluster autoscaler to scale nodes.
• Benefits: Ensures efficient use of resources, reduces costs, and prevents bottlenecks during peak loads.

Airflow uses Executors for Autoscaling pods. 

Refer to these guides for implementing HPA on EKS and using the Cluster Autoscaler in EKS.
Use HPA carefully with Airflow components. It can help scale stateless or worker workloads when the metrics match the bottleneck, but scaling the scheduler or webserver blindly can create more noise than capacity. The Cluster Autoscaler solves a different problem: it adds or removes nodes when pending pods cannot be scheduled. In practice, tune both together so worker pods can scale out during DAG bursts without leaving the cluster permanently overprovisioned.

b. Task Parallelism

• Implementation: Configure Airflow to handle parallel task execution by optimizing the number of worker pods and setting appropriate concurrency limits.
• Benefits: Accelerates workflow execution by running multiple tasks simultaneously, improving overall performance.

Check out this guide for Implementing parallelism in Airflow.

c. Use of ARM Instances

• Implementation: Consider running workloads on ARM-based instances like AWS Graviton for cost efficiency.
• Benefits: ARM instances often provide a better cost-to-performance ratio, especially for compute-intensive tasks.

A quick guide to Creating an EKS cluster with ARM instances.

d. Use of HTTPS for ingress host

• Implementation: Consider having HTTPS for the Airflow URL using TLS/SSL certificates with the Ingress controller in Kubernetes.
• Benefits: HTTPS encrypts data to enhance the security of information being transferred. This is especially crucial when handling sensitive data, as encryption helps protect it from unauthorized access during transmission.

Refer to this guide to Install NGINX ingress and configure TLS.

3. Monitoring

a. Collect actionable metrics and alerts

• Implementation: Expose Airflow metrics to Prometheus so the platform team can watch scheduler lag, task duration, worker saturation, queue depth, pod restarts, and database connectivity. Use Grafana dashboards for day-to-day visibility, and configure Prometheus Alertmanager to page on symptoms that actually break pipelines, such as a stuck scheduler, exhausted workers, repeated task failures, or missing DAG heartbeats.
• Benefits: It provides visibility into Airflow’s performance, allowing you to identify and address issues proactively and enabling quick response to potential problems, reducing downtime and maintaining workflow reliability.

Refer to the “How to set up Prometheus and Grafana with Airflow” guide.

b. Log Collection

• Implementation: Set up centralized logging with tools like Elasticsearch, Logstash, Kibana (ELK stack or EFK stack), or Grafana Loki.
• Benefits: Simplifies troubleshooting by consolidating logs from all Airflow components into a single, searchable interface.

Refer to this guide on how to Setup Elastic, Fluentd, and Kibana on EKS.

‍

4. High Availability

a. Redundant Components

• Implementation: Deploy multiple replicas of Airflow’s web server, scheduler, and worker nodes to ensure redundancy.
• Benefits: Increases resilience by preventing single points of failure, ensuring that workflows continue even if one component goes down.

Use the Helm chart to run the Airflow components as separate Kubernetes pods, then verify that each component is scheduled, healthy, and using the expected service account:

1. Set Replicas for the Scheduler:

In your values.yaml file set the scheduler.replicas to the desired number of replicas. For example:

scheduler:
  replicas: 2

2. Set Replicas for the Web Server:

Similarly, set the web.replicas to deploy multiple web server pods:

web:
  replicas: 2

3. Deploy the Helm Chart:

Apply the Helm chart with the updated values.yaml file:

helm upgrade --install airflow apache-airflow/airflow -f values.yaml

This configuration ensures that multiple scheduler and web server pods are deployed, contributing to the high availability of your Airflow setup.

Airflow helm chart’s value.yaml file can be found here.

b. Database High Availability

• Implementation: Use a highly available database solution like Amazon RDS with Multi-AZ deployment for Airflow’s metadata database.
• Benefits: Ensures continuous operation and data integrity even during a database failure.

Refer to Amazon RDS with the Multi-AZ deployment guide.

c. Backup and Disaster Recovery

• Implementation: Regularly backup Airflow’s database and configuration files. Implement a disaster recovery plan that includes rapid failover procedures.
• Benefits: Protects against data loss and enables quick recovery in case of catastrophic failures.

Read this document to set up automated backups in Amazon RDS.

Refer to this AWS page to learn about “Backup and Restore of EKS.”

Conclusion

Setting up Apache Airflow on Amazon EKS is a powerful way to manage your workflows at scale, but it requires careful planning and configuration to ensure it’s production-ready. Following this guide, you've deployed Airflow on EKS, created a simple DAG, connected Airflow with a private Git repository, and learned about different ways to implement security, performance, high availability, monitoring, and logging. With these optimizations, your Airflow deployment is now more efficient, cost-effective, and ready to handle the demands of real-world data orchestration.

Frequently Asked Questions

1. What is Apache Airflow?

• Apache Airflow is an open-source tool that helps in orchestrating and managing workflows through Directed Acyclic Graphs (DAGs). It automates complex processes like ETL (Extract, Transform, Load) jobs, machine learning pipelines, and more.

2. Why deploy Airflow on Amazon EKS?

• Deploying Airflow on Amazon EKS offers scalability, flexibility, and robust workflow management. EKS simplifies Kubernetes management, allowing you to focus on scaling and securing your Airflow environment.

3. What are the prerequisites for deploying Airflow on EKS?

• You need an AWS account, an EKS cluster, kubectl configured on your local environment, a dynamic storage class using EBS volumes, and Helm for package management.

4. How do I monitor Airflow on EKS?

• You can integrate Prometheus and Grafana for monitoring. Using Loki for log aggregation can also help in centralized log management and troubleshooting.

5. What Kubernetes add-ons are recommended for a production-grade Airflow setup?

• Essential add-ons include External Secret Operator for secure secrets management, Prometheus and Grafana for monitoring, and possibly Loki for logging.

6. Can Airflow be integrated with external databases like RDS?

• Yes, it’s common to configure Airflow to use an external PostgreSQL database hosted on Amazon RDS for production environments, providing reliability and scalability for your metadata storage.

7. How can I access the Airflow UI on EKS?

• You can access the Airflow UI by setting up a LoadBalancer service or using an Ingress Controller with a DNS pointing to your load balancer for easy access.

8. How do I manage DAGs in a production environment?

• For production, keep DAGs in a private Git repository and let the Airflow deployment pull them with the Helm chart’s `gitSync` sidecar instead of baking DAGs into the image for every change. Pin the sync target to a branch, tag, or commit strategy that matches your release process, store credentials in Kubernetes Secrets, and treat DAG updates like application releases: reviewed, tested, and easy to roll back when a bad DAG breaks scheduling.