Apache Airflow: an open-source platform designed for orchestrating complex workflows. Airflow’s Directed Acyclic Graphs (DAGs) provide a visual representation of pipeline tasks and their dependencies, allowing for easy management and monitoring.

Kubernetes: the container orchestration giant adds scalability, resilience, and portability to your applications. When combined with Airflow, Kubernetes becomes the playground for executing and scaling tasks efficiently and reliably.

In this blog post, we will cover how to deploy Airflow using Kubernetes.

Prerequisites

Before you start, make sure that the following tools are installed on your local system.

1. Create airflow namespace

To begin, let’s create a dedicated namespace for Airflow within the Kubernetes cluster.

kubectl create ns airflow

2. Add Airflow Helm repository

Now that the Airflow namespace is in place, we need to add the Airflow Helm repository. Use the following commands:

helm repo add apache-airflow https://airflow.apache.org
helm repo update
helm search repo airflow

To confirm that the Airflow repository has been added, run:

helm repo list

The result should include the name of the Airflow repository: 

Deploy Apache Airflow with Kubernetes

3. Verify the Deployment

To ensure that Airflow has been deployed successfully, you can check the status of the pods.

kubectl get pods -n airflow

You should get results similar to the following.

Deploy Apache Airflow with Kubernetes

4. Port Forwarding for Airflow

Once Airflow is deployed, a service named airflow-webserver is also deployed. To get the name of the service, run the command:

kubectl get svc -n airflow

The result should resemble the screen below:

Deploy Apache Airflow with Kubernetes

Now, run the following command to forward the service’s port to your local machine.

kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow

5. Log in to Airflow

After setting up Apache Airflow, please wait for a moment.

Open a web browser and navigate to http://localhost:8080. You should see the Apache Airflow login page.

Log in using the following credentials:

  • Username: admin
  • Password: admin

Please note that you should not use these for production environments.

6. Cleanup

Once you’ve completed your tasks and want to uninstall the Airflow Helm Charts Release, you can use the following command:

helm uninstall airflow  --namespace airflow

You might also find it necessary to keep an eye on your Airflow installation. To achieve this, you can establish a monitoring platform by following the instructions in our guide Deploy Prometheus Operator in Kubernetes.

Similar Posts