Senior Apache Airflow Platform Engineer

Roles & Responsibilities

• Architect, deploy, and manage production-grade Apache Airflow/Astronomer platforms on Kubernetes.

• Manage platform upgrades, high availability, scalability, capacity planning, security, and performance optimization.

• Automate infrastructure provisioning and platform lifecycle using Terraform, Helm, GitOps, and CI/CD pipelines.

• Develop reusable DAG templates, custom operators, shared libraries, testing frameworks, and self-service tools.

• Implement monitoring, logging, alerting, SLA tracking, incident management, and disaster recovery.

• Optimize workflow scheduling, task concurrency, retries, backfills, and resource utilization.

• Implement access controls, secrets management, audit logging, and security best practices.

• Collaborate with data engineering and platform teams, mentor engineers, and define platform standards and roadmaps.

Required Skills

• 6+ years of experience in data engineering, platform engineering, or infrastructure systems.

• 3+ years of hands-on Apache Airflow production experience at scale.

• Strong expertise in Astronomer, Airflow DAGs, operators, sensors, dynamic task mapping, scheduling, and workflow optimization.

• Strong Kubernetes, Helm, Terraform, GitOps (Argo CD/Flux), and CI/CD experience.

• Proficiency in Python and Bash scripting.

• Experience with Prometheus, Grafana, centralized logging, monitoring, and SRE practices.

• Knowledge of AWS, Azure, or GCP, including IAM, networking, and cloud storage.

• Experience with security, RBAC, SSO/OIDC, secrets management, incident response, and high-availability architecture.

Nice to Have

• Experience with Snowflake, BigQuery, Redshift, Databricks, Spark, or dbt.

• Familiarity with OpenLineage, Marquez, data quality frameworks, and data contracts.

• Experience with multi-region disaster recovery, platform migrations, and cost optimization.

• Contributions to Apache Airflow or development of internal developer platforms.