Experience : 8–10 years overall, with 3+ years of Azure experience.
Required Skills
- Strong hands-on experience with Azure Data Factory (ADF) — pipelines, data flows, triggers, integration runtimes, parameterization, error handling, and retries.
- Strong experience with Azure Databricks — Notebooks, Jobs/Workflows, PySpark/Scala, Delta Lake, cluster configuration, and optimization.
- Advanced Python or Scala with strong coding standards, modularization, and unit testing (pytest).
- Expert SQL — CTEs, window functions, query tuning, and large datasets.
- Experience with ADLS Gen2, Delta Lake, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
- Experience with Git, branching strategies, and CI/CD for ADF/Databricks.
- Exposure to Infrastructure as Code (IaC) is a plus.
Roles and Responsibilities
- Coding & Development: Design, develop, and optimize ADF/Databricks data pipelines and solutions.
- UI Testing: Perform functional/UI testing, data validation, and troubleshooting.
- Stakeholder Management: Collaborate with stakeholders, understand requirements, and communicate project status/issues effectively.