Design, develop, and maintain scalable data pipelines using Azure services.
Build ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
Develop data transformation logic using PySpark, Spark SQL, and Python.
Manage Delta Lake and Lakehouse architectures.
Optimize SQL queries and Spark workloads for performance.
Integrate data from APIs, databases, and third-party applications.
Implement data quality, validation, and monitoring processes.
Collaborate with analysts, data scientists, and business stakeholders.
Ensure data security, governance, and compliance.
Responsibilities
Design, develop, and maintain scalable data pipelines using Azure services.
Build ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
Develop data transformation logic using PySpark, Spark SQL, and Python.
Manage Delta Lake and Lakehouse architectures.
Optimize SQL queries and Spark workloads for performance.
Integrate data from APIs, databases, and third-party applications.
Implement data quality, validation, and monitoring processes.
Collaborate with analysts, data scientists, and business stakeholders.
Implement CI/CD pipelines using Azure DevOps.
Ensure data security, governance, and compliance.
Required Skills
Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, PySpark, Python, SQL, Delta Lake, Git, Power BI, REST APIs