Data EngineerDesign and implement scalable data pipelines and architectures on Azure Databricks. Optimize ETL/ELT workflows, ensuring efficiency in data processing, storage, and retrieval. Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions.
Ensure best practices in data governance, security, and compliance within Azure environments. Troubleshoot and fine-tune Spark jobs for optimal performance and cost efficiency. Lead the migration of Azure SQL to Azure Databricks, ensuring a seamless transition of data workloads.
Design and implement scalable data pipelines to extract, transform, and load (ETL/ELT) data from Azure SQL into Databricks Delta Lake. Optimize Azure SQL queries and indexing strategies before migration to enhance performance in Databricks