Responsibilities
- Lead the architecture, design, and implementation of advanced analytics solutions using Azure Databricks/Fabric. The ideal candidate will have a deep understanding of big data technologies, data engineering, and cloud computing, with a strong focus on Azure Databricks along with Strong SQL.
- Work closely with business stakeholders and other IT teams to understand requirements and deliver effective solutions.
- Oversee the end-to-end implementation of data solutions, ensuring alignment with business requirements and best practices.
- Lead the development of data pipelines and ETL processes using Azure Databricks, PySpark, and other relevant tools.
- Integrate Azure Databricks with other Azure services (e.g., Azure Data Lake, Azure Synapse, Azure Data Factory) and on-premise systems.
- Provide technical leadership and mentorship to the data engineering team, fostering a culture of continuous learning and improvement.
- Ensure proper documentation of architecture, processes, and data flows, while ensuring compliance with security and governance standards.
- Ensure best practices are followed in terms of code quality, data security, and scalability.
- Stay updated with the latest developments in Databricks and associated technologies to drive innovation.
Essential Skills
- Strong experience with Azure Databricks, including cluster management, notebook development, and Delta Lake.
- Proficiency in big data technologies (e.g., Hadoop, Spark) and data processing frameworks (e.g., PySpark).
- Deep understanding of Azure services like Azure Data Lake, Azure Synapse, and Azure Data Factory.
- Experience with ETL/ELT processes, data warehousing, and building data lakes.
- Strong SQL skills and familiarity with NoSQL databases.
- Experience with CI/CD pipelines and version control systems like Git.
- Knowledge of cloud security best practices.
Data Architect in plano at Unknown Company
This position is listed as full time and onsite.