Mid Level Data EngineeringAzureData FactoryDataBricksSQLAzure ClientProject: We are establishing Agile Data Warehouse in cloud and many new AI practices to enable personalization in various capabilities to improve employee experience.Responsibilities:Identify opportunities for Data Engineering and AI to enhance the core product platform, select the best machine learning techniques to the specific business problem and then build the models that solve the problem. Architect and design DW and Analytics solutions and cloud services Own the end-end process, from recognizing the problem to implementing the solution. Establish DataOps principles and best practices Design, develop, implement and maintain scalable ETL processes on Azure Cloud using Azure Synapse implement systems and processes to monitor data quality Ad hoc analyses troubleshooting data-related issues Develop and maintain propriety data warehouse tools Offer technical guidance to internal teams and business partners Work with offshore team, business analysts and other data engineering team members to ensure alignment of requirements, methodologies and best practices Participate in scrum process and deliver stories/features according to the schedule Participate in product support activities as needed by the team Individual contributor and Technical lead for Data Engineering - Data pipelines, Data modelling and Data warehouseTop Requirements:a. AI/ Client Solution designb. Strong problem solving and troubleshooting skills with the ability to exercise mature judgment.c. MLOpsRequirements: Undergraduate degree or equivalent experience. BE/BTech or equivalent experience BE/BTech or equivalent experience in Data engineering Advanced SQL knowledge, capable to write optimized queries for faster data workflows. Proven work experience in Python, SQL, Any RDBMS.
Experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS Must be extremely well versed with handling large volume data and work using different tools to derive the required solution. To be able to work in a fast-paced agile development environment. Preferred: 5+ years of hands-on software engineering experience. Strong knowledge of Azure Data Lake, Synapse, ADF, Power BI. Strong knowledge of Data Modelling techniques like Data Vault 2.0 Understanding of CI/CD framework is an added advantage.
Experience with Git / Azure DevOps Ability to interpret/write custom shell scripts. Python scripting is a plus.
Experience with AzureML Expert in Azure Synapse, Azure Container Registry, Azure App services