Our client, the nation’s largest provider-driven healthcare performance improvement company, is looking for a Sr. Azure Data Engineer with solid, real-time hands-on experience in:
- PySpark
- Databricks (notebooks, jobs, workflows, External/Managed tables, SQL Warehouse, AI/ML)
- ADLS (Blob Storage)
Location: Downtown Chicago/60606 (3 days onsite Tues, Wed, Thurs)
What you will be doing:
Key Responsibilities
- Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing).
- Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
- Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling).
- Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations).
- Provision and maintain Azure components using Pulumi (Infrastructure as Code).
- Participate in code reviews, documentation, and operational support (triage + root cause analysis).
Must-have Skills (Required)
- 5+ years' experience as a data engineer
- 3+ years doing ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.
- 3+ years in SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation.
- 2+ years using Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage).
- PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage.
- 1+ years using Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS.
- 1+ year using Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration.
- Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments.
Nice-to-have Skills
- R: Ability to support/translate validation rules with SQL scripts and create data quality reports.
- TypeScript: Useful for pulumi pipeline to create Azure components.
- Java: Useful for integration with existing services/components.
- .NET: Useful for integration with existing services/components.
- Angular / Spring Boot: Minor troubleshooting or coordination with app teams.