Senior Data Engineer- Remote, India ( Immediate Joiner)

Role & responsibilities

• Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.
• Build and optimize data architectures and data models (Lakehouse / medallion, dimensional) for efficient data storage and retrieval.
• Develop ETL/ELT processes to transform and load data from various sources into data warehouses and data lakes.
• Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL and Delta Lake, orchestrated with Databricks Workflows.
• Integrate data from enterprise source systems including SAP (ABAP/CDS extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.
• Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB) and Azure DevOps (Git repositories, YAML build and release pipelines), promoting code across dev, QA and production.
• Implement data quality, validation, freshness and reconciliation checks with pipeline observability.
• Ensure data integrity, quality, and security across all data systems.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
• Document data processes, architectures, and data flow diagrams.

Preferred candidate profile

• 7 - 8 years of hands-on data engineering experience building and running production data pipelines at scale.
• Strong expertise in Azure and Azure data services (ADLS Gen2, Azure Databricks, Azure DevOps).
• Deep hands-on experience with Databricks: PySpark, Spark SQL, Delta Lake, Lakehouse / medallion architecture and Databricks Workflows.
• CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure DevOps (Git, YAML build/release pipelines, multi-environment promotion). (Must-have)
• Strong Snowflake experience (data modeling, performance tuning, loading and optimization).
• Proficiency in SQL and Python.
• Experience integrating data from SAP and other enterprise ERP / source systems into a data lake or warehouse.
• Solid data modeling (dimensional, star/snowflake, Lakehouse) and ETL/ELT design.
• Building data-quality, validation, reconciliation and pipeline monitoring / observability.