Lead Data Engineer with Netezza to Databricks

Good day,

Role: Lead Data Engineer with Netezza to Databricks

Location: Cincinnati, OH (Onsite)

Mode: C2C

Contract

We are seeking an experienced Lead Data Engineer to lead a large-scale enterprise data modernization initiative involving the migration of legacy data warehouse workloads from IBM Netezza to Databricks Lakehouse Platform. This individual will be responsible for driving technical delivery, overseeing end-to-end migration execution, establishing engineering best practices, and coordinating activities across onshore and offshore delivery teams.

The ideal candidate combines deep expertise in data engineering, cloud modernization, ETL migration, and Databricks architecture with proven experience leading distributed teams and managing stakeholder communications across business and technology organizations.

Key Responsibilities

Migration Leadership

• Lead the end-to-end migration of data assets, ETL pipelines, reporting workloads, and analytical processes from Netezza to Databricks.
• Drive modernization opportunities beyond lift-and-shift migrations, including optimization for cloud-native architectures.
• Establish migration standards, coding guidelines, testing approaches, and deployment practices.
• Define and track migration KPIs, risks, dependencies, and delivery milestones.

Databricks Architecture & Engineering

• Design scalable and high-performance data solutions leveraging Databricks, Spark, Delta Lake, and cloud-native services.
• Develop and review ETL/ELT pipelines, data ingestion frameworks, and transformation logic.
• Optimize data models, partitioning strategies, query performance, and storage patterns.
• Implement data quality, observability, security, and governance controls.
• Support CI/CD, Infrastructure as Code, and automated deployment methodologies.

Team Leadership & Coordination

• Act as the technical lead for a blended onshore/offshore delivery model.
• Coordinate daily activities across engineering teams, ensuring alignment on priorities, deliverables, and timelines.
• Conduct design reviews, code reviews, sprint planning, and technical workshops.
• Mentor junior and senior engineers while promoting engineering best practices.
• Resolve technical blockers and facilitate cross-functional collaboration.

Stakeholder Management

• Partner with business stakeholders, data consumers, architects, and program leadership.
• Translate business requirements into technical solutions and migration priorities.
• Provide regular status reports, risk assessments, and executive updates.
• Drive issue resolution and decision-making across multiple workstreams.

Governance & Quality Assurance

• Ensure adherence to enterprise architecture, security, compliance, and data governance standards.
• Develop migration testing strategies, including reconciliation, validation, and performance testing.
• Establish operational support processes and transition plans for production environments.

Required Qualifications

Technical Skills

• 8+ years of data engineering experience with at least 2+ years in a lead or technical leadership role.
• Strong expertise in Databricks, Apache Spark, and large-scale data processing.
• Experience with SQL optimization, data modeling, and performance tuning.
• Proficiency in Python, SQL, and ETL/ELT development frameworks.
• Experience with Azure
• Knowledge of Delta Lake, Unity Catalog, workflow orchestration, and data governance concepts.
• Experience implementing data quality and reconciliation frameworks.

Leadership & Delivery

• Experience managing globally distributed onshore/offshore teams.
• Strong project coordination and stakeholder management skills.
• Ability to manage multiple workstreams while maintaining delivery quality and timelines.
• Excellent communication, facilitation, and problem-solving skills.

Thanks & Regards,

Ganesh B

Sr. US IT Recruiter

Phone: +1 4705321168

ganesh@datumtg.com

12000 Findley Road, Suite 350, Johns Creek, GA 30097