Primary Skills: AWS Data Engineering (Expert), Data Mesh Architecture (Expert), Databricks & Modern Data Platforms (Expert), DevSecOps & Infrastructure as Code (Advanced), AI-Augmented Data Engineering (Advanced)
Contract Type: W2 Only Duration: 18+ Months Location: San Francisco, CA (Hybrid) Pay Range: $90 - $100 on W2
Job Summary
We are seeking a Lead Cloud Data Engineer - Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives. The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices. This role will drive the implementation of scalable data products, federated governance, and AI-augmented solutions while providing technical leadership across cross-functional engineering teams.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3.
- Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks.
- Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta.
- Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks.
- Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation.
- Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation.
- Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments.
- Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption.
- Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions.
- Create and maintain technical architecture documentation, implementation standards, and engineering best practices.
Must-have Skills
- 8+ years of experience in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms.
- Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions.
- Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance.
- Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta.
- Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks.
- Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices.
- Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks.
- Strong SQL, Python, and cloud-native data engineering expertise.
- Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments.
- Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills.
Nice-to-have Skills
- Experience supporting enterprise data warehouse modernization initiatives.
- Knowledge of hybrid cloud and on-premises data platform integration.
- Experience implementing self-service analytics and reusable data product frameworks.
- Familiarity with cost optimization strategies for large-scale cloud data platforms.
- Experience working within highly regulated industries such as Financial Services or Government.
Preferred Qualifications
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical field.
- AWS Certifications (Solutions Architect, Data Analytics, or Developer) preferred.
- Experience leading enterprise-scale cloud modernization and AI transformation initiatives.
- Strong background in enterprise architecture, technical leadership, and cross-functional collaboration.