Lead Data EngineerLocation: 350 Mission Street, San Francisco, CA, USADuration: 12+ monthsOpen for C2C/W2Note: Candidates must be located in PST Time zone or MST Time zone. Candidates in CST or EST will be rejected.Required Skills: SQL, Databricks, Azure Data Factory/ADF, DataStage, Python, PySparkDescription:8–10 Years Experience | Databricks Migration & Data ArchitectureWe are seeking a highly skilled Lead Data Engineer with deep hands-on experience in modern data engineering, BI platforms, and large-scale data migrations. This role requires strong technical leadership, end-to-end solution design capabilities, and the ability to work closely with both onshore and offshore engineering teams.Core Technical Requirements:8–10 years of software development and data engineering experienceMinimum 7 years hands-on experience with: SQL, Databricks, ADF, Datastage (or other ETL tools), SSAS cubes, Cognos, Tableau, ThoughtSpot, and other BI toolsAdvanced SQL and Python proficiencyStrong database foundation across SQL and NoSQL systemsExperience writing SQL for raw data processing, Kafka ingestions, ADF pipelines, data validation, and QAFamiliarity with APIs for ingestion and data integrationExperience with modern cloud platforms (Azure, AWS, or equivalent)Experience with Spark, Hive, and other distributed compute enginesProficiency with CI/CD frameworks such as GitHub, Jenkins, etc.Programming/scripting familiarity: Scala, Java, Python, R (any combination)Key Responsibilities:Data Architecture & ModelingDesign scalable, future-state data architectures and data modelsTranslate business requirements into logical and physical data modelsEnsure data quality, governance, and reduced redundancyData Integration & ETLLead data integration from diverse sources (databases, warehouses, external systems)Design and implement ETL/ELT pipelines to ensure data accuracy, consistency, and availabilitySolution Engineering & BI ImplementationEvaluate and implement BI tools, reporting platforms, and analytics technologiesConduct POCs and guide tool configuration and rolloutLead development of analytical solutions using SQL, Databricks, Tableau, and related technologiesPerformance OptimizationMonitor and tune performance across data pipelines and BI environmentsOptimize queries, data structures, and caching strategiesLeadership & CollaborationServe as the technical lead to offshore engineering teamsWork cross-functionally with product, engineering, data, and business stakeholdersTranslate complex concepts for both technical and non-technical audiencesParticipate in design reviews, code reviews, and solution walkthroughsContinuous ImprovementStay current on modern data engineering trends and best practicesIdentify opportunities for automation, optimization, and innovationGovernance & SecurityEstablish and maintain data governance policiesDefine access controls, data privacy practices, and retention standardsAdditional Expectations:Experience writing technical documentation, including design docs and source-to-target mappingsAbility to convert business requirements into clear, actionable Jira storiesStrong analytical skills: capable of working with large datasets to identify trends and insightsExperience building highly scalable BI solutions and distributed data applicationsComfortable working in a fast-paced, agile, startup-like environmentStrong interpersonal skills; collaborative, clear communicator, and able to mentor othersSelf-motivated, organized, and capable of managing multiple prioritiesEnjoys contributing to a positive, fun, high-performing team culture