Senior Lead Data EngineerLocation: San Jose, CA 95125 Hybrid(Locals)Key Responsibilities:Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQLManage and optimize the infrastructure ensuring reliability and performanceDesign and implement data models and schemas for enterprise data warehouse and data martsWrite and maintain scripts in Python for various applicationsDevelop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databasesWork closely with analysts and product managers to understand data requirements and deliver high-quality dataProvide technical guidance and mentorship to other team members fostering best practices in data engineeringMonitor system performance, identify bottlenecks, implement solutions to ensure high data availabilityRequired Skills:High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processingGood knowledge in Python for data manipulation and scriptingMastery of advanced SQL and deep experience working with relational databases and data warehousesHands-on experience with workflow management tools like Apache AirflowFamiliarity with Unix shell scripting for automation and system management