Big Data EngineerAs a Big Data Engineer on the Google Cloud Platform the candidate will help design, develop, deploy, maintain code in Google Cloud Platform using services such as Big Query, Pub Sub, Data Flow, Data Fusion, Google Cloud Storage, Composer, and Looker. Also the role requires enabling mining and data analyzing to help AI leaders, researchers and other business teams make data driven decisions for data collection, diversity, training, and evaluation. The engineer will spend a majority of the time hands-on writing code and peer reviewing high performance, high quality, and well tested and well architected code.
In addition to establishing and maintaining good working relationships internally, this position will participate in various cross-functional organizational initiatives.Spend a majority of the time hands-on writing code and peer reviewing high performance, high quality, and well tested and well architected code.MUST HAVE SKILLS (Most Important):Hands-on experience in GCP Suite (2+ years minimum) such as Big Query, PubSub, DataFlow, Data Fusion, Google Cloud Storage, Dataproc, Composer, and Looker.Solid understanding of Google Cloud architectureStrong work experience in SQL/Data warehousing3+ years of Working experience on tools like Spark, HBase, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.3+ years of programming experience in Scala, Java or Python3+ years of working experience on Hadoop/HDFS, HiveExperience developing and deploying ETL / ELT processes and documentation including physical data model, source to target mappings, ETL / ELT packages (Matillion, Fivetran, Spark etc.)Experience with metadata management, data governance, data catalogs, and data discoveryExperience with data quality monitoring and alerting on dynamic data sources, including anomaly detectionKnowledge of GIT/Jenkins and pipeline automation is a must.Demonstrate excellent verbal and written communications skills with technical and non-technical clientsDemonstrate creative, strategic thinking and problem solving; able to exercise independent judgment in a highly complex environment using leading-edge technologyDESIRED SKILLS:Certification in GCP Data Engineer/ArchitectPrior experience working with: container technology such as Docker, version control systems (Github), build management and CI/CD tools (Concourse, Jenkins), and monitoring tools (App Dynamics, etc.)Working knowledge on the Migration from Hadoop/HDFS to GCPWorking experience on large migration Projects is a big plus.Cost Optimization and Workload managementGood knowledge of Teradata and/or MS SQLServer suiteExperience in creation of analytical dashboards using data visualization tools like Kibana, Tableau, Qlik and associated architectures.Experience in an Agile environment utilizing Rally/One JiraExcellent verbal and written communications skills with technical and non-technical clientsJOB DUTIES:Design and build data engineering solutions using Google Cloud Platform (GCP) services: BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc.Monitoring and improving the quality of the components throughout the development life cycleDesigning and Developing data modules that convert underlying raw data to more readily usable formats for reportingPerforming data deep dives to pinpoint ProblemsIdentify the application bottlenecks and opportunities to optimize performance and optimize the performanceWork on extracting, Loading, Transforming, cleaning, and validating data using cloud ETL/ELT toolsWork with the team to conduct training workshops to identify data sources, flows, and requirementsPreparing required project documentation and tracking and reporting regularly on the status of projects to all project stakeholdersWorking with Business Analysts and other business teams to get the required data to build the semantic views.Periodically update senior management with the status of the project with excellent written and verbal communication skillsTroubleshoot production issues and coordinate with the support team for code deployment.EDUCATION/CERTIFICATIONS:Bachelors degree with 3-5+ years of experience in Data Engineering/BI areas with 2+ years data engineering on GCP and using Bigquery.