Mid-Level Big Data EngineerThe Big Data Engineers on AWS will be responsible for analyzing requirements, prototyping data analysis solutions (primarily in Hive, Hadoop, python), collecting, preparing data for modeling and productionizing the data preparation pipelines on AWS. Candidates need to have strong capabilities in large data warehouses using relational and/or Hadoop based systems, UNIX scripting, as well as a database skills (Postgres). The complexity driving this project is the domain understanding and volume of financial data in the petabytes of disparate data sets on an integrated platform providing interactive analytics to the users.Required Skills and Responsibilities:Data Analysis and data profiling.
Understand logical, physical and conceptual data models.Excellent understanding of concepts and hands-on experience on Big Data platforms, Hadoop, Hive, Spark on AWSMust have extensive experience with data preparation pipeline implementations, data storage and distribution on AWS with a security mindsetOptimization and tuning of data structures and queries on Big Data platforms on AWSA compelling track record of building large scale systems utilizing Big Data TechnologiesStrong Programming experience Python and/or Java or Scala and building user defined functionsExperience with Data Analytics tools like Databricks/Dataiku/Domino for data wrangling to ideate analyticsUnix / Shell scripting, strong SQL, including analytic functions.AWS RDS Postgres database SQL skillsExperience with AWS Devops functionality with security mindset: Jenkins, Console, Services, Roles, Security groups, AMI refreshes etc.