Big Data Engineer ConsultantBig Data Engineers serve as the backbone of the Strategic Analytics organization, ensuring both the reliability and applicability of the team's data products. They have extensive experience with ETL design, coding, and testing patterns as well as engineering software platforms and large-scale data infrastructures.Big Data Engineers have the capability to architect highly scalable end-to-end pipeline using different open source tools, including building and operationalizing high-performance algorithms.Big Data Engineers understand how to apply technologies to solve big data problems with expertKnowledge in programming languages like Java, Python, Linux, PHP, Hive, Impala, and Spark.Extensive experience working with both 1) big data platforms and 2) real-time / streaming deliver of data is essential.Big data engineers implement complex big data projects with a focus on collecting, parsing, managing, analyzing, and visualizing large sets of data to turn information into actionable deliverables across customer-facing platforms.They have a strong aptitude to decide on the needed hardware and software design and can guide the development of such designs through both proof of concepts and complete implementations.Additional qualifications should include:Tune Hadoop solutions to improve performance and end-user experienceProficient in designing efficient and robust data workflowsDocumenting requirements as well as resolve conflicts or ambiguitiesExperience working in teams and collaborate with others to clarify requirementsStrong co-ordination and project management skills to handle complex projectsExcellent oral and written communication skillsJob Responsibilities Big Data Engineer:Translate complex functional and technical requirements into detailed design.Design for now and future successHadoop technical development and implementation.Loading from disparate data sets. by leveraging various big data technology e.g.
KafkaPre-processing using Hive, Impala, Spark, and PigDesign and implement data modelingMaintain security and data privacy in an environment secured using Kerberos and LDAPHigh-speed querying using in-memory technologies such as Spark.Following and contributing best engineering practice for source control, release management, deployment, etcProduction support, job scheduling/monitoring, ETL data quality, data freshness reportingSkills:5-8 years of Python or Java/J2EE development experience3+ years of demonstrated technical proficiency with Hadoop and big data projects5-8 years of demonstrated experience and success in data modelingFluent in writing shell scripts (bash, korn)Writing high-performance, reliable and maintainable code.Ability to write MapReduce jobsAbility to set up, maintain and implement Kafka topics and processesUnderstanding and implementation of Flume processesGood knowledge of database structures, theories, principles, and practices.Understand how to develop code in an environment secured using a local KDC and OpenLDAP.Familiarity with and implementation knowledge of loading data using Sqoop.Knowledge and ability to implement workflow/schedulers within OozieExperience working with AWS components (EC2, S3, SNS, SQS)Analytical and problem- solving skills, applied to Big Data domainProven understanding and hands-on experience with Hadoop, Hive, Pig, Impala, and SparkGood aptitude in multi-threading and concurrency concepts.B.S. or M.S. in Computer Science or engineeringFounded in 2007, InterSources Inc is a Small Business Enterprise (SBE), Minority Business Enterprise (MBE) & Women-Owned Small Business (WOSB) Certified Company specializing in providing IT Consulting, IT Staffing Solutions, and Software solutions.
We have been recipients of Various Awards under "Fastest Growing IT Consulting and Software Company " and "Excellence in Technology Services "