Sr. Data Engineer (Azure, Snowflake, Big Data) | TELECOMMUTE | EDM |

Sr. Data Engineer (Azure, Snowflake, Big Data)

Contract

Summary of the project/initiative:

• The contractor will be supporting the Clinical Solutions team by filling in on our new platform called OHBI. OHBI is built on Azure and Snowflake.
• The contractor will be responsible initially for the Snowflake work tied to our ingestion process.
• If the candidate proves successful on the Snowflake side of things, the contractor may start to do some work on the ADF side of things where they would do the ingestion of the data directly from the source system.
• There is also a lot of work that we have in our backlog around DevOps and overall environment improvements which also could get assigned, depending on the individual’s skill set.

Describe the team:

• The current team size is approx. 21 people.
• The team make up consists of a Product Owner, Scrum Master, Solution Architect, Lead Engineer, 2 Data Architects, 1 Data Analyst and the rest of the team are Data Engineers.
• The team consists of both onshore and offshore resources.

Top 5-10 responsibilities:

• The team uses the Agile Methodology and works in 2-week sprints and does releases every 2 weeks. The resource selected, will be responsible for the work assigned to them by the Product Owner and Lead Engineer.
• Tasks include but are not limited to the following:
• Building Snowflake objects (tables, pipes, streams, tasks) to support the automated loading of data
• Developing ADF pipelines to move data from enterprise sources (on-prem and cloud) to Snowflake
• Helping with DevOps work and other backlog work
• Analyzing user reported issues to determine root cause and proposed solutions
• Managing cloud infrastructure (Azure, Terraform, Jenkins, Kubernetes, etc.)
• ELT modeling with DBT

Ideal candidate background:

• Ability to work and understand ELT - concepts and methodologies
• Data movement within ADF pipelines, scheduling, and administration
• Setting up alerting and notification
• Working experience with SQL databases like Oracle, SQL Server and MySQL and ability to code in Python
• Working experience with various data file format (CSV, Parquet, AVRO, JSON, etc.)
• Working experience developing and optimizing Azure Data Factory pipelines to move data large amounts of data
• Working experience ingesting data from Kafka, AWS, Databricks and REST APIs to Snowflake
• Experience migrating database changes across environments in cloud platforms
• Knowledge of Hadoop, Spark and other data analytics and warehouse tools
• Performing root cause analysis of existing models and proposing a solution
• Experience developing and maintaining configuration and process management tools/scripts
• Demonstrated experience delivering successful Continuous Integration solutions using Jenkins, Azure DevOps, or other mainstream CI solutions
• Git experience working with enterprise products delivered in a SaaS/Cloud model

Skills/attributes required:

• Undergraduate degree or equivalent experience
• 2-3 years hands on experience with Snowflake, building pipes, streams, Snow SQL
• 2-3 years of experience with Big Data Technologies to include different types of frameworks, Spark/Scala, Kafka, or Airflow
• 2 years of experience with Kubernetes
• 1-2 year of experience with Azure Cloud knowledge
• 2 years Python scripting
• 1-2 years Java and Java script
• 3 years of experience with Agile concepts and working in an Agile environment
• Team player and contributor
• Ability to produce high quality work in a short period of time
• Ability to work remotely and be focused on the job at hand

Skills/attributes preferred:

• Technical skills, ability to communicate and express themselves during the interview. It is key that contractor be able to mesh with the team.
• The team is highly skilled, yet very modest and gets a lot of work done in a short amount of time.