Technical Data Lead

Technical Data Lead

Location – Fort Mill, SC

Responsibilities

• Design, develop, and implement data pipelines using AWS Glue and Lambda to support data ingestion and transformation.
• Utilize PySpark for data processing and analysis, ensuring efficient data handling and performance optimization.
• Implement Infrastructure as Code (IAC) using Terraform (TF) to automate deployment and management of cloud resources.
• Develop and maintain SQL queries for data extraction, transformation, and loading (ETL) processes.
• Manage containerized applications using Docker and orchestrate them with Amazon EKS.
• Design and optimize data warehouse solutions using Snowflake (SF) or Redshift (RS) to support analytics and reporting.
• Work with data lake technologies such as Iceberg, Lake Formation, and Delta tables to enhance data accessibility and governance.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Ensure data quality, integrity, and security throughout the data lifecycle.
• Stay updated with the latest industry trends and technologies to continuously improve data engineering practices.

Mandatory Skills

• Strong experience with AWS services, particularly Glue and Lambda.
• Proficiency in PySpark for data processing.
• Expertise in Infrastructure as Code (IAC) using Terraform.
• Solid SQL skills for data manipulation and querying.
• Experience with containerization using Docker and orchestration with EKS.
• Hands-on experience with data warehousing solutions like Snowflake or Redshift.
• Familiarity with data lake technologies such as Iceberg, Lake Formation, and Delta tables.
• Proven ability to work in a cloud environment.

Preferred Skills

• Experience with data governance and data quality frameworks.
• Knowledge of data modeling and database design principles.
• Familiarity with Agile methodologies and project management tools.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 8-12 years of experience in data engineering or a related role.
• Proven track record of delivering data solutions in a cloud environment.
• Strong understanding of data architecture and data management best practices.