Data Engineer - SQL, Python, Linux

Title : Big Data Engineer
Location : DC Metro Area Target Start Date : ASAP Type :contract
We are seeking an experienced Data Engineer to design, develop, and optimize large-scale data processing pipelines in a cloud-based environment. The ideal candidate will have strong hands-on experience with Big Data technologies, AWS cloud services, and software engineering best practices. This role requires a highly analytical problem-solver who can build scalable, reliable data solutions, troubleshoot complex processing challenges, and collaborate with cross-functional teams to deliver high-quality enterprise applications. Design, develop, and maintain scalable data pipelines using Big Data technologies such as Apache Spark, Hadoop, Hive, and Trino. Optimize data processing workflows for performance, scalability, reliability, and cost efficiency. Troubleshoot complex data processing issues, including data skew, resource constraints, job failures, and data quality discrepancies. Develop and maintain automated unit, integration, and end-to-end tests to ensure data accuracy and system reliability. Monitor, support, and troubleshoot production data pipelines, proactively identifying opportunities for improvement. Leverage AWS cloud services to build and maintain large-scale data processing solutions. Apply software engineering best practices, including CI/CD, test automation, code reviews, and configuration management. Bachelor's degree in Computer Science, Engineering, or a related technical field with 5 years of relevant experience, or an advanced degree with 2 years of experience. Strong hands-on experience with Big Data technologies, particularly Apache Spark , Hadoop, Hive, or Trino. Experience designing, developing, and optimizing large-scale data processing pipelines. Hands-on experience with AWS services such as S3, EMR, Glue, Lambda, and Athena . Experience troubleshooting distributed data processing challenges, including data skew, memory limitations, resource allocation, and job failures. Strong software development fundamentals, including object-oriented programming, database technologies, and automated testing. Familiarity with CI/CD pipelines, build automation, and software configuration management. Strong analytical and problem-solving skills, including the ability to independently investigate data discrepancies and validate processing results. Advanced degree in Computer Science, Engineering, or a related discipline. Familiarity with financial services, capital markets, regulatory reporting, or other data-intensive industries. AWS certifications or additional cloud computing credentials. Demonstrated interest in emerging technologies through technical projects, hackathons, open-source contributions, or technical publications. Ideal Candidate The ideal candidate combines strong software engineering fundamentals with deep expertise in Big Data processing and AWS cloud technologies. They are comfortable troubleshooting complex distributed systems, optimizing Spark workloads, and independently solving challenging data engineering problems. They bring a proactive mindset, strong communication skills, and a commitment to building scalable, reliable, enterprise-grade data solutions.

Our expertise spans across project services, contract-to-hire, direct search, and managed services onshore, nearshore, and hybrid. For over 25 years, we have connected thousands of consultants with meaningful roles through a personal, communication-driven approach, partnering with a diverse client base to build high-performing teams and create lasting impact. We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.