Big Data Engineer with Pyspark

Big Data Engineer

Location: Irving, TX

Job Type: Full-Time (Hybrid)

Experience: 7+ Years

F2F Interview

Job Summary

We are seeking an experienced Big Data Engineer with 7+ years of professional experience in designing, developing, and implementing scalable data solutions. The ideal candidate will have strong hands-on experience with Hadoop, Spark, PySpark, Kafka, Hive, SQL, and cloud-based data platforms.

The candidate will be responsible for building high-performance data pipelines, processing large and complex datasets, and working closely with data scientists, analysts, application developers, and business stakeholders.

Key Responsibilities

• Design, develop, and maintain scalable Big Data pipelines and data processing solutions.

• Develop batch and real-time data processing applications using Apache Spark/PySpark.

• Build and optimize ETL/ELT pipelines for large-scale data processing.

• Work with Hadoop, HDFS, Hive, Spark, Kafka, and other Big Data technologies.

• Develop complex SQL queries and optimize data processing jobs for performance.

• Design data ingestion and transformation processes from multiple structured and unstructured sources.

• Develop real-time streaming pipelines using Kafka and Spark Streaming.

• Perform data cleansing, transformation, validation, and quality checks.

• Troubleshoot data pipeline failures and optimize jobs for scalability and reliability.

• Work with cloud-based data platforms and storage services.

• Collaborate with data architects, data scientists, analysts, and engineering teams.

• Implement data security, governance, and access-control best practices.

• Participate in code reviews, testing, deployment, and production support.

• Contribute to Agile/Scrum development activities and technical documentation.

Required Skills

• 7+ years of experience in Big Data Engineering

• Strong hands-on experience with Apache Spark and PySpark

• Strong programming experience with Python and/or Scala

• Strong SQL skills and experience with relational databases

• Hands-on experience with Hadoop, HDFS, Hive, and MapReduce

• Experience with Apache Kafka and real-time data processing

• Strong experience building ETL/ELT data pipelines

• Experience with data ingestion, transformation, and processing of large datasets

• Knowledge of distributed computing and Big Data architecture

• Experience with data warehouse/data lake concepts

• Experience with Git and CI/CD

• Strong troubleshooting and performance-tuning skills

Cloud & Modern Data Technologies

Experience with one or more of the following is highly preferred:

• AWS: S3, EMR, Glue, Redshift, Lambda

• Azure: ADLS, Databricks, Azure Data Factory, Synapse

• GCP: BigQuery, Dataproc, Cloud Storage

• Databricks

• Snowflake

Preferred Qualifications

• Experience developing both batch and streaming data pipelines.

• Experience with Apache Airflow or similar workflow orchestration tools.

• Knowledge of data modeling and data warehousing concepts.

• Experience with Docker/Kubernetes is a plus.

• Understanding of data governance, security, and data quality frameworks.

• Experience working in Agile/Scrum environments.

• Strong communication and analytical problem-solving skills.