Our client is seeking a skilled and passionate Databricks Engineer to design and build scalable, cloud-based data pipelines for enterprise analytics platforms. The role focuses on creating robust ETL/ELT workflows using Databricks and Spark, with opportunities to leverage advanced features such as Delta Live Tables (DLT) and cloud-native tools. The ideal candidate has strong hands-on experience in PySpark and SQL, and a proven ability to build pipelines from the ground up while collaborating with cross-functional teams.
KEY RESPONSIBILITIES
- Design and develop scalable batch and streaming data pipelines in Databricks using PySpark, SQL, and Delta Lake.
- Build ETL/ELT processes to transform and integrate data from diverse sources into unified analytical datasets.
- Leverage Delta Live Tables (DLT) and Databricks Workflows to automate and orchestrate pipelines (preferred).
- Optimize pipelines for performance, cost-efficiency, and reusability by applying Spark tuning and partitioning strategies.
- Collaborate with architects, data modelers, and analysts to implement business logic into pipelines.
- Implement data quality checks, monitoring, and observability practices across pipelines.
- Participate in Agile delivery processes, including sprint planning, code reviews, and deployments.
- Work with cloud platform teams to ensure proper storage, networking, and security setup.
- Deliver high-quality outputs, meet deadlines, and maintain strong client relationships.
- Provide consultative guidance, seek creative solutions, and proactively identify opportunities for improvement.
- Stay current with evolving technology trends and best practices in data engineering.
REQUIRED SKILLS & EXPERIENCE
- 5+ years of experience in data engineering, including at least 3+ years with Databricks.
- Strong hands-on experience with Apache Spark and/or Delta Live Tables (DLT).
- Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
- Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).
- Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
- Experience with structured and semi-structured data from APIs, cloud storage, and enterprise systems.
- Strong understanding of modern data architecture concepts (Medallion architecture, lakehouse patterns).
- Knowledge of Git-based version control, CI/CD practices, and infrastructure automation.
- Excellent problem-solving and communication skills, particularly in client-facing settings.
- Bachelor’s degree in Computer Science, Information Systems, or equivalent experience.
PREFERRED QUALIFICATIONS
- Experience with Delta Live Tables (DLT) for declarative pipeline development.
- Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other integration platforms.
- Databricks or cloud provider certifications (Azure/AWS).
- Familiarity with data governance tools (Unity Catalog, Purview, Collibra).
- Experience with REST APIs and streaming technologies (Kafka, Event Hubs, etc.).
CERTIFICATIONS
Databricks Certified Data Engineer or equivalent cloud certifications (preferred).
EDUCATION
Bachelor’s degree in computer science, Information Systems, or a related field (Master’s degree preferred).
#J-18808-Ljbffr