Databricks Engineer

Databricks Data Engineer

Location: New York, NY — Hybrid

Qualifications, skills, and all relevant experience needed for this role can be found in the full description below.

About the Role

We are seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions. You will work closely with data architects, analysts and business stakeholders to deliver reliable, high-quality data for analytics, reporting and AI initiatives.

This is a hands-on engineering role requiring strong experience with Databricks, PySpark and SQL.

Key Responsibilities

• Design, develop and maintain data pipelines using Databricks, PySpark and SQL.
• Build batch and streaming workflows to ingest data from databases, APIs and enterprise applications.
• Develop and maintain Delta Lake tables across bronze, silver and gold layers.
• Implement data transformation, validation and quality controls to ensure accurate, reliable datasets.
• Optimise Spark workloads, cluster configurations and processing costs.
• Implement governance, access controls and data discovery through Unity Catalog.
• Troubleshoot pipeline failures and resolve performance and reliability issues.
• Build reusable engineering components and follow coding, testing and documentation standards.
• Support version control and automated deployments through Git and CI/CD pipelines.
• Collaborate with stakeholders to translate business requirements into technical solutions.

Required Experience

• Strong commercial experience in data engineering, including hands-on Databricks development.
• Proficiency in Python, PySpark and SQL.
• Experience building ETL/ELT pipelines and working with large datasets.
• Practical knowledge of Apache Spark, Delta Lake and lakehouse architecture.
• Experience with at least one major cloud platform: Azure, AWS or GCP.
• Understanding of data modelling, warehousing and data quality practices.
• Experience troubleshooting distributed processing workloads and improving performance.
• Familiarity with Git, CI/CD and software engineering best practices.
• Strong communication skills and the ability to collaborate with technical and business teams.
• Ability to work on a hybrid basis in New York.

Preferred Experience

• Databricks certification, such as Data Engineer Associate or Professional.
• Experience with Unity Catalog, Lakeflow Jobs and Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables).
• Experience with streaming, change data capture and incremental processing. xzgxqkn
• Familiarity with Terraform or other infrastructure-as-code tools.
• Experience supporting data platforms used for analytics, machine learning or AI.