Azure Databricks Platform Architect

Job Title:

Azure Databricks Platform Architect

Location:

Raritan, NJ (Onsite)

Job Summary

We are seeking an experienced Azure Databricks Platform Architect with 8+ years of experience in data engineering, cloud platforms, and enterprise data architecture. The ideal candidate will be responsible for designing, implementing, governing, and optimizing scalable Lakehouse platforms using Azure Databricks and related Azure services. This role requires hands-on expertise in Databricks, Apache Spark, Delta Lake, Unity Catalog, cloud security, DevOps, and platform administration.

Key Responsibilities

• Design and architect enterprise-scale data platforms using Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory, Azure Synapse Analytics, Azure Key Vault, Microsoft Entra ID, and related Azure services.
• Define and implement Lakehouse architecture patterns using Delta Lake, Medallion Architecture, Unity Catalog, and scalable data governance practices.
• Lead the design of secure, scalable, and high-performance batch, streaming, and near real-time data processing solutions.
• Define Databricks workspace architecture, cluster policies, job orchestration strategies, workload isolation, access control models, and platform standards.
• Establish data ingestion, transformation, quality, and serving frameworks using PySpark, Spark SQL, Delta Live Tables, Lakeflow, Structured Streaming, and CI/CD deployment practices.
• Implement security best practices, including RBAC, secrets management, private networking, encryption, data masking, catalog governance, and compliance controls.
• Optimize Databricks workloads for performance, reliability, scalability, and cost efficiency through cluster tuning, query optimization, partitioning, caching, and resource management.
• Collaborate with enterprise architects, cloud infrastructure teams, security teams, data engineers, data scientists, analysts, and business stakeholders to translate requirements into platform solutions.
• Provide technical leadership, architecture reviews, design governance, mentoring, and best-practice guidance to engineering teams.
• Support production stability through monitoring, logging, observability, incident response, root cause analysis, and continuous improvement practices.

Required Skills

• Deep expertise in Azure Databricks platform architecture, workspace administration, cluster configuration, job orchestration, and platform governance.
• Strong hands-on experience with Apache Spark, PySpark, Spark SQL, Delta Lake, Delta Tables, and performance tuning.
• Strong understanding of Lakehouse architecture, Medallion Architecture, data lake design, data warehouse integration, and data serving patterns.
• Hands-on experience with Unity Catalog, data governance, access controls, lineage, catalog/metastore design, and secure data sharing.
• Experience with Azure services including:
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure Data Factory
• Azure Synapse Analytics
• Azure Key Vault
• Azure Monitor
• Microsoft Entra ID
• Event Hubs
• Azure DevOps
• Knowledge of CI/CD automation, Git-based development, release management, and Infrastructure as Code (Terraform, ARM, or Bicep).
• Experience designing secure cloud networking patterns including private endpoints, virtual networks, managed identities, service principals, firewalls, and encryption.
• Ability to design batch, incremental, streaming, CDC, and event-driven data processing solutions.
• Strong understanding of data quality, monitoring, logging, observability, cost optimization, and operational excellence practices.
• Excellent architecture documentation, solution design, communication, mentoring, and stakeholder management skills.

Preferred Skills

• Azure Data Factory
• Azure SQL

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or a related field.
• Minimum 8 years of experience in data engineering, cloud data platforms, data architecture, or analytics platform implementation.
• At least 3 years of hands-on experience designing and implementing Azure Databricks-based solutions.
• Strong experience in enterprise-scale data platform architecture, Lakehouse design, data modeling, ETL/ELT patterns, and cloud-native engineering.
• Experience working in Agile delivery models with cross-functional teams across architecture, engineering, infrastructure, security, and operations.

Certifications Required / Preferred

Required

• Microsoft Certified: Azure Databricks Data Engineer Associate or equivalent Azure Databricks certification.
• Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.

Preferred

• Databricks Azure Platform Architect Accreditation or Azure Databricks Platform Architect Badge.
• Microsoft Certified: Azure Solutions Architect Expert.

We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, citizenship status, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable law.

https://www.e-verify.gov/sites/default/files/everify/posters/IER_RighttoWorkPoster.pdf