Databricks Architect (Lead Data Platform Architect)

Databricks Architect (Lead Data Platform Architect)

New York, United States | Posted on 06/25/2026

We are looking for a highly skilled Databricks Architect to design, build, and scale enterprise-grade Lakehouse data platforms . This role will drive architecture strategy, platform standardization, and enterprise data modernization initiatives , leveraging Databricks and cloud ecosystems.

The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture , along with strong leadership in driving large-scale data transformations.

Key Responsibilities

• Define and implement end-to-end Databricks Lakehouse architecture .
  
• Design scalable systems for:
  • Batch & real-time data processing
    
  • Structured & unstructured workloads
    
• Establish medallion architecture (Bronze, Silver, Gold layers) as a standard.
  
• Lead deployment and optimization of:
  • Azure Databricks / AWS Databricks / GCP Databricks
    
• Define standards for:
  • Workspace design & cluster strategy
    
  • Job orchestration
    
  • Data storage (Delta Lake)
    
• Drive adoption of:
  • Unity Catalog
    
  • MLflow
    
  • Databricks SQL & Photon
    
• Batch (ADF, Airflow)
  
• Data modeling (star schema, data vault, dimensional models)
  
• Guide engineering teams on best practices in Spark/PySpark optimization .
  

Performance & Cost Optimization

• Query performance
  
• Implement cost governance strategies (auto-scaling, job clusters, spot instances).
  

Data Governance & Security

• Fine-grained access control (RBAC/ABAC)
  
• Ensure compliance with data privacy and regulatory standards .
  
• Integrate Databricks with:
  • Data sources (ERP, CRM, APIs, IoT)
    
  • BI tools (Power BI, Tableau)
    
  • ML pipelines and AI platforms
    
• Collaborate with cloud architects for:
  • Networking, security, and storage strategies.
    
• Provide architectural guidance to data engineers, scientists, and TPMs .
  
• Conduct design reviews and enforce architecture governance .
  
• Mentor teams on emerging patterns:
  • Data Mesh
    
  • DataOps / MLOps
    
  • GenAI workloads on Databricks
    

Skills & Qualifications

Mandatory Skills

• 12+ years of experience in data engineering, architecture, or platform design .