Principal Data Engineer

We are looking for an experienced Principal Data Engineer to lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP) .

The ideal candidate will have strong hands-on expertise in Snowflake, Delta Lake, Databricks, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark , along with deep knowledge of data engineering architecture, ETL/ELT pipelines, data modeling, cloud platforms, and performance optimization.

This role requires a combination of hands-on technical expertise, architectural leadership, and the ability to establish engineering standards, mentor senior engineers, and deliver scalable, secure, and cost-efficient data solutions for finance and revenue data.

Key Responsibilities

• Architect, design, and lead the development of enterprise-scale ETL/ELT data pipelines for finance and revenue data, including billing, general ledger (GL), operating expenses, and revenue reporting.
• Establish standards for batch, incremental, and streaming data ingestion using Change Data Capture (CDC), watermarking, and event-driven processing .
• Design scalable, idempotent, fault-tolerant, and high-performance data pipelines across enterprise platforms.
• Lead the architecture and optimization of Snowflake and Databricks platforms for large-scale data processing and analytics.
• Implement and optimize Delta Lake, Delta Live Tables (DLT), and Unity Catalog for data processing, governance, and secure data access.
• Develop modular, reusable, and maintainable data transformation models using dbt , including testing and CI/CD integration.
• Build and optimize distributed data processing solutions using Python, PySpark, and advanced SQL .
• Establish orchestration standards using Apache Airflow and/or Azure Data Factory (ADF) .
• Design and implement real-time and near-real-time data processing solutions using Kafka, Azure Event Hubs, and Spark Structured Streaming .
• Establish enterprise frameworks for data quality, validation, observability, reconciliation, and SLA monitoring.
• Optimize query performance, compute utilization, storage, partitioning, clustering, and overall platform costs.
• Implement CI/CD pipelines and engineering best practices using GitHub Actions and/or Azure DevOps .
• Ensure data security, governance, lineage, and auditability through RBAC, row-level and column-level security, and appropriate protection of PII and CPNI.
• Support enterprise data modeling, including star schemas, snowflake schemas, Data Vault, and Slowly Changing Dimensions (SCD Type 1/2).
• Collaborate with finance and business stakeholders to implement reliable data solutions for revenue recognition, reconciliation, financial reporting, and period-end close processes.
• Lead architecture and design reviews, establish reusable engineering frameworks, and ensure adherence to technical standards and non-functional requirements.
• Mentor senior engineers and technical leads, resolve complex technical challenges, and drive modernization and continuous improvement initiatives.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong hands-on experience designing and developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions.
• Advanced proficiency in SQL, Python, PySpark, and data transformation techniques .
• Strong understanding of data lakehouse architecture, data modeling, incremental processing, CDC, and pipeline optimization.
• Experience with data orchestration, workflow scheduling, and dependency management.
• Strong knowledge of data quality frameworks, automated testing, observability, and production monitoring.
• Experience implementing CI/CD, version control, code reviews, and automated deployment practices.
• Understanding of data security, access controls, governance, lineage, and compliance requirements.
• Demonstrated ability to lead technical design decisions, establish engineering standards, and mentor engineering teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• Willingness to work onsite 3 days per week in Bellevue, WA, or Frisco, TX .

Preferred Qualifications

• Experience with Azure Data Factory, ADLS Gen2, Azure Event Hubs, and Azure Key Vault .
• Experience with Apache Airflow, Kafka, and Spark Structured Streaming .
• Experience with Terraform or Bicep for infrastructure as code.
• Experience with GitHub Actions and Azure DevOps for CI/CD automation.
• Familiarity with data quality tools such as Great Expectations and dbt testing frameworks.
• Experience with financial and revenue data domains, including billing, general ledger, revenue recognition, and reconciliation.
• Experience optimizing cloud data platforms for performance, scalability, reliability, and cost efficiency.
• Experience defining enterprise data engineering architecture, reusable design patterns, and platform standards.
• Experience mentoring senior engineers and collaborating with architects, product managers, and business stakeholders.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers. o lead the design, development, and optimization of enterprise-scale data pipelines and modern data platforms supporting Finance and Revenue Data Management Platform (RDMP) .