Senior Databricks EngineerOne of our large clients in the energy industry is seeking a hands on Senior Databricks Engineer to support a large-scale modernization initiative focused on migrating complex pricing, forecasting, and trading models to Databricks. This role will primarily support the Pricing workstream, which is the most complex and business-critical component of the program. The consultant will partner closely with technical leads, data engineers, and business stakeholders to translate existing Python and PySpark pricing models into scalable, production-ready Databricks solutions.
This individual must be highly technical and execution-focused, with proven experience building and deploying enterprise Databricks solutions rather than simply designing architectures or providing theoretical guidance. Success in this role requires strong collaboration skills, as the engineer will work directly with pricing model owners and business users to gather requirements, validate model outputs, and ensure successful migration and adoption.Key Responsibilities:Lead the migration of existing Python and PySpark pricing models into the Databricks platform.Design, develop, test, and deploy scalable Databricks solutions supporting pricing analytics and business operations.Build and maintain Databricks notebooks and collaborative development workflows.Develop and optimize large-scale data ingestion pipelines from multiple enterprise systems.Translate legacy pricing logic into modern cloud-native Databricks implementations while maintaining business accuracy.Configure and maintain connectivity between Databricks and Azure SQL Server / Azure SQL Managed Instance environments.Support data governance and cataloging efforts using Unity Catalog.Create reusable Python libraries and package them as wheel files for deployment across Databricks environments.Implement job orchestration, workflow automation, and scheduled processing within Databricks.Utilize Databricks Repos and CI/CD pipelines to support source control, testing, and deployment practices.Participate in code reviews and promote engineering best practices across the development team.Partner directly with business stakeholders to gather requirements, document functionality, and validate migrated pricing models.Troubleshoot production issues and optimize performance of data pipelines and model execution.Collaborate with architects, technical leads, and business teams to ensure successful project delivery.Contribute to the overall cloud modernization strategy supporting Pricing, Forecasting, and Trading capabilities.