Data Engineer - Databricks

Data Engineer - DatabricksTampa, FL, US Remote Vaco $USD - Direct HireWe are seeking a Data Engineer with strong Databricks expertise to modernize and scale our Business Intelligence (BI) capabilities. This role will design and build data pipelines, deploy machine learning solutions, and operationalize intelligent analytics to drive decision-making across the organization. The ideal candidate blends data engineering best practices with applied machine learning, MLOps, and AI.Key ResponsibilitiesProject Responsibility: End-to-end data pipelines and integrationsTechnical Competencies:Advanced SQL optimization and complex query designKafka streaming applications and connector developmentDatabricks workflow development with medallion architectureData governance implementation and compliancePerformance tuning for large-scale data processingData security and privacy best practicesApache NiFi pipeline development for invoice and PO processingIntegration with purpose-built data stores (Druid, MongoDB, OpenSearch, Postgres)Build and maintain end-to-end ML pipelines for training, deployment, and monitoring of models.Design and optimize data architectures for large-scale ML workloadsExplore and implement LLM-based solutions, RAG architectures, and generative AI for business use cases.Soft Skills:Cross-functional collaboration with product and engineering teamsTechnical mentoring for junior data engineersAnalytical thinking for complex data problemsStakeholder communication for data requirementsProcess improvement and efficiency focusQuality mindset for data accuracy and reliability Vendor Management:Direct communication with data platform vendorsEvaluates vendor tools for specific data use casesProvides technical feedback on vendor product roadmapsCoordinates with vendors for data integration projectsBachelor's/Master's in Computer Science, Data Engineering, Statistics, or related field.5 years in data engineering; 2 years applying ML in production.