InfoBeans is hiring an experienced Senior / Lead Data Engineer to design, build, and optimize enterprise-scale data platforms. The ideal candidate has strong hands-on expertise with Azure Databricks, PySpark, ADF, Delta Lake, and modern data warehousing. In this role, you will work closely with architects, analysts, and business stakeholders to deliver high-performance, reliable, and scalable data solutions. If you thrive in fast-paced environments and enjoy solving complex data challenges, we'd love to connect.
Title: Senior / Lead Data Engineer (Databricks) Experience: 10+ years Contract: C2H / W2 / 1099 Duration: 12 months
Job Description: Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategies Delta Lake Internals & Performance Tuning - Optimizing Delta performance, Understanding Delta logs & transaction protocol , Time travel & schema evolution best practices , CDC (Change Data Capture) patterns , Multi-hop architecture (Bronze/Silver/Gold) Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies) , Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using repos Unity Catalog & Enterprise Data Governance - Fine-grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions models Databricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queries Streaming & Real-Time Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late-arriving data, Stateful stream processing Advanced Cluster & Compute Management - Spot vs on-demand clusters, Photon runtime vs standard , Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboards Best Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints) DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub Actions
Nice to have: Azure Data Factory - Understanding of Pipelines, Activities, and Datasets , Linked Services configuration , Integration Runtime types (Auto-IR, Self-Hosted IR, Azure IR) , Source and Sink concepts in ADF ETL/ELT Data Integration - Building data ingestion pipelines (batch + incremental loads) , Designing end-to-end data transformation
Lead Full Stack Engineer Engineer* in charlotte at Unknown Company
This position is listed as contract and onsite.