Senior Azure Data EngineerLocation: Hybrid in Midtown, NYC Local Only/No RelocationMoI: VideoTotal IT experience: Years working with: Azure/Data Bricks Years working with: 3rd party data integration Years working with: Capital Markets/Trading platforms Years working with: Design, develop, and operate robust, testable Python/PySpark data pipelines.PLEASE Only send me candidates in the NY/NJ area.We need a senior (10+ Years) Azure Data Engineer with extensive experience working in Capital Markets and on actual trading platforms. This is a hands on position integrating 3rd party data who will Own the end-to-end lifecycle of market, alternative, and vendor data—from ingestion to production use—on our Azure + Databricks (PySpark) stack. Candidates must Design, develop, and operate robust, testable Python/PySpark data pipelines as well as & build ingestion frameworks for multiple vendor data sources (APIs, SFTP, flat files, web endpoints), including schema evolution, PII handling, and resiliency/retry patterns.CANDIDATES MUST HAVE RECENT, EXTENSIVE EXPERIENCE INTEGRATING THIRD PARTY DATA FEEDS INTO CAPITAL MARKETS TRADING PLATFORMS.The Role Own the end-to-end lifecycle of market, alternative, and vendor data—from ingestion to production use—on our Azure + Databricks (PySpark) stack. The mandate spans three core functions:Platform & Infra: Build the Azure/Databricks backbone for scalable batch/stream workloads.Pipelines: Design, develop, and operate robust, testable Python/PySpark data pipelines.Support: Production support, monitoring, SLAs, and fast-turn ad-hoc needs for the PM/analyst pod.This is a hands-on role for an engineer who enjoys ownership, polish, and speed.
What You'll DoDesign & build ingestion frameworks for multiple vendor data sources (APIs, SFTP, flat files, web endpoints), including schema evolution, PII handling, and resiliency/retry patterns.Implement Databricks/PySpark transformations, Delta Lake/parquet storage patterns, and efficient table layouts/partitioning for downstream analytics.Stand up and harden Azure services (e.g., Databricks, Storage, Key Vault; plus orchestration such as ADF/Jobs/Workflows) with IaC where practical.Establish observability (logging/metrics, data quality checks, SLAs, alerts) and CI/CD for reproducible deployments.Production support: on-call during market hours for critical pipelines; drive root-cause analysis and permanent fixes.Partner with the PM and analyst to translate investment questions into data models, marts, and fast retrieval patterns.Create lightweight internal tools or UIs as needed (JavaScript experience is a plus) to improve discovery/self-service.Document datasets, lineage, contracts, and runbooks for durable team knowledge.What You'll Bring8-10+ years of hands-on Data Engineering (data engineer first, not primarily analytics).Strong Python and PySpark in Databricks; excellent SQL.Solid Azure experience (Databricks, storage, secrets, orchestration such as ADF/Jobs/Workflows).Proven track record ingesting third-party/vendor data at scale with rigorous data quality controls.Production mindset: testing, version control (Git), packaging, deployment, and monitoring.Clear communication and urgency to support a PM/analyst pod with early start times.Domain: Retail & Consumer exposure helpful but not required; financial-services background not required.Nice-to-haves: JavaScript for small internal tools, Delta Live Tables, Airflow, dbt, Terraform/Bicep.
Data Engineer/ Capital Markets/ Azure/Databricks/Pyspark in new york at Unknown Company
This position is listed as contract and hybrid.