Senior Data Architect/Lead Data Engineer

Senior Data Architect/Lead Data EngineerImmediate need for a talented Senior Data Architect/Lead Data Engineer. This is a 12 months contract opportunity with long-term potential and is located in Jersey City NJ (Hybrid).Job ID: 26- 20307Pay Range: $60/hr - $68/hour. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).Key Responsibilities:Enterprise & Surveillance ArchitectureDefine target state architecture and strategic roadmap for surveillance data processing platforms leveraging Hadoop, Spark, PySpark, and Python.Establish standard architectural patterns for alert generation, enrichment, aggregation, and reporting pipelines.Ensure architecture aligns with enterprise technology standards and Surveillance control expectations.Solution Design & Delivery SupportTranslate surveillance business requirements (e.g., market misconduct detection, regulatory coverage) into scalable technical designs.Review and approve detailed technical designs, ensuring alignment with functional intent, regulatory requirements, and architecture standards.Provide hands-on architectural guidance to engineering teams during development, testing, and implementation.Design and standardization of Spark/PySpark frameworks supporting surveillance alert generation and enrichment.Modernization and optimization of large-scale Hadoop surveillance workloads to improve performance, stability, and control coverage.Implementation of enterprise-consistent architecture patterns enabling audit readiness, lineage, and regulatory traceability.Support end-to-end delivery across the SDLC, minimizing rework and technical debt.Architecture Governance & LeadershipParticipate in and lead architecture review forums and design walkthroughs.Mentor senior engineers and promote adoption of standard frameworks and best practices.Influence enterprise surveillance platform strategy through architecture governanceDocumentation & CommunicationMaintain high-quality documentation, including business requirements, data definitions, and process flows.Proactively communicate risks, dependencies, and potential timeline impacts to stakeholders.Non-Functional Requirements & Production ReadinessOwn non-functional requirements including performance, scalability, resiliency, SLAs, auditability, and disaster recovery.Define and enforce production readiness criteria, including monitoring, alerting, logging, and operational hand-off.Partner with support and operations teams to ensure platform stability and timely issue resolutionData Controls, Risk, and ComplianceEnsure designs meet Surveillance data control requirements: completeness, accuracy, timeliness, lineage, and traceability.Align architecture with enterprise security standards, including encryption, access control, and PII handling.Support internal audits, regulatory exams, and model governance reviews by providing architectural documentation and control evidence.Regular Points of ContactSurveillance Technology teams to clarify requirements and prioritize architectural decisionsEnterprise Architecture, Information Security, and Production Support teamsCoordinate with platform, data governance, and vendor teams as requiredDirect impact on strengthening risk and compliance controlsExposure to complex financial markets and Dodd-Frank regulatory requirementsHigh-visibility collaboration across compliance and technology organizationsAn environment that values innovation, accountability, and measurable impactKey Requirements and Technology Experience:Key skills; Python, Hadoop, PySpark10 years of experience in data engineering, platform engineering, or architecture rolesDeep expertise in PySpark, including performance tuning and optimizationStrong Python development experience in large-scale distributed environmentsSolid knowledge of Hadoop ecosystem (HDFS, Hive/Impala, YARN)Proven experience designing and governing enterprise, regulator-facing data platformsStrong understanding of non-functional requirements and production support modelsClear written and verbal communication skills with ability to influence across organizationsFinancial services experience, particularly in Market Surveillance, AML, Fraud, or Risk TechnologyExperience supporting regulatory or audit-facing platformsKafka and Spark Structured Streaming exposureFamiliarity with orchestration tools (Airflow, Control-M, Oozie)Knowledge of data governance, lineage, and data quality controlsCharacteristics of a Top PerformerDemonstrates strong ownership and accountability in regulator-facing environmentsBalances strategic architecture thinking with practical delivery executionProactive in identifying risks, control gaps, and optimization opportunitiesEffective communicator across engineering, compliance, and leadership audiencesOur client is a leading Financial Industry, and we are currently interviewing to fill this and other similar contract positions.

If you are interested in this position, please apply online for immediate consideration.Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.By applying to our jobs you agree to receive calls, AI-generated calls, text messages, or emails from Pyramid Consulting, Inc. and its affiliates, and contracted partners.

Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages.

You can reply STOP to cancel and HELP for help. You can access our privacy policy here.