Lead Software Engineer - Data Engineer

Salary: $61,000 - 101,000 per year Requirements:

• We require formal training or certification in software engineering concepts plus 5+ years of applied experience.
• We look for 5+ years of hands-on experience building production data engineering and/or software engineering solutions across design, development, testing, and operations.
• We need practical experience delivering system design, application development, testing, and operational stability for large-scale data pipelines.
• We expect advanced proficiency in one or more programming languages, with strong Python skills and hands-on PySpark experience.
• We require advanced Spark SQL capability and solid SQL fundamentals, including data modeling, query tuning, and execution plan analysis.
• We value proven experience leading the effective use of approved AI-assisted software development tools, with the ability to define standards for validating outputs for correctness, performance, and security.
• We seek a strong understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, and resilient, secure delivery practices.
• We require experience with AWS data management patterns, including S3 and AWS Glue Data Catalog, with other cloud-based data platform experience also considered.
• We require platform experience delivering and operating Spark workloads on EMR and/or Databricks, including tuning, troubleshooting, monitoring, and cost/performance optimization.
• We require production experience with Apache Iceberg, including table design, partitioning strategy, file layout optimization, schema evolution, compatibility controls, compaction, small-file mitigation, snapshot retention, metadata maintenance, safe backfills, rewrites, and reprocessing patterns.
• We require proficiency in automation and continuous delivery methods, including CI/CD, automated testing, and repeatable deployments for data pipelines.
• Preferred: familiarity with Kafka, including topic design, producer/consumer patterns, schema evolution, and operational considerations.
• Preferred: experience with Delta Lake concepts and trade-offs versus Iceberg.
• Preferred: experience with Spark Structured Streaming and streaming ETL patterns.
• Preferred: working knowledge of Java for interoperability or use with existing JVM-based components.
• Preferred: experience using AI-assisted engineering tools and workflows such as GitHub Copilot or Claude, including spec-driven development, prompt-assisted refactoring, and code review in enterprise-safe ways.

• We execute creative software solutions, design, development, and technical troubleshooting, with a strong focus on data engineering and Spark-based ETL/ELT.
• We develop secure, high-quality production code in Python, PySpark, and Spark SQL, and we review and debug code written by others across Spark jobs, SQL logic, and end-to-end data issues.
• We drive adoption of enterprise-approved AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes, while establishing consistent validation standards and encouraging reuse of effective patterns.
• We apply knowledge of software development life cycle tools, including authorized AI-assisted development and automation capabilities, to increase the value delivered through automation.
• We identify opportunities to remove or automate fixes for recurring issues in order to improve application and system stability, including data pipeline reliability and lakehouse maintenance automation.
• We lead evaluation sessions with external vendors, startups, and internal teams to assess architectural designs, technical credentials, and fit within existing systems and information architecture.
• We lead communities of practice across software engineering to increase awareness and adoption of emerging technologies, especially in Spark performance, Iceberg best practices, and data platform operations.
• We contribute to a team culture built on diversity, opportunity, inclusion, and respect.

• AI
• AWS
• AWS Glue
• CI/CD
• Cloud
• Copilot
• Databricks
• ETL
• GitHub
• Support
• JVM
• Java
• Kafka
• Python
• PySpark
• SQL
• Security
• Spark

More:

We are JPMorganChase, working with hackajob to connect us with exceptional professionals for this Lead Software Engineer role within our Commercial & Investment Bank (CIB) Regulatory Reporting Team. We build and deliver trusted, market-leading technology products in a secure, stable, and scalable way as part of an agile team. We offer a competitive total rewards package and a range of benefits and programs, including comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, and financial coaching, with additional compensation details shared during the hiring process. We are committed to diversity, inclusion, equal opportunity, and reasonable accommodations. Our Commercial & Investment Bank is a global leader across banking, markets, securities services, and payments, serving corporations, governments, and institutions in more than 100 countries.

last updated 41 week of 2026