AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
When you\'re looking for a place to grow, make an impact, and work with people who care, we\'d love to meet you!
ABOUT THE ROLE
We are looking for a Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.
WHAT YOU WILL DO
- Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
- Make architectural decisions on partitioning strategy, file formats, schema and data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
- Own the design of scheduled batch workflows (DAGs) on the client\'s Airflow setup, defining pipeline structure, dependencies, and triggering strategy, while driving architectural discussions. Not responsible for administering Airflow itself.
- Drive use of the client\'s AWS data stack (S3-backed data lake, Athena, EKS/Kubernetes), and partner directly with the client\'s DevOps team to clarify functional and non-functional requirements.
- Review pull requests and enforce code quality standards.
- Guide senior developers and ensure alignment with the client\'s engineering practices.
MUST HAVES
- 7+ years of engineering experience, with a proven track record designing and implementing ETL pipelines and making architectural decisions for large-volume data systems .
- Hands-on experience with OLAP-style analytical data architecture . Experience with Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies is acceptable; a specific stack isn\'t mandatory as long as the OLAP depth is real.
- Hands-on experience designing against a data lake sitting on object storage (S3 or equivalent) queried via a serverless engine — including partitioning strategy, file formats (Parquet/ORC), and the cost/performance tradeoffs that come with them. Athena specifically is a plus, not a requirement.
- Deep familiarity with DAG-style workflow definition and triggering . The client orchestrates most batch processing through Airflow, so this role needs either substantial prior Airflow experience they can draw on to drive architectural conversations, or enough depth in a comparable orchestrator (Dagster, Prefect, Luigi, Step Functions) to ramp on Airflow quickly and lead those conversations from day one. Managing the Airflow deployment itself is out of scope.
- Practical experience across the AWS data ecosystem, including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes , with the ability to drive infrastructure conversations with DevOps.
- Strong backend proficiency in Python, including FastAPI or Flask .
- Comfortable working with REST and GraphQL.
- Experience with Docker and PostgreSQL for the transactional and application layer.
- Highly comfortable working in Mac/Linux terminal-centric environments.
- Practical, hands-on use of AI-assisted development tools (e.g., Claude Code) , paired with the critical judgment to challenge AI output when it compromises long-term maintainability — including the leadership presence to set the standard for how the team uses AI tooling responsibly (e.g., flagging risky AI-driven shortcuts during PR review).
- Strong soft skills: the ability to hold and defend a technical opinion — challenging a stakeholder's or a tool\'s proposed "quick fix" with sound reasoning in pursuit of a solution that scales and is maintainable long-term, while still being pragmatic enough to ship.
- Upper-intermediate English level.
NICE TO HAVES
- Direct production experience with Athena.
- Working knowledge of TypeScript and React to guide integrations and review frontend-adjacent pull requests.
- Production experience building AI features using AWS Bedrock, LangChain, Pydantic AI, or similar technologies.
- Experience with monorepo tooling such as Nx or modern package managers such as Poetry, UV, or Yarn.
- Experience with Redis, caching layers, or SageMaker.
- Experience with marketing data structures, campaign management APIs, or digital advertising metrics.
PERKS AND BENEFITS
- Growth without limits : build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
- Competitive compensation : get recognition that reflects your skills and impact, with regular performance and compensation reviews
- Flexibility : work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
- Meaningful, modern projects : build impactful products using modern technologies alongside global teams and leading brands
- Collaborative culture : join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
- Well-being & support : access local well-being programs and people-focused support tailored to your location