Director of AI Automation Engineering

To enhance the quality and reliability of AI initiatives, the full-time Director of AI Automation Engineering will manage automated testing infrastructure, evaluation frameworks, and CI/CD processes while collaborating closely with AI Engineers and product teams in a remote capacity. Key Responsibilities Own the infrastructure for continuous verification of autonomous agents and manage their lifecycle before production Create automated evaluation frameworks for LLM outputs, ensuring quality and safety benchmarks are met Develop release-gating infrastructure to govern agent deployment and integrate testing into CI/CD pipelines Required Qualifications 5+ years of experience in building or evaluating production AI/ML systems, particularly in agentic AI or LLM-based automation Direct experience with evaluating LLM outputs and implementing safety and accuracy benchmarks Proficiency in building CI/CD pipelines with integrated test gates, such as GitHub Actions Experience in agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus Familiarity with Python and standard testing frameworks (pytest or equivalent)