AI Quality EngineerAn experienced AI Quality Engineer is sought to establish and execute the quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions. This is a specialized engineering position focused on validating AI systems, Retrieval-Augmented Generation (RAG) architectures, agent workflows, and orchestration frameworks. The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior.Location: Merrifield, Virginia (Hybrid) preferred. Open to Remote U.S.Key ResponsibilitiesDefine and implement AI certification, validation, and production readiness standards for enterprise AI agents and intelligent automation solutions.Develop and execute evaluation frameworks that measure accuracy, relevance, groundedness, completeness, hallucination rates, and user satisfaction.Design and maintain AI validation datasets, benchmark scenarios, and regression test suites using platforms such as LangSmith and Azure AI Foundry.Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search, Azure AI Foundry, LangChain, and LangGraph.Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps, Azure Functions, Azure Databricks, and related cloud-native services.Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, and retrieval pipelines.Validate security, governance, auditability, and compliance controls.Create production readiness and certification checklists covering observability, monitoring, logging, and operational readiness.Analyze LangSmith traces, agent execution logs, and AI evaluation metrics to identify quality concerns.Partner with development teams to remediate quality, accuracy, security, and performance issues prior to production deployment.Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities and developer experience improvements.Required QualificationsEducation: A Bachelor's Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related technical field is required.Experience: This role requires 5+ years of experience in software quality engineering, test architecture, platform engineering, software development, or machine learning engineering. A minimum of 2+ years of experience working directly with Generative AI, Large Language Models (LLMs), RAG architectures, or AI agents is necessary.Technical Skills: Candidates must have experience with one or more of the following: Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, AI agent frameworks, or RAG architectures.
Experience with cloud-native application architectures and Azure services is also required. A strong understanding of DevSecOps, CI/CD, and enterprise software development practices is expected.Preferred QualificationsExperience building or validating enterprise AI agents and multi-agent systems.Experience within Platform Engineering, Developer Experience (DevEx), or Site Reliability Engineering (SRE) organizations.Experience with AI observability platforms such as LangSmith.Experience validating Azure AI Search implementations and vector databases.Experience implementing AI governance or Responsible AI frameworks.Experience in financial services, banking, or other regulated industries.Familiarity with Azure DevOps, GitHub, and enterprise developer platforms.Knowledge of identity and access management concepts including Entra ID and managed identities.Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRateds Best of Staffing® in Talent Satisfaction in the United States and Great Place to Work® in the United Kingdom and Mexico.