AI Test Engineer

Mid-Level AI Software Test EngineerWe are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready. The ideal candidate has a strong software testing background, experience building automated test frameworks, and an interest in AI technologies such as LLMs, agents, RAG systems, MCP integrations, and machine learning models.

AI engineering organizations increasingly emphasize AI evaluation frameworks, automation, reliability, governance, and production quality, making testing a critical function in AI delivery.Key ResponsibilitiesDevelop comprehensive testing strategies for AI applications, platforms, and services.Validate AI model outputs for accuracy, consistency, reliability, and safety.Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.Validate AI guardrails, business rules, permissions, and governance controls.Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations.Build and maintain automated test frameworks for AI applications.Develop automated evaluation pipelines for AI responses and workflows.Integrate AI testing into CI/CD pipelines.Implement automated quality scoring and regression detection.Create reusable test data, mocks, simulators, and validation frameworks.Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities.Validate integrations with external systems, data sources, and enterprise services.Verify performance, reliability, scalability, and resiliency of AI workloads.Execute load and stress testing for AI services.Partner with software engineers, AI engineers, product owners, architects, and security teams.Participate in design reviews and provide quality feedback during development.Contribute to test strategy, quality standards, and best practices.Support production readiness reviews and defect triage activities.Required QualificationsBachelor's degree in Computer Science, Software Engineering, Information Systems, or related field.3–6 years of software testing, QA automation, or quality engineering experience.Experience developing automated test solutions using:PythonJavaJavaScript/TypeScriptC#Experience with API testing and automation tools.Strong understanding of:Test automationSoftware development lifecycle (SDLC)Agile methodologiesCI/CD pipelinesExperience testing distributed systems, web applications, and APIs.Preferred QualificationsExperience testing:Generative AI applicationsLLM-based systemsAI agentsRAG applicationsMCP-based integrationsFamiliarity with:OpenAIClaudeGeminiAzure OpenAIExperience building evaluation and benchmarking frameworks for AI solutions.Experience testing cloud-native applications on Azure, AWS, or GCP.Knowledge of responsible AI, AI governance, and AI risk management practices. AI initiatives are typically expected to align with enterprise governance, risk, privacy, and compliance requirements.Technical SkillsAI TestingPrompt TestingResponse EvaluationHallucination DetectionAgent Workflow ValidationRAG ValidationAI Safety TestingAI Regression TestingAI BenchmarkingQuality EngineeringTest AutomationAPI TestingIntegration TestingPerformance TestingLoad TestingSecurity TestingDefect AnalysisRoot Cause InvestigationTools & TechnologiesSeleniumPlaywrightPostmanJUnit / PyTestGitHub ActionsJenkinsDockerKubernetesCloud Platforms (Azure, AWS, GCP)Success MeasuresEstablishes reliable automated AI testing coverage.Detects AI quality issues before production deployment.Reduces regression defects across AI releases.Improves confidence in AI model and agent behavior.Enables safe, scalable delivery of AI-powered features.Ensures AI solutions meet quality, security, and governance requirements.Example ProjectsTesting AI chat assistants and copilotsValidating MCP tools and AI agent workflowsEvaluating RAG search quality and grounding accuracyAutomating AI response evaluation frameworksTesting AI-powered trading assistants and workflow automationPerformance testing AI services and orchestration platformsTypical LevelExperience: 3–6 years software quality engineering, with 1–3 years exposure to AI/ML or Generative AI technologies.Equivalent Titles:AI Software Test EngineerAI Quality EngineerGenerative AI Test EngineerAI Automation EngineerAI Validation EngineerSoftware Engineer in Test (AI) (SDET-AI)"Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of – Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans."