Evaluate AI-assisted service experiences to identify recurring performance issues and customer friction points.
Investigate issues such as misrouting, incorrect escalations, weak retrieval, poor grounding, low-confidence responses, and hallucination risks.
Perform root cause analysis and translate findings into actionable recommendations.
Partner with technical and business stakeholders to improve AI behavior and customer outcomes.
Develop and execute test plans for prompt changes, routing updates, retrieval improvements, and new AI capabilities.
Validate expected versus actual behavior before production releases.
Identify risks, document findings, and recommend improvements.
Support release readiness efforts and help reduce avoidable regressions.
Support pilots, A/B tests, and iterative rollouts of AI capabilities.
Define hypotheses, success criteria, and measurement approaches.
Analyze results and provide recommendations for scaling, adjusting, or retiring solutions.
Ensure decisions are informed by both customer outcomes and business impact.
Communicate AI performance issues in ways that resonate with technical and non-technical audiences.
Create structured recommendations, issue summaries, and performance narratives.
Help stakeholders understand tradeoffs, risks, and opportunities.
Influence prioritization decisions through evidence-based recommendations.
Partner with service design, journey management, and knowledge teams to ensure AI experiences align with customer needs and intended service outcomes.
Help optimize escalation paths and self-service strategies.
Identify opportunities to reduce customer effort while improving resolution quality.
Track key performance indicators such as containment, digital resolution, customer sentiment, escalation trends, and AI accuracy signals.
Identify emerging issues early and recommend mitigation strategies.
Provide clear context around what changed, why it matters, and what should happen next.
Actively leverage AI tools to improve workflows, analysis, and decision-making.
Stay informed on evolving AI technologies, trends, and best practices.
Help BambooHR continuously improve how AI is used to support customers and employees.
Requirements
5+ years of experience in customer experience operations, digital support, systems optimization, product operations, service design operations, technical program management, analytics, or a related field.
Experience diagnosing complex system behaviors and driving measurable improvements through influence and cross-functional partnership.
Working knowledge of AI-assisted support systems, including prompt behavior, retrieval and grounding concepts, routing logic, escalation patterns, and common AI failure modes.
Experience evaluating AI systems, chatbots, virtual assistants, or automated workflows in production or production-like environments.
Experience designing and supporting testing efforts, pilots, experiments, A/B testing, or iterative rollouts.
Strong analytical and problem-solving skills with the ability to connect technical findings to customer and business outcomes.
Excellent written and verbal communication skills.
Strong stakeholder management and cross-functional collaboration abilities.
Ability to operate effectively in ambiguity and influence without direct authority.
Bachelor's degree or equivalent practical experience.