QA AI Evaluator

About the role

The QA AI Evaluator will be responsible for assuring the quality, reliability, governance, and business effectiveness of Agentic AI solutions. The role focuses on evaluating AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, consistent, and trusted outcomes.

• Define AI evaluation frameworks, standards, KPIs, and success criteria.
• Validate agent accuracy, reasoning, reliability, workflow quality, and business outcomes.
• Assess prompts, LLM responses, hallucination risks, and AI-generated content quality.
• Benchmark models, tools, prompts, and agent configurations to improve performance.
• Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
• Provide clear insights, risks, recommendations, and scorecards to leadership and stakeholders.

What you’ll be doing

•Strong experience in AI Testing, Quality Engineering, AI/LLM Evaluation, or Agentic AI Assurance.
•Deep understanding of LLMs, Prompt Engineering, RAG, AI Agents, and Multi-Agent Architectures.
•Expertise in defining evaluation frameworks, quality metrics, KPIs, scorecards, and benchmarking approaches.
•Experience validating AI accuracy, reasoning, reliability, hallucinations, and business outcomes.
•Knowledge of Responsible AI, Security, Privacy, Governance, Risk, and Compliance practices.
•Strong analytical and problem-solving skills with experience in root cause analysis and continuous improvement.
•Familiarity with AI evaluation tools, test automation, and data-driven quality assessment techniques.
•Ability to collaborate effectively with Product, Engineering, Architecture, QA, and Business stakeholders.
•Excellent communication and stakeholder management skills, including presenting insights to leadership.
•Experience driving AI adoption, change management, and business transformation initiatives.

Essential Skills / Experience

•4-6 Years of experience with strong experience in AI testing, quality engineering, AI/LLM evaluation, or Agentic AI assurance.
•Good understanding of LLMs, prompt engineering, RAG, AI agents, and multi-agent workflows.
•Ability to define evaluation frameworks, quality metrics, KPIs, scorecards, and success criteria.
•Hands-on experience validating AI accuracy, reasoning, reliability, hallucination risk, relevance, completeness, and business outcomes.
•Strong analytical, problem-solving, stakeholder management, and communication skills.

Desirable Skills / Experience

•Minimum 3-4 years of experience with AI evaluation tools, automation frameworks, and data-driven quality assessment techniques.
•Exposure to benchmarking models, prompts, tools, and agent configurations to improve performance.
•Experience supporting AI adoption, user acceptance, change management, or wider business transformation initiatives.