QA AIML Consultant

QA AIML Consultant

Job Description: AI SDET Engineer

Position Overview

We are seeking an experienced AI Software Development Engineer in Test (SDET) to define and advance quality engineering across web, API, mobile, cloud, and AI-enabled products. The ideal candidate combines strong software engineering fundamentals with deep test automation expertise and can validate both deterministic software and probabilistic AI behavior. This role partners closely with Engineering, Product, Design, SRE, Security, and Data/ML teams to build quality into the full delivery lifecycle, improve release confidence, and lead the evolution of our testing strategy, tooling, and people.

Responsibilities

· Own the end-to-end quality strategy, including risk-based test planning, requirement traceability, coverage analysis, test matrices, release-readiness criteria, and production validation.

· Design and maintain scalable automation frameworks for UI, API, mobile, integration, and end-to-end testing using tools such as Selenium, Playwright, Cypress, Appium, TestNG, and pytest.

· Establish maintainable test architecture through modular design, page or screen objects, reusable fixtures, parametrization, test-data management, and appropriate use of BDD, mocking, and stubbing.

· Define API and contract-testing practices for REST and GraphQL services using tools such as Postman, REST Assured, or Karate, while promoting effective unit testing and close collaboration with developers.

· Automate test execution, reporting, quality gates, and deployment controls in Jenkins, GitHub Actions, GitLab CI, or Azure DevOps pipelines.

· Lead performance, load, reliability, resiliency, and security validation using tools such as k6, JMeter, or Gatling; incorporate chaos testing and OWASP-aligned security checks where appropriate.

· Use logs, traces, and metrics from platforms such as ELK, Splunk, Prometheus, Grafana, and Jaeger to diagnose failures, reproduce production issues, and drive root-cause analysis.

· Build automated evaluation harnesses and regression suites for machine-learning models, large language models, retrieval-augmented generation systems, and agentic workflows.

· Define quality KPIs and service expectations, including escape rate, flaky-test rate, automation effectiveness, mean time to detect, and release quality; communicate trends and corrective actions to stakeholders.

· Create effective defect triage and release-readiness processes, influence product and design decisions, and ensure quality risks are visible and acted upon.

· Protect personal and sensitive information through secure test-data generation, access controls, masking, redaction, retention practices, and GDPR/PII-aware validation.

· Mentor and lead QA and SDET engineers, contribute to hiring and onboarding, and strengthen engineering practices across distributed cross-functional teams.

Core Qualifications

· 5+ years of professional experience in software quality engineering, test automation, or an SDET role, with evidence of ownership of complex quality initiatives.

· Proficiency in at least one programming language such as Python, Java, C#, or JavaScript/TypeScript, supported by strong knowledge of data structures, algorithms, object-oriented design, networking, and operating-system fundamentals.

· Hands-on experience designing test frameworks and maintainable automation architecture with Selenium, Playwright, Cypress, Appium, TestNG, pytest, or comparable tools.

· Strong knowledge of test design techniques, risk-based testing, integration and end-to-end strategy, TDD/ATDD practices, requirement traceability, coverage analysis, and test-matrix management.

· Experience testing REST and GraphQL APIs and service contracts with Postman, REST Assured, Karate, or equivalent tooling; ability to use mocks and stubs to isolate dependencies.

· Practical experience with load and performance testing, reliability or chaos testing, security-testing fundamentals, and OWASP principles.

· Experience integrating automated tests and quality gates into CI/CD pipelines and working with Git, Docker, basic Kubernetes concepts, and infrastructure-as-code tools such as Terraform.

· Strong debugging and root-cause analysis skills using logs, distributed traces, metrics, and production observability platforms.

· Ability to communicate clearly with technical and non-technical stakeholders, influence decisions, and partner effectively with Development, SRE, Product, Design, Security, and Data/ML teams.

· Experience working with quality KPIs and SLAs, supporting release-readiness assessments, and participating in defect triage.

· Working knowledge of GDPR and PII obligations, including secure test-data creation, masking, redaction, and least-privilege access.

AI Testing Expertise

· Experience testing non-deterministic AI systems and translating product expectations into measurable evaluation criteria, representative datasets, scoring rubrics, statistical thresholds, and repeatable acceptance gates.

· Ability to evaluate large language model quality across correctness, groundedness, faithfulness, relevance, completeness, consistency, latency, token usage, cost, and failure behavior rather than relying on exact-match assertions alone.

· Hands-on understanding of retrieval-augmented generation testing, including document ingestion, chunking, embeddings, retrieval recall and ranking, context relevance, citation accuracy, answer groundedness, and stale or conflicting knowledge.

· Experience building golden datasets and regression suites; curating, labeling, versioning, and reviewing evaluation data while preventing leakage, contamination, duplication, and unrepresentative test coverage.

· Knowledge of adversarial and safety testing for prompt injection, jailbreaks, indirect injection, sensitive-data disclosure, insecure tool use, hallucinations, harmful content, toxicity, bias, fairness, and model or agent misuse.

· Ability to test agentic and tool-calling workflows, including multi-step planning, permissions, memory and state, tool selection, retries, timeouts, partial failures, human approvals, and safe recovery.

· Experience validating model, prompt, retrieval, and configuration changes through versioned experiments, reproducible environments, baseline comparisons, confidence intervals, and controlled rollout or canary strategies.

· Understanding of model and data drift, feedback-loop risks, online quality monitoring, human-in-the-loop review, and production signals that detect degradation after release.

· Familiarity with AI evaluation and observability tools such as OpenAI Evals, promptfoo, Ragas, DeepEval, LangSmith, Langfuse, MLflow, Weights & Biases, or equivalent frameworks.

· Knowledge of AI-specific privacy and security risks, including training-data exposure, model inversion, membership inference, prompt and response logging, retention controls, and safe use of synthetic data.

Nice-to-Have Skills

· Experience testing mobile, embedded, distributed, event-driven, or highly regulated systems.

· Experience with cloud-native testing on AWS, Azure, or Google Cloud and with ephemeral test environments managed through Kubernetes and Terraform.

· Familiarity with MLOps practices, model registries, feature stores, data validation, experiment tracking, dataset or artifact versioning, and quality-engineering enablement or open-source tooling.