The main responsibilities of the position include:
Act as the primary technical enabler for QA, building scalable AI/ML frameworks, libraries, and tooling for the CX Agent Suite and beyondDesign and build AI evaluation pipelines that assess the CX Agent Suite's multi-agent responses (Intent Detector, FAQ Agent, Missing-Deposits Agent) for accuracy, relevance, tone, hallucination rate, safety/guardrail compliance, and task completion — extending or replacing the current DeepEval-based evaluation harnessEvaluate and prototype Strands Agents (or comparable agent-orchestration frameworks) for building self-evolving, autonomous QA agents, and drive the go/no-go decision on adoptionCollaborate with QA, Data Science, and Engineering to integrate AI-driven testing into the existing GitLab CI/CD pipeline (dev → test → staging → prod), alongside Terraform-provisioned, EKS/AgentCore-hosted servicesBuild resilient and adaptive automation that can detect and respond to changes in agent behavior, Bedrock Guardrails configuration, and routing logicDevelop data-driven quality analytics, including root-cause analysis, quality trends, and intelligent test prioritization, leveraging existing observability and evaluation data from OpenTelemetry, CloudWatch, X-Ray traces, and LangFuse offline evaluation runsLead research into emerging AI testing methodologies and mentor engineers through code reviews, workshops, and architectural guidance Main requirements:
BSc/MSc in Computer Science, AI, or a related discipline6+ years of hands-on experience in QA/Test Automation, with strong experience designing and maintaining automation frameworks1+ years of experience applying AI/ML in software testing or QA process improvement, with a proven track record of bringing AI/ML solutions into production workflows Strong Python skills, with experience working in Python/AWS-centric technology environments; Java and/or TypeScript is a plusHands-on experience with LLM evaluation techniques, including LLM-as-a-judge, human-in-the-loop evaluation, RAG, and multi-agent orchestration patterns Practical experience with DeepEval or comparable evaluation frameworks, along with familiarity with agent-orchestration SDKs such as Strands AgentsKnowledge of LLM tooling, vector databases, and MLOps pipelinesExperience integrating AI tooling into enterprise CI/CD environments, particularly GitLab, and working with containerized cloud-native environments such as Docker and Kubernetes/EKSStrong communicator, able to influence technical decisions and drive engineering standards across teams The following will be considered an advantage:
Experience with autonomous QA agents or agentic orchestration frameworks for self-evolving test suitesExperience with LLM observability tools such as LangFuse, LangSmith, or Arize, particularly for measuring probabilistic and adversarial robustness Knowledge of AI ethics, fairness, and bias detection for guardrail and model validationExperience with gRPC, WebSockets, and/or HTTP/2Experience with AWS Bedrock, plus familiarity with GCP Vertex AI and/or Azure AI Benefit from:
Attractive remuneration packageIntellectually stimulating work environmentContinuous personal development and international training opportunities The Hiring Experience: What Awaits You
Let’s Connect – Intro Chat with Talent AcquisitionDeep Dive – First Interview with Your Future TeamFinal Connection – Final Interview