Senior AI QA Engineer

About the Company 

Waverley Software is a global software engineering powerhouse dedicated to solving complex digital challenges. We partner with innovation-driven clients to build production-ready enterprise applications using cutting-edge technologies. Our culture thrives on engineering excellence, transparent communication, and a passion for pushing the boundaries of what is possible in the AI landscape.

Role Summary 

We are looking for a Senior AI Quality Assurance Engineer to ensure the reliability, accuracy, and security of our AI-driven products. During project initiation, you will consult with clients to define quality metrics and testing strategies for probabilistic systems. Once in the execution phase, you will build and run the automated testing pipelines, evaluating model outputs, mitigating hallucinations, and assuring overall software quality.

Key Responsibilities

• Pre-Sales Support: support pre-sales team when needing QA insights/inputs.
• Test Automation: Architect and maintain automated CI/CD testing pipelines for both traditional software features and AI-specific workflows.
• Model Evaluation: Systematically evaluate LLM and RAG pipeline outputs for contextual accuracy, retrieval relevance, and latency.
• Quality Assurance: Execute comprehensive API, UI, and integration testing to ensure enterprise-grade stability.

Required Qualifications

• 6+ years of QA automation engineering experience using tools like Selenium, Cypress, or Playwright.
• Senior-level QA capabilities, including generating comprehensive Test Plans, defining QA technical standards, and owning the end-to-end Defect Management Lifecycle.
• Strong scripting skills in Python and/or JavaScript.
• Experience testing RESTful APIs and modern web applications.
• Hands-on experience with LLM evaluation frameworks and observability tools (e.g., Ragas, TruLens, LangSmith, or Promptfoo).
• Data expertise, including data management, preparation, generation, and quality assessment.
• Knowledge of CI/CD pipelines and automated deployment processes.
• Strong communication skills, with the ability to communicate clearly to both technical and non-technical stakeholders.

Preferred Qualifications (Nice-to-Haves)

• Experience with AI security testing, including prompt injection vulnerabilities.
• Familiarity with load and performance testing for AI endpoints (JMeter, Locust).
• Background in DevOps and CI/CD configuration (GitHub Actions, Jenkins, GitLab CI).