Job Requisition ID #

26WD101776

Position Overview

At Autodesk, we're exploring new ways to make design and engineering software smarter, more intuitive, and more personal. AI is helping us rethink how millions of people interact with products like AutoCAD, Revit, Autodesk Construction Cloud, and Forma.

Our Personalized Experiences Team works across Autodesk to bring AI, machine learning, and personalization into our products. We collaborate with product teams, experiment with new ideas, and build capabilities that benefit customers across the Autodesk ecosystem.

Responsibilities

- Design and implement quality engineering strategies for AI-powered personalization, recommendation systems, LLM applications, retrieval-augmented generation (RAG), and agentic systems

- Build and maintain scalable automated functional, integration, API, end-to-end, and performance testing frameworks using TypeScript, Python, Playwright, Puppeteer, Pytest, or equivalent tools across web applications and cloud services

- Develop golden datasets, benchmark suites, regression tests, and automated evaluation pipelines for nondeterministic AI behavior, including rubric-based, statistical, and LLM-as-a-judge evaluation methods

- Evaluate RAG and personalization systems for retrieval quality, groundedness, relevance, ranking accuracy, consistency, and response quality across different user contexts

- Test agentic workflows, including tool selection and execution, multistep orchestration, memory, error handling, authorization boundaries, and end-to-end task completion

- Design automated quality checks for modular and distributed systems, including API contracts, integration boundaries,



component compatibility, architectural regressions, and service reliability

- Integrate automated tests and AI evaluations into CI/CD pipelines, establishing quality gates and release readiness criteria

- Collaborate with applied scientists and developers to identify AI failure modes, evaluate security and privacy risks, and improve reliability, performance, observability, and production quality metrics

- Champion quality engineering best practices, leverage AI-assisted development tools responsibly, and mentor developers across cross-functional teams

Minimum Qualifications

- BS or MS in Computer Science, Software Engineering, or a related field, or equivalent practical experience

- 6 or more years of professional experience in software engineering, quality engineering, test automation, or related disciplines

- Robust hands-on programming experience with TypeScript and Python, including designing and maintaining scalable automated testing frameworks

- Experience with browser automation using Playwright, Puppeteer, or equivalent tools, and Python-based testing using Pytest or equivalent frameworks, covering functional, API, integration, and end-to-end testing

- Practical experience testing or evaluating AI/ML-powered applications, including LLM-based systems,



and developing automated evaluations for correctness, relevance, consistency, and reliability

- Understanding of LLMs, embeddings, RAG, and AI agent architectures, including common failure modes, nondeterministic behavior, evaluation datasets, and regression testing approaches

- Experience testing complex cloud services, distributed systems, or modular applications, including API contracts, integration boundaries, debugging, and system reliability

- Experience with CI/CD pipelines, cloud environments (AWS preferred), performance testing, and production observability using logs, metrics, and traces

- Understanding of data validation, security, privacy, and Responsible AI considerations when testing AI-powered systems

- Ability to work independently, communicate technical findings clearly, and collaborate effectively with applied scientists, developers, and product partners

Preferred Qualifications

- Experience with AI evaluation tools such as DeepEval, Ragas, LangSmith, promptfoo, Braintrust, or similar frameworks, including benchmark design, LLM-as-a-judge, and statistical or human evaluation methods

- Experience testing agentic applications built using LangGraph, LangChain, or comparable orchestration frameworks, including adversarial testing and AI safety evaluation

- Experience evaluating RAG systems using retrieval metrics, alongside groundedness, faithfulness, and answer quality

- Experience with recommendation systems, ranking models, personalization, or search relevance, including offline and online experimentation and A/B testing

- Famili

📌 Senior AI Quality Developer, Personalization (Toronto)
🏢 Autodesk
📍 Toronto