18 Nov
|
JoinStartup
|
Winnipeg
18 Nov
JoinStartup
Winnipeg
Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine conversation analysis, annotation, rubric design, and statistical sampling to create an accurate picture of agent performance. Your work will support Cresta’s AI Receptionist, which handles millions of conversations monthly across hundreds of customers.Key Responsibilities:
Define rubrics across adherence, outcome, and conversation experienceReview production conversations and classify failure modes weeklyBuild automated judges, golden sets,
and regression suitesCalibrate judges against human labels and measure agreementReport prioritized quality findings to engineering and product ownersRequirements:
Three or more years in software engineering, applied ML, or researchOne to two years working with production LLM systemsPython programming skills and experience with LLM APIsExperience building end-to-end evaluation systemsDegree in a related field or equivalent practical experienceBring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.
#J-18808-Ljbffr
📌 Ai Agent Evaluation Engineer (Winnipeg)
🏢 JoinStartup
📍 Winnipeg