AI Agent Evaluation Engineer (Winnipeg)

AI Agent Evaluation Engineer (Winnipeg)

17 Nov
|
JoinStartup
|
Winnipeg

17 Nov

JoinStartup

Winnipeg

Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.

As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine conversation analysis, annotation, rubric design, and statistical sampling to create an accurate picture of agent performance. Your work will support Cresta’s AI Receptionist, which handles millions of conversations monthly across hundreds of customers.

Key Responsibilities

Define rubrics across adherence, outcome, and conversation experience

Review production conversations and classify failure modes weekly

Build automated judges, golden sets, and regression suites





Calibrate judges against human labels and measure agreement

Report prioritized quality findings to engineering and product owners

Requirements

Three or more years in software engineering, applied ML, or research

One to two years working with production LLM systems

Python programming skills and experience with LLM APIs

Experience building end-to-end evaluation systems

Degree in a related field or equivalent practical experience

Bring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.

📌 AI Agent Evaluation Engineer (Winnipeg)
🏢 JoinStartup
📍 Winnipeg

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: ai agent evaluation engineer (winnipeg) / winnipeg

Subscribe to this job alert:

Get the latest job offers by email for: ai agent evaluation engineer (winnipeg) / winnipeg