Join Rootly as an AI Evaluation Expert, where your insights will directly influence the future of incident management. This role focuses on rigorous testing and optimization of AI features.As our AI Quality Engineer, you'll play a vital role at Rootly, requiring over five years in QA or AI/ML evaluation. Your mission is to design and execute tests, ensure quality, and collaborate closely with teams to refine AI features. This position values individuals with sharp analytical capabilities and effective communication skills.Key Responsibilities:
Execute prompt-based tests covering various inputs
Analyze AI behavior for relevance and accuracy
Create frameworks for ongoing AI performance evaluation
Document findings on AI behavior and improvements
Define quality metrics for stakeholder reportsRequirements:
5+ years in QA, AI, or related roles
Expertise with LLM and agentic AI systems
Excellent prompt engineering instincts
Comfortable with evaluation tools and scripts
Robust communicator for technical findingsShape the future of incident management with your evaluation expertise at Rootly.#J-18808-Ljbffr

📌 Rootly Ai Evaluation Expert Toronto
🏢 XYZ Venture Capital
📍 Toronto