Key Responsibilities

Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry backed analysis, remediation flows, and ticketing behavior.

Design evaluation suites for LLM and agentic behavior, including expected-answer checks, rubric-based grading, regression datasets, tool-call validation, and safety/approval checks.

Use or help implement evaluation frameworks such as Pydantic Evals / Pydantic AI, Strands Evals, LangSmith, DeepEval, Ragas, promptfoo, or similar tools.

Validate multi-turn support scenarios, clarification flows, knowledge retrieval, script/remediation recommendations, escalation paths, and failure handling.

Test on-device agent behavior where needed, including Windows service/tray behavior, telemetry collection, anomaly detection, local remediation handoff, logs, and resource impact.

Debug quality issues directly by reading logs, tracing requests, reproducing failures, and making small code/test changes without heavy engineering hand-holding.

Partner with engineering and product to define release gates, quality metrics, evaluation rubrics, and confidence thresholds for pilot readiness.

Contribute to CI quality checks, test fixtures, mocked integrations, regression suites, and test data management.

Identify risks in AI behavior, including hallucinated diagnosis, unsafe remediation suggestions, missing consent, weak ticket summaries, brittle tool use, and poor escalation behavior.

Responsibilities

4+ years of experience in QA engineering, SDET, test automation,



software engineering, or similar hands‑on quality roles.

Strong Python experience, including writing production‑quality tests and debugging application code.

Experience testing backend services, APIs, async workflows, integrations, logs, and distributed systems.

Demonstrated ability to work independently in a codebase, identify root causes, and make targeted fixes or test improvements.

Experience with contemporary test frameworks such as pytest, unittest, Playwright, xUnit, or similar.

Comfortable testing ambiguous AI/non‑deterministic behavior using datasets, rubrics, assertions, metrics, and regression baselines.

Ability to distinguish product bugs, prompt/model behavior issues, data issues, integration failures, and test harness problems.

Strong written communication for documenting repro steps, risks, test plans, and release readiness.

Qualifications

Direct experience with Pydantic, Pydantic AI, or Pydantic Evals.

Experience testing LLM applications, agentic systems, RAG, tool calling, MCP servers, or AI assistants.

Experience building eval datasets, LLM‑as‑judge flows, deterministic evaluators, and regression dashboards.

Windows endpoint, device telemetry, PowerShell, .NET, MQTT, or on‑device agent testing experience.

Experience with IT support workflows, ServiceNow, ticketing, endpoint management, or enterprise support tooling.

Ability to contribute small application changes, not only test changes.

#J-18808-Ljbffr

📌 AI Quality Engineer (British Columbia)
🏢 Centraprise
📍 British Columbia