Product Quality Engineer (AI & Agentic Systems)

About This Role
Collective OS needs a product-minded engineer who can turn business intent into executable evidence. You will define what good behavior means, create realistic personas and scenarios, build automation, and evaluate agent outputs where exact-text assertions are not sufficient. This is not a downstream manual-testing function. It is a hands-on engineering role that reports into the CTO organization and works as an embedded partner to Product, Design, and Engineering from discovery through release and production learning.

What You Will Own

- Translate product decisions, business rules, and agent specifications into clear behavioral contracts and acceptance criteria.

- Create representative firm and operator personas, together with their contacts, relationship networks, target organizations, integrations, consent states, and data conditions.

- Build and maintain critical-path automation across browser, API, worker, integration, and data layers, including Playwright-based journeys.

- Design AI evaluation suites that combine deterministic checks, structured rubrics, repeated trials, calibrated graders, and periodic human review.

- Test signal detection, relationship-sensitive routing, scoring, recommendations, grounding, memory, permissions, and graceful failure across realistic and adversarial conditions.

- Define release evidence and production quality signals; turn operator corrections and escaped failures into durable regression cases.

- Improve the shared simulation, scenario, trace, and evaluation tooling used by Product and Engineering.

What Success Looks Like

Quality becomes an observable product discipline rather than a final approval gate.

- High-value journeys and high-risk behavioral boundaries are covered by reusable personas, scenarios, and gold datasets.

- Model, prompt, routing, scoring, and data-source changes are evaluated through measurable evidence rather than isolated demos or intuition.

- Failures are diagnosable across ingestion, detection, routing, agent decision, persistence,



and presentation.

- Release decisions include concise evidence about coverage, quality deltas, known risks, and explicit exceptions.

- Production feedback becomes a fast learning loop for the product and a permanent part of the regression system.

What We Are Looking For

- Robust experience in software quality, test engineering, product engineering, developer productivity, or AI evaluation for complex production systems.

- Hands-on experience evaluating LLM applications or agentic systems, including tool use, retrieval, grounding, memory, and non-deterministic outputs.

- Strong TypeScript and/or Python skills and experience building maintainable test or evaluation infrastructure.

- Practical depth in Playwright or equivalent browser automation, plus API, integration, contract, worker, and data-pipeline testing.

- Ability to design gold datasets, rubrics, sampling strategies, automated graders, and human calibration workflows.

- Strong product and business judgment: you can decide whether an output is useful to an agency owner, not only whether it is syntactically valid.

- Comfort diagnosing asynchronous and distributed systems with retries, partial failure, eventual consistency, and multiple data stores.

- Clear cross-functional communication and the ability to create alignment in ambiguous, fast-moving product work.

Helpful Experience

- B2B SaaS, sales or relationship intelligence, professional services, graph analytics, recommendation systems, or calibrated scoring.

- Establishing a quality or evaluation discipline in an early-stage company and mentoring others in quality practices.

- Model and prompt versioning, offline and online evaluation, observability, and cost-quality tradeoffs.

FIRST 90 DAYS

Map the current quality surface and baseline the highest-risk flows; establish the first canonical persona,

scenario, and gold-dataset library; introduce repeatable AI evaluations and release evidence; define the

production feedback-to-regression loop and next-stage roadmap.

Pay: $30.00-$40.00 per hour

Benefits:

- Casual dress
- Company events
- Work from home

Work Location: Remote

📌 Product Quality Engineer (AI & Agentic Systems) (Canada)
🏢 Collective OS
📍 Canada