Own agents, the agent builder, the evaluation builder, and agentic memory as products, including their roadmaps, sequencing, and business outcomes

Own the applied-science delivery roadmap and cross-team orchestration for the Caseware AI platform

Sequence evaluation, agentic-memory, and platform workstreams into a single delivery plan

Own the product acceptance bar for every AI feature and establish evaluation baselines

Work with domain experts and partner accounting and audit firms to validate AI outputs against professional standards

Serve as the practitioner voice and SME proxy by shadowing audit workflows, testing prototypes, and identifying actionable failure modes

Partner daily with engineering to convert problem statements into requirements and acceptance criteria

Monitor AI model developments, architectures, agent patterns, and competitor activity

Define and own product health metrics, including task completion, correction rates, override frequency, trust signals, and time-to-completion

Collaborate with applied scientists, engineers, domain experts, and external partner firms

Make autonomous product decisions and own product outcomes

Requirements

6+ years of experience in technology-focused product management or product development

3–4 years working deeply with AI or machine learning–driven products

Software engineers transitioning into product management are welcome

Proven ability to drive delivery across multiple technical teams, coordinating dependencies, shared interfaces, and timelines across applied science and platform engineering

Direct experience building and running evaluation frameworks for production AI products

Experience defining acceptance bars,



running human-in-the-loop and practitioner validation, tracking regressions across releases, and using results to decide what ships

Technical fluency in LLMs, RAG, prompt engineering, embeddings, and agentic patterns

High agency and ownership of outcomes

Intellectual curiosity about AI, including following model releases, reading research, and understanding frontier-lab developments

Experience representing the voice of customers or practitioners and working with domain experts or subject matter specialists

Experience defining quality bars in high-stakes, regulated, or professional-grade contexts

Comfort with ambiguity

Genuine curiosity about auditing and appreciation for high-quality work

Willingness to build accounting and audit domain fluency on the job

Core Competencies
Demonstrates expertise in technology-focused product management with a solid emphasis on AI and machine learning-driven products. Capable of defining product health metrics, managing cross-team orchestration, and ensuring high-quality outcomes in regulated environments.

Highest-signal resume keywords

Product Management

AI Product Development

Evaluation Frameworks

Technical Fluency in LLMs

Cross-Team Collaboration

Hard Skills

Product Roadmap Ownership

AI Feature Evaluation

Acceptance Criteria Definition

Human-in-the-Loop Validation

Prompt Engineering

Agentic Patterns

Task Completion Metrics

Correction Rates

Override Frequency

Time-to-Completion

Soft Skills

High Agency

Intellectual Curiosity

Comfort with Ambiguity

Industry Keywords

Applied Science

Audit Workflows

Professional Standards

Domain Expertise

Accounting Fluency

#J-18808-Ljbffr

📌 Technical Product Manager – AI Products (Ontario)
🏢 Jobtailor
📍 Ontario