This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test - Agentic Automation based in Canada.

This is an opportunity for an experienced SDET to help shape how AI agents are tested, evaluated, and trusted in production. You will join a platform engineering team working on a large-scale, cloud-based cybersecurity setting, building the automation that supports reliability, performance, and quality. Alongside traditional functional and integration testing, you will contribute to agentic test generation, AI-powered workflows, and eval-driven development. Your work will help ensure that AI systems behave accurately, safely, and consistently before reaching production. You will collaborate with engineers across the development lifecycle in a remote-first environment that values autonomy, experimentation, and continuous improvement.

Accountabilities

- Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
- Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
- Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
- Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
- Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
- Identify and mitigate unreliable AI outputs, including hallucinated assertions,



flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
- Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
- Test integrations across platform systems, services, data, applications, and customer-facing features.
- Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
- Share technical knowledge and contribute to team-wide quality and evaluation standards.

Requirements

- Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
- At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
- Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
- Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
- Experience testing web services and APIs, including REST, gRPC, and JSON.
- Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles,



throughput, and error rates.
- A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
- Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
- Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.

Nice to have

- End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.
- Experience with prompt engineering and designing agent context or tools.
- Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
- Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
- Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
- Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.

Benefits

- Annual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.
- Market-leading compensation and equity award opportunities.
- Comprehensive physical and mental wellness programs.
- Competitive vacation allowance and holidays.
- Paid parental and adoption leave.

#J-18808-Ljbffr

📌 Software Development Engineer in Test - Agentic Automation (Canada)
🏢 Jobgether
📍 Canada