Responsibilities

Build automated testing frameworks, validation checks, and CI/CD pipeline integrations with reproducible diagnostics.

Develop methods to explore complex software behavior, identify coverage gaps, and detect defects.

Build automated checks that validate software behavior and data against requirements.

Develop evaluation frameworks for generative AI and LLM-based systems using LLM-as-a-judge patterns, rubric-based scoring, and automated calibration.

Apply generative AI, NLP, machine learning, heuristic, deterministic, and model-based approaches to software testing problems.

Integrate testing tools into development pipelines, support delivered software, and produce reproducible diagnostics.

Prototype machine learning and LLM capabilities and collaborate with engineering, design, and quality partners on requirements, code reviews, and blockers.

Requirements

- 5+ years of professional software engineering experience, including 2+ years of hands-on experience building, integrating, or evaluating AI/ML and LLM-based systems.
- Master of Science in Computer Science, Mathematics, or a related field, or equivalent professional engineering experience.
- In-depth experience with LLMs, generative AI, and NLP concepts, architectures, and failure modes.




- Experience developing generative-system evaluation frameworks, including dataset curation, model-as-a-judge design, validation criteria, error analysis, and quantitative and qualitative metrics.
- Proficiency in Python, algorithms, data structures, modular design, automated testing, debugging, and version control.
- Experience preparing datasets, defining validation criteria, and performing error analysis using quantitative ML metrics.
- Experience with NLP or large language models, including prompting, fine-tuning, or retrieval-augmented generation.

Benefits

British Columbia compensation range of 122,300–170,700 CAD, depending on location.

Three weeks of vacation per year to start and 10 days of sick time annually.

Paid top-up to EI/QPIP benefits up to 100% of base salary for eligible maternity, parental, and adoption leave.

Extended health, dental, and vision coverage; life insurance; disability insurance; and a retirement plan for regular full time employees.

Certain roles may be eligible for bonus and other incentive programs.

Additional benefits may include healthcare coverage, mental well-being support, retirement savings, paid time off, family leaves, and complimentary games.

📌 Software Development Engineer (Full Time) (Vancouver)
🏢 Electronic Arts
📍 Vancouver