Priority may be given to the following designated employment equity groups: women, Indigenous Peoples* (First Nations, Inuit and Métis), persons with disabilities and racialized persons*.

* The Employment Equity Act, which is under review, uses the terminology Aboriginal peoples and visible minorities.

Candidates are asked to self-declare when applying to this hiring process.

City:Ottawa

OrganizationalUnit:Digital Technologies

Classification:RO

Tenure:Term

Duration: 2 years

Language Requirements:English

Work arrangements:

- Due to the nature of the work and operational requirements, this position may be eligible for a limited hybrid work arrangement (combination of working onsite and telework).

At the NRC, we recognize that Indigenous candidates may have important connections to their communities and you may be eligible for an exception to this work arrangement. Alternative work arrangements may also be considered to accommodate candidates as required. To learn more about these options, please contact the NRC Hiring team using the contact information below.

The program

The NRC’s Postdoctoral Fellowship program offers PDFs access to unique world-class facilities and the opportunity to work alongside multi-disciplinary teams of expert researchers and technicians on projects of critical importance to Canada. PDFs will carry out research on innovative research projects, with opportunities for career development (publications and/or industry interaction).

PDFs will be offered appointments to the staff of the National Research Council on a term basis and will be offered salaries and advantages currently available to Research Officers. Salaries commensurate with experience are taxable and subject to other deductions. As a guide,



the current annual PhD recruiting rate is $83,985. In addition PDFs will be allocated a travel fund to support their work.

Project title

Uncertainty Quantification and Calibration for Large Language Models

The role

Are you looking for an opportunity to conduct cutting-edge research with real-life impacts? Do you want to grow in a team environment that provides guidance and autonomy? Do you want ownership over outcomes allowing you to present your work at competitive international venues? Look no further we have a rewarding opportunity for you

Current human-AI interaction is plagued by over-reliance on AI predictions, an inability todistinguish AI-generated content, and the risks of AI-generated falsehoods and deceptions. Quantifying the uncertainty in AI generations is a pathway to mitigating all of these risks.

Traditionally in AI research, a well-designed certainty measure assigns a probability of correctness to each generated output. Combining AI outputs with these interpretable confidence scores is immensely valuable for informed human-AI interaction. In the new era of generative large language models (LLMs), uncertainty quantification becomes less well-defined due to the open-ended nature of LLM outputs (i.e., it is necessary to simultaneously estimate the uncertainty in what is being said, and how it is being said,



while dealing with an infinite output space). However, LLMs are now the backbone of most human-facing AI systems. How to meaningfully capture these uncertainties to provide useful context on LLM generations is an active research question.

Separately, recent work suggests that LLMs represent human-interpretable concepts (e.g., emotions, truthfulness, harmfulness, etc.) at intermediate points in their computation. This opens up a new opportunity to inspect uncertainty in the LLMs’ encoding of these concepts, which would enable new insights and intervention mechanisms. To date, we are not aware of any work that applies certainty calibration methods to intermediate model representations.

This project develops methods for uncertainty quantification that can be applied to both output generations, and intermediate features. Foundational work on this subject is both technically challenging and highly applicable to a wide range of open problems. The anticipated research outcomes are appropriate for publication in top-tier international venues for Computational Linguistics and Machine Learning research, such as ACL* conferences, ICLR, or ICML.

The Research Problem

Large language models produce text by iteratively generating a probability distribution over possible next-word choices, then sampling from that distribution. The problem of uncertainty quantification aims to consolidate that probability distribution into a few meaningful and interpretable certainty metrics. Certainty metrics can answer questions about reproducibility (i.e., if asked the same thing again, how likely is the LLM to respond in the same way?), or embedded knowledge (i.e

📌 Postdoctoral Fellow (Ottawa)
🏢 Aimlroles
📍 Ottawa