Perceptions of Linguistic Uncertainty by Language Models and Humans

Catarina Belem

UC Irvine

The NLP Reading Group is excited to have Catarina Belem, a PhD at UC Irvine, who will be discussing Perceptions of Linguistic Uncertainty by Language Models and Humans. The talk will happen on Zoom and in A14 on Friday April 25th at 1PM.

Talk Description

Uncertainty expressions such as ‘probably’ or ‘highly unlikely’ are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans quantitatively interpret these expressions, there has been little inquiry into the abilities of language models in the same context. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model’s own certainty about that statement. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI and AI-AI communication.

Paper link: paper

Speaker Bio

Catarina is a 4th year PhD Candidate in Computer Science at University of California Irvine. Advised by Padhraic Smyth and Sameer Singh, Catarina’s research has focused on the evaluation of the trustworthiness and reliability of LLMs, touching topics like fairness, hallucination, and the role between humans, LLMs, and uncertainty.

Logistics

Date: April 25th
Time: 1PM
Location: A14 or Zoom (See email)