Representation-level information aggregation in LLMs

Yu Bai

McGill University and Beijing Institute of Technology

The NLP Reading Group is delighted to have Yu Bai give a talk about “Representation-level information aggregation in LLMs”.

Talk Description

Large language models (LLMs) have demonstrated remarkable abilities across a variety of tasks. However, there is a lack of interpretability regarding how certain tasks are conducted using prompts and in-context demonstrations. In this talk, I will introduce our work on the phenomenon of information aggregation during the inference of LLMs, where information in most token representations is aggregated into the representations of a few tokens through the attention mechanism. I will describe the identification of this phenomenon and subsequent work that leverages this scenario to enable LLMs to process long sequences without any additional training.

Speaker Bio

Yu Bai is a fifth-year Ph.D. student at Beijing Institute of Technology and is currently a visiting student advised by Jackie Chi Kit Cheung. His research interests include inference mechanisms in large language models (e.g., in-context learning) and cross-lingual techniques.

Logistics

Date: June 17th
Time: 11:00AM
Location: F01 or via Zoom (See email)