LLMs/VLMs with Nonparametric Memory: Experiments on Query Generation and Visual Question Answering

Amal Zouaq

Polytechnique Montréal

The NLP Reading Group is excited to host Amal Zouaq, from Polytechnique Montréal, who will be speaking in person in A14 at 1PM on Friday February 28th about LLMs/VLMs and Nonparametric Memory.

Talk Description

LLMs and VLMs have shown tremendous progress in visual question answering and knowledge base question answering. However, they often rely on their parametric memory to generate answers or queries, which limits robustness and generalization (e.g., when questions use different templates or URIs in the test set), and leads to URI hallucinations (Diallo et al., 2024). This talk explores the limitations of LLMs in query generation and demonstrates how nonparametric memories can improve robustness to out-of-distribution evaluation and mitigate URI hallucinations. One of the works in our lab, PGMR (Sharma et al., 2025a), drastically reduces URI hallucinations while significantly improving performance across both LLMs and datasets. We also extend the concept of nonparametric memories to visual question answering. Here, we show how a modular code generation approach coupled with a nonparametric memory, in the form of a function library, enhances the geometric reasoning capabilities of VLMs, yielding an average improvement of over 16% across various question complexities on the GeomVerse dataset (Sharma et al., 2025b).

Speaker Bio

Amal Zouaq is a Full Professor at Polytechnique Montreal, an Associate Academic Member at MILA, an IVADO Professor and a member of the CLIQ-AI consortium. She holds an FRQS (Dual) Chair in AI and Digital Health and is the co-leader of the IVADO R³AI Strategic Research Regroupement 3 on natural language processing. She is the director of the LAMA-WeST Lab (http://www.labowest.ca/) which focuses on challenges at the interface of natural language processing and the semantic Web, including representation learning, natural language interfaces and question answering, automated reasoning, knowledge base learning and alignment, ontology learning and modeling, and information extraction and generation.

Logistics

Date: February 28th
Time: 1PM
Location: A14 or via Zoom (See email)