Balancing between Efficiencies, Modularity, and Performance of (L)LMs

Prof. Ivan Vulić

University of Cambridge and PolyAI

The NLP Reading Group is delighted to receive Prof. Ivan Vulić who will be giving a talk in-person on “Balancing between Efficiencies, Modularity, and Performance of (L)LMs”.

In addition to his talk, Professor Vulić will be at Mila and available to chat in person after his talk. Further details as well as the sign-up form can be found in the email announcing the talk.

Prof. Vulić’s visit is funded by IVADO under the R3AI initiative.

Talk Description

Following the rapid increase in size of deep learning models and large language models (LLMs) in particular, there is a pressing and crucial need for parameter-efficient and modular learning strategies that can simultaneously deal with 1) such large model sizes and 2) the lack of annotated data for many tasks, modalities, domains, and languages, and which 3) enable the creation of composable and reusable model components. At the same time, 4) innovations on the modularity and efficiency fronts should not compromise task performance of LLMs. In this talk, I will thus reflect on the balance between performance and various types of efficiency (e.g., memory-, sample-, parameter-efficiency) through the optics of modular and composable learning. I will provide a brief overview of some of the recent work from my research group aiming to optimise this balance through 1) automatic module configuration search for optimising the parameter efficiency-performance Pareto front; 2) learning composable sparse subnetworks for smaller-scale language models and then scaling them in a memory-efficient way to large language models; and 3) zero-shot and few-shot tokenizer transfer as a prerequisite for improved model merging.

Speaker Bio

Ivan Vulić is a Principal Research Associate (grade equivalent to Associate Professor) and a Royal Society University Research Fellow in the Language Technology Lab, University of Cambridge. He is also a Senior Scientist at PolyAI. He is a member of the Steering Committee of the newly established Centre for Human Inspired Artificial Intelligence (CHIA) at Cambridge. His core expertise and interests are in cross-lingual, multilingual and multi-modal representation learning, modularity and composability of Machine Learning (ML) models, sample-efficient, parameter-efficient and few-shot ML, transfer learning for enabling responsible cross-lingual NLP applications for low-resource languages and dialects, multi-modal semantics, conversational AI and multilingual dialogue modeling , cross-lingual, multilingual and multi-modal information search, etc. He has published numerous papers at top-tier NLP and IR conferences and journals, and his research work also resulted in several best paper awards. He serves as an area chair and regularly reviews for all major NLP and Machine Learning conferences and journals. Ivan has given numerous invited talks at academia and industry, and co-organised a number of NLP conferences and workshops. Ivan holds a PhD in Computer Science from KU Leuven awarded summa cum laude. In 2021 he was awarded the annual Karen Spärck Jones Award from the British Computing Society for his research contributions to Natural Language Processing and Information Retrieval.

Logistics

Date: July 29th
Time: 11:00AM
Location: Auditorium 1 or via Zoom (See email)