Natural Language Understanding with Deep Learning / Computational Semantics

Description

The field of natural language processing (NLP) has seen multiple paradigm shifts over decades, from symbolic AI to statistical methods to deep learning. We review this shift through the lens of natural language understanding (NLU), a branch of NLP that deals with “meaning”. We start with what is meaning and what does it mean for a machine to understand language? We explore how to represent the meaning of words, phrases, sentences and discourse. We then dive into many useful NLU applications.

Throughout the course, we take several concepts in NLU such as meaning or applications such as question answering, and study how the paradigm has shifted, what we gained with each paradigm shift, and what we lost? We will critically evaluate existing ideas and try to come up with new ideas that challenge existing limitations. We will particularly work on making deep learning models for language more robust.

This course will also delve into how large language models like ChatGPT are built, and latest advances on how to train/use these models for the task at hand.

Prerequisites

You are expected to have done one of the following courses at McGill: natural language processing (COMP/LING 550) or computational linguistics (COMP/LING 445) or applied machine learning (COMP 551) or From Language to Data Science (COMP/LING 345). Make sure you are comfortable with advanced Python programming. If you have done similar courses at other universities, feel free to take the course. If you are not sure, email the instructor.

Grading

Assignments (45%): Automatic grading + written report

  1. Basics of deep learning and neural networks for NLP (10%)
  2. word2vec and word representation (10%)
  3. Char-RNN and ELMo (12.5%)
  4. Transformers and applications (12.5%)

Project (50%): You will do a project in groups of two.

  • Proposal (10%)
  • Presentation (10%)
  • Final report (30%)

Participation (5%): Class participation amongst other things. Details to be determined.

Topics (Tentative)

Topic Subtopics      
Word meaning distributional semantics word embeddings evaluation  
Phrase and sentence meaning logical representation sentence embeddings evaluation  
Meaning in context word senses contextual word embeddings fine-tuning  
Interpretability feature-based vs deep learning models linguistic tests probing  
Compositionality syntax and semantic interfaces inductive priors tests for compositionality limitations
Reasoning inference question answering other applications  
Discourse conversational systems      
Language and physical world model-theoretic semantics grounded environment reinforcement learning  
Bias word association tests probing    

Schedule

Lecture Date Topic Due dates Additional Readings
1 Jan 4 Course outline and perceptron    
2 Jan 9 Multi-layer perceptron and Deep Neural Networks Jan 7: Assignment 1 release  
3 Jan 11 Word Embeddings   Distributed Representations of Words and Phrases and their Compositionality Neural Word Embedding as Implicit Matrix Factorization
4 Jan 16 Word embeddings (cont..), Bias (McGill NLP: Nicolas Meade)   Enriching Word Vectors with Subword Information Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings Semantics derived automatically from language corpora contain human-like biases
5 Jan 18 Sentence Representations (CNN, RNN, LSTMs)    
6 Jan 23 LSTMs, Back propagation through time (McGill NLP: Arisides Milios)    
7 Jan 25 ELMo, Attention    
8 Jan 30 Transformers and Pretraining Jan 27: Assignment 1 due; Assignment 2 release  
9 Feb 1 Analyzing large-scale models for syntax, Structure Prediction    
10 Feb 6 Large Language Models   Coreference Resolution: Survey End-to-end Neural Coreference Resolution Improving Machine Learning Approaches to Coreference Resolution
11 Feb 8 Large Language Models    
12 Feb 13 Prompting, Incontext learning, Parameter efficient fine tuning (McGill NLP: Arkil Patel) Feb 12: Assignment 2 due; Assignment 3 release  
13 Feb 15 Efficient incontext learning (McGill NLP: Aristides Milios, virtual)    
14 Feb 20 Efficient training and inference methods   Building Machine Translation Systems for the Next Thousand Languages Quality at a glance: An audit of web-crawled multilingual datasets
15 Feb 22 Question Answering, Retrieval-augmented Language Models and Information Retrieval    
16 Feb 27 Guest Speaker: Julia Kruetzer Feb 26: Assignment 3 due; Assignment 4 release  
17 Feb 29 Vision and Language (McGill NLP: Benno Krojer)    
18 Mar 5 Reading week    
19 Mar 7 Reading week    
20 Mar 12 Vision and Language (McGill NLP: Benno Krojer) Mar 11: Project proposal deadline  
21 Mar 14 Alignment, RLHF, DPO (McGill NLP: Amirhossein Kazemnejad)    
22 Mar 19 Retrieval-augmented QA Mar 18: Assignment 4 due  
23 Mar 21 Guest Speaker: Hinrich Schuetze    
24 Mar 26 Guest Speaker: Dzmitry Bahdanau (Human-Machine Interaction through language)    
25 Mar 28 Guest Speaker: Dzmitry Bahdanau (Human-Machine Interaction through language)    
26 Apr 2 Project presentations    
27 Apr 4 Project presentations    
28 Apr 9 Project presentations Apr 5: Final project report submission  

FAQs

Question: What are some books for learning basics of linguistics?
Bender 2013: Linguistic Fundamentals for Natural Language Processing (login with McGill credentials for free access))

Question: What are some books books on deep learning for NLP?
Jurafsky and Martin 2019: Speech and Language Processing
Tunstall et al. 2022: Natural Language Processing with Transformers (login with McGill credentials)
Eisenstein 2019: Introduction to Natural Language Processing
Goldberg 2017: Neural Network Methods for Natural Language Processing (login with McGill credentials)

Language of Submission

In accord with McGill University’s Charter of Students’ Rights, students in this course have the right to submit in English or in French any written work that is to be graded.

Academic Integrity

McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the Code of Student Conduct and Disciplinary Procedures (see www.mcgill.ca/students/srr/honest/ for more information)

Inclusivity

As the instructor of this course I endeavor to provide an inclusive learning environment. However, if you experience barriers to learning in this course, do not hesitate to discuss them with me or the Office for Students with Disabilities.