Natural Language Understanding with Deep Learning / Computational Semantics
- Course Code: COMP 599 & LING 484/782 (Fall 2022)
- Instructor: Siva Reddy (office hours: Tuesday 12:00pm–12:45pm and 2:15pm–3:00pm, ENGMC 104N)
- Teaching Assistants: Aristides Milios, Nicholas Meade, Xing Han Lu
- Classroom: ENGMD 280, Macdonald Engineering Building
- Time: Tuesday and Thursdays, 10:05 AM to 11:25 AM
- Links: MyCourses
Description
The field of natural language processing (NLP) has seen multiple paradigm shifts over decades, from symbolic AI to statistical methods to deep learning. We review this shift through the lens of natural language understanding (NLU), a branch of NLP that deals with “meaning”. We start with what is meaning and what does it mean for a machine to understand language? We explore how to represent the meaning of words, phrases, sentences and discourse. We then dive into many useful NLU applications.
Throughout the course, we take several concepts in NLU such as meaning or applications such as question answering, and study how the paradigm has shifted, what we gained with each paradigm shift, and what we lost? We will critically evaluate existing ideas and try to come up with new ideas that challenge existing limitations. We will particularly work on making deep learning models for language more robust.
Prerequisites
You are expected to have done one of the following courses at McGill: natural language processing (COMP/LING 550) or computational linguistics (COMP/LING 445) or applied machine learning (COMP 551) or From Language to Data Science (COMP/LING 345). Make sure you are comfortable with advanced Python programming. If you have done similar courses at other universities, feel free to take the course. If you are not sure, email the instructor.
Grading
Assignments (60%): Automatic grading + written report
- Basics of deep learning and neural networks for NLP (15%)
- word2vec and word representation (15%)
- Char-RNN and ELMo (15%)
- Transformers and applications (15%)
Project (35%): You will do a project in groups of two.
- Proposal (5%)
- Presentation (5%)
- Final report (25%)
Participation (5%): Class participation amongst other things. Details to be determined.
Topics (Tentative)
| Topic | Subtopics | |||
|---|---|---|---|---|
| Word meaning | distributional semantics | word embeddings | evaluation | |
| Phrase and sentence meaning | logical representation | sentence embeddings | evaluation | |
| Meaning in context | word senses | contextual word embeddings | fine-tuning | |
| Interpretability | feature-based vs deep learning models | linguistic tests | probing | |
| Compositionality | syntax and semantic interfaces | inductive priors | tests for compositionality | limitations |
| Reasoning | inference | question answering | other applications | |
| Discourse | conversational systems | |||
| Language and physical world | model-theoretic semantics | grounded environment | reinforcement learning | |
| Bias | word association tests | probing |
Schedule
| Lecture | Date | Topic | Due dates | Additional Readings |
|---|---|---|---|---|
| 1 | Sep 1 | Course outline and perceptron | ||
| 2 | Sep 6 | Multi-layer perceptron and Deep Neural Networks | Sep7: Assignment 1 release | |
| 3 | Sep 8 | Word Embeddings | Distributed Representations of Words and Phrases and their Compositionality Neural Word Embedding as Implicit Matrix Factorization | |
| 4 | Sep 13 (add or drop) | Word embeddings (cont..), Bias | Enriching Word Vectors with Subword Information Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings Semantics derived automatically from language corpora contain human-like biases | |
| 5 | Sep 15 | Sentence Representations (CNN, RNN, LSTMs) | ||
| 6 | Sep 20 | LSTMs, Back propagation through time | ||
| 7 | Sep 22 | ELMo, Attention | ||
| 8 | Sep 27 | Transformers and Pretraining | Sep 27: Assignment 1 due; Assignment 2 release | |
| 9 | Sep 29 | Large Language Models | ||
| 10 | Oct 4 | Guest Speaker: Rahul Aralikatte (Coreference resolution) | Coreference Resolution: Survey End-to-end Neural Coreference Resolution Improving Machine Learning Approaches to Coreference Resolution | |
| 11 | Oct 6 | Analyzing large-scale models for syntax, Structure Prediction | ||
| 12 | Oct 11 | Reading week | Oct 12: Assignment 2 due; Assignment 3 release | |
| 13 | Oct 13 | No Lecture | ||
| 14 | Oct 18 | Guest Speaker: Julia Kreutzer (Machine Translation in the wild) | Building Machine Translation Systems for the Next Thousand Languages Quality at a glance: An audit of web-crawled multilingual datasets | |
| 15 | Oct 20 | Efficient training and inference methods | ||
| 16 | Oct 25 | Efficient training and inference methods | Oct 26: Assignment 3 due; Assignment 4 release | |
| 17 | Oct 27 | Question Answering and Information Retrieval | ||
| 18 | Nov 1 | Retrieval-augmented Language Models, Conversational Models | Nov 4: Project proposal deadline | |
| 19 | Nov 3 | Language Model Debiasing | ||
| 20 | Nov 8 | Guest Speaker: Dzmitry Bahdanau (Human-Machine Interaction through language) | ||
| 21 | Nov 10 | Guest Speaker: Dzmitry Bahdanau (Human-Machine Interaction through language) | ||
| 22 | Nov 15 | Language Models and Bias | Nov 18: Assignment 4 due | |
| 23 | Nov 17 | Language grounding | ||
| 24 | Nov 22 | In-class project office hours | ||
| 25 | Nov 24 | In-class project office hours | Nov 28: Project presentation video recoding and slides upload due | |
| 26 | Nov 29 | Project presentations | ||
| 27 | Dec 1 | Project presentations | Dec 5: Final project report submission |
FAQs
Question: What are some books for learning basics of linguistics?
Bender 2013: Linguistic Fundamentals for Natural Language Processing (login with McGill credentials for free access))
Question: What are some books books on deep learning for NLP?
Jurafsky and Martin 2019: Speech and Language Processing
Tunstall et al. 2022: Natural Language Processing with Transformers (login with McGill credentials)
Eisenstein 2019: Introduction to Natural Language Processing
Goldberg 2017: Neural Network Methods for Natural Language Processing (login with McGill credentials)
Language of Submission
In accord with McGill University’s Charter of Students’ Rights, students in this course have the right to submit in English or in French any written work that is to be graded.
Academic Integrity
McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the Code of Student Conduct and Disciplinary Procedures (see www.mcgill.ca/students/srr/honest/ for more information)
Inclusivity
As the instructor of this course I endeavor to provide an inclusive learning environment. However, if you experience barriers to learning in this course, do not hesitate to discuss them with me or the Office for Students with Disabilities.