CS 295: Statistical NLP Winter 2017

Introduction to Statistical NLP: What is NLP?; Why is it important?; Challenges for NLP; Learning goals and topics; Course logistics and information; [ Slides ]

Readings: JM Chapter 1; Science Article;

Other links: Python Numpy Tutorial; Lillian Lee's article; Jason Baldridge interview;

Text Classification: What is Classification?; Evaluation Metrics: Accuracy and F1; Statistical Signficance; Naive Bayes: Model, Estimation, and Problems; Course Project details; [ Slides ]

Readings: JM Chapter 6 (3rd ed);

Other links: Linear Algebra Overview; Review of Probability Theory; Online Sentiment Analyzer; Sentiment Analysis Survey;

Classification Contd. and (Some) Document Representation: Logistic Regression: Model, Estimation, Extensions; Introduction to Neural Networks; Document Vectors: Term-Doc Matrix, Vector Models, Cosine Distance; [ Slides ]

Readings: JM Chapter 7 (3rd ed);

Other links: Deep Learning Tutorial; Computational Linguistics and Deep Learning, Manning; L2 = Gaussian Prior; IDF Theory;

Vector Space Models: Latent Semantic Analysis; Intro to Vector Space Models; Hierarchical Brown Clustering; Skip-Gram Model (word2vec): Model, Estimation, Applications; [ Slides ]

Readings: JM Chapter 15 (3rd ed); JM Chapter 16 (3rd ed);

Other links: Blog article on Vector representations; JAIR survey (Turney and Pantel); Semi-supervised learning using word representations; word2vec Explained; word2vec as Matrix Factorization;

N-Gram Language Models: Word Embeddings: Negative Sampling, Neural View; Introduction to Language Modeling: Task and Evaluation; Generative Models: Unigram, Bigram, Trigram, and Smoothing; [ Slides ]

Readings: JM Chapter 4 (3rd ed);

Other links: Michael Collins' notes; Google Books N-Grams Viewer; Visualizing Language Models;

Discriminative Language Models: Discriminative Models: Featurized Language Models; Introduction to Recurrent Neural Networks; Variations: Stacking and Bi-directionality; Language Modeling using NNs; [ Slides ]

Readings:

Other links: DeepDrumpf; Neural Networks and Deep Learning; Chris Olah's Blog; Char-RNNs; Practical Neural Networks for NLP;

Sequence Labeling: Introduce to Tagging; Part of Speech Tagging; NB Classification for Sequences; Hidden Markov Models; Viterbi Decoding; EM Algorithm; [ Slides ]

Readings: JM Chapter 9 (3rd ed); JM Chapter 10 (3rd ed);

Other links: Collins' Notes on HMMs; Collins' Notes on Forward-backward;

Sequence Labeling Contd: Forward Backward Algorithm (HMMs); Maximum Entropy Markov Models; Greedy and Beam Search; Conditional Random Fields; Forward Backward and Viterbi for CRFs; Neural Sequence Tagging; [ Slides ]

Readings: JM Chapter 10 (3rd ed);

Other links: Collins' Notes on MEMMs and CRFs; Collins' Notes on Forward Backward; Original CRF Paper;

Syntactic Parsing: Constituents; Syntactic Parse Trees; Context Free Grammars; Chomsky Normal Form; CKY Algorithm; Evaluation; [ Slides ]

Readings: JM Chapter 11 (3rd ed); JM Chapter 12 (3rd ed);

Other links: Collins' Notes on PCFGs;

Syntactic Parsing Contd; Dependency Parsing: Probabilistic Context Free Grammars; Lexical PCFGs; Dependency Grammar; Evaluating Dependency Trees; Transition-based Inference; Graph-based Inference; Eisner Algorithm; [ Slides ]

Readings: JM Chapter 14 (3rd ed);

Other links: Online demo of Stanford parser; Collins' Notes on Lexicalized PCFGs;

Semantics: Roles and Relations: Log-linear models; Likelihood Training; Structured Perceptron; Word Senses and Disambiguation; Roles: Thematic and Semantic; Semantic Role Labeling; [ Slides ]

Readings: JM Chapter 17.1-17.4 (3rd ed); JM Chapter 22 (3rd ed);

Other links: WordNet; VerbNet; PropBank; FrameNet;

Logical Forms: Need for logical forms; Mapping language to Logic; Syntax vs Semantics; Lambda-Calculus; Limitations of Lambda-Calculus; Combinatory Categorical Grammars; CCG Types and Combinators; [ Slides ]

Readings: Mark Steedman's Tutorial;

Other links: Semantic Parsing with CCGs tutorial; Zettlemoyer paper on logical form parsing; Perci Liang paper on latent logical forms;

CCGs Contd, Information Extraction: CCGs and Lambda Calculus; CCG Modeling; Learning CCGs; [ Slides ]; What is Information Extraction; Applications of Information Extraction; Role of NLP in IE; Named Entity Recognition; Features for NER; [ Slides ]

Readings: JM Chapter 21.1 (3rd ed);

Other links: Demo of extended NER from Dan Roth;

Relation Extraction: Relation Extraction and Applications; Rule-Based Relation Extraction; Supervised Models of Relation Extraction; Distantly Supervised Relation Extraction; Unsupervised Relation Extraction; [ Slides ]

Readings: JM Chapter 21.2-6 (3rd ed);

Other links:

Machine Translation: Intro to Machine Translation; Challenges, and Rule-Based; Statistical MT; Parallel Corpora; Components of an MT System; MT Evaluation; Word Alignment Models; [ Slides ]

Readings: JM Chapter 21.2-6;

Other links: Eisenstein Ch 19.1-19.2; Collins' Notes on IBM Models;

Machine Translation Contd: EM Training for Word Alignment; Intro to Phrase-based MT; Learning Phrase Lexicons; Monotonic Word Alignment; Stack Decoding; Monotonic Phrase Decoding; [ Slides ]

Readings: Collins' Notes on Phrase-Based Translation;

Other links: Online Stack Decoder demo;

Syntax MT; Neural MT: Non-Monotonic Phrase Decoding; Hypothesis Recombination; Multi-Stack Decoding; Overview of Syntax-Based MT; Neural MT: Seq2Seq; RNNs and extensions; GRUs and LSTMs; Google's Neural MT Model; [ Slides ]

Readings: Seq2Seq Learning;

Other links: Neubig's Neural MT Tutorial; Zero shot translation by Google's NMT; Google's "Interlingua";

Coreference, Entity Linking, and QA: Intro to Coref Resolution, Applications; Winograd Schema; Intro to Pragmatics; Types of References: Names, Pronouns, Nominals; Machine Learning for Coref Resolution; Evaluation of Coref Resolution; Entity Resolution and Linking; Applications of Linking; Evaluating Entity Linking; Intro to QA; Applications of QA; [ Slides ]

Readings:

Other links:

Question Answering and Entailment: Factoid Question Answering; Overview of the Watson project; IR-Based Factoid QA: Question Processing, Passage Retrieval, Answer Processing; Answer Type Prediction; Other Extensions: AskMSR, FALCON, Allen AI Challenge; Introduction to Textual Entailment; Applications of Entailment; [ Slides ]

Readings: JM Chapter 28;

Other links:

Discourse and Summarization; Wrapup: Introduction to Discourse; Coherence vs Semantics; Coherence Indicators: Connectors, Lexical Chains, Relations; Applications of Coherence; Intro to Summarization; Types: Single vs Multiple Docs, Query-specific vs Generic, Extractive vs Abstractive; Summarization Pipeline; ROUGE Evaluation; Course Wrapup; [ Slides ]

Readings:

Other links: