Natural Language Processing and Text Mining
Introduction
What Question Does It Ask?
Natural language processing studies how computers analyze, understand and generate human language; text mining extracts information and patterns from large collections of text. At the heart of the course are text representation, which turns text into numbers, and language models that predict the next word.
At a glance
linguistics, statistics and deep learning
text
label, translation, summary, answer
accuracy, BLEU, human evaluation
medium
large text corpora
Introduction
A Short History
Georgetown-IBM experiment: more than sixty Russian sentences are translated into English automatically.
Weizenbaum's ELIZA imitates a therapist with simple pattern matching.
Statistical methods replace rule-based systems; IBM's translation models lead the way.
Mikolov et al.'s word2vec embeds words as vectors that reflect meaning.
The transformer architecture is introduced, followed by pretrained models such as BERT and GPT.
Large language models reach a mass audience through chat interfaces.
The Course
Core Concepts
Splitting text into word or subword units.
Root and suffix analysis; critical in agglutinative languages like Turkish.
Representing text by word counts, ignoring order.
High weight for words frequent in a document but rare in the corpus.
Words with similar meanings lie close in vector space.
A model that predicts the probability of the next word.
Finding names of people, places and organizations in text.
Finding hidden topics in documents with methods such as LDA.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Regular expressions, tokenization, normalization
- Morphology: stemming and morphological analysis
- N-gram language models and smoothing
- Text classification: naive Bayes, logistic regression
- Vector semantics: TF-IDF and word2vec
- Part-of-speech tagging and hidden Markov models
- Named entity recognition and sequence labeling (CRF)
- Syntactic parsing: dependency trees
- Language modeling with recurrent networks
- Transformers, BERT and pretraining
- Machine translation and evaluation with BLEU
- Topic modeling (LDA) and information extraction
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Is the Customer Review Positive?
An online store gets thousands of reviews a day; negative ones should go straight to customer service. Pick a sample review or write your own; the model scores the words one by one.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Artificial Intelligence | language understanding and generation |
| Programming | string handling, file reading and regular expressions in Python |
| Data Structures | dictionaries, hash tables and tries: for storing a vocabulary |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you process strings in Python and search text with a regex? | An intro programming course, then a short regex tutorial |
| Can you count word frequencies using a dictionary (hash table)? | Hash tables from a data structures course |
| Can you explain conditional probability and Bayes' rule? | Probability and Statistics |
| Can you evaluate a classifier with precision, recall and F1? | Machine Learning |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| subfield | This course is a subfield of Artificial Intelligence | language understanding and generation |
| shares methods | Shares methods with Deep Learning | word embeddings, transformer architecture |
| shares methods | Shares methods with Probabilistic Graphical Models | hidden Markov model, conditional random field |
| shares methods | Shares methods with Generative Models | text generation |
| shares methods | Shares methods with Speech Recognition and Audio Processing | language model, speech to text |
| shares methods | Shares methods with Data Mining | text classification, topic modeling |
| shares methods | Shares methods with Information Retrieval and Recommender Systems | text representation, TF-IDF, semantic search |
Differences
Where It Differs (1/2)
| Criterion | NLP | Speech and Audio | IR & Recommenders |
|---|---|---|---|
| Core question | How does a computer analyze, understand and generate human language? | How are words, speakers and events extracted from a sound wave? | How are the most relevant documents or products picked from millions of candidates? |
| Approach | linguistics, statistics and deep learning | acoustic signal and language model | matching and ranking |
| Input | text | audio waveform | query, documents, user history |
| Output | label, translation, summary, answer | text, speaker, sound event | ranked result list |
| Success measure | accuracy, BLEU, human evaluation | word error rate (WER) | precision, recall, NDCG |
| Math load | medium | medium-high | medium: vector spaces |
| Data needs | large text corpora | many hours of recordings | lots of interaction data |
- Speech and Audio: Speech recognition goes from the sound wave to text; NLP analyzes meaning once the text is available.
- IR & Recommenders: Information retrieval fetches the documents most relevant to a query; NLP analyzes and generates the language inside them in detail.
Differences
Where It Differs (2/2)
| Criterion | NLP | Data Mining | Generative Models |
|---|---|---|---|
| Core question | How does a computer analyze, understand and generate human language? | How are hidden, useful patterns found in large piles of data? | How does a model generate new samples that look like real data? |
| Approach | linguistics, statistics and deep learning | discovery-driven analysis | learning the data distribution |
| Input | text | large tables or transaction data | huge, mostly unlabeled data |
| Output | label, translation, summary, answer | rules, clusters, outliers | new samples: images, text, audio |
| Success measure | accuracy, BLEU, human evaluation | interestingness, support, confidence | realism and diversity (FID etc.) |
| Math load | medium | medium | high |
| Data needs | large text corpora | large | very large |
- Data Mining: Data mining looks for patterns in any kind of data; this course handles the structure specific to language (words, suffixes, sentences).
- Generative Models: Generative models produce all kinds of samples, including images and audio; NLP also covers understanding and analysis tasks.
Practice
Applications
Automatic translation between languages.
Finding the attitude in reviews and posts.
Finding answers in documents or a knowledge base.
Making a long text short and to the point.
Systems that talk with users in natural language.
Sorting news, petitions or emails by topic.
Practice
Risks and Ethics
Language models fluently invent sources that do not exist; in the US, lawyers have filed briefs citing case law a chatbot made up.
Embeddings produce analogies like "man is to programmer as woman is to homemaker"; translation renders neutral pronouns by stereotype.
Models work best in English; on Turkish and languages with little data the same system makes noticeably more errors.
Emails, messages and posts are personal data under GDPR and similar laws; text mining can also be used to monitor and profile people.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| The model understands text like a human. | Language models learn statistical patterns; fluent text is not always correct or consistent. |
| Any method built for English works for Turkish. | In agglutinative Turkish one root takes hundreds of forms; tokenization and morphology need separate care. |
| Large language models ended classic NLP. | TF-IDF, regular expressions and small models are still fast, cheap and enough for most jobs. |
| Text mining is just counting words. | Counting is the start; extracting entities, relations and topics requires language structure. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Jurafsky and Martin, Speech and Language Processing, 2nd ed., Prentice Hall, 2009.
- Manning and Schütze, Foundations of Statistical Natural Language Processing, MIT Press, 1999.
- Bird, Klein and Loper, Natural Language Processing with Python, O'Reilly, 2009.
- Eisenstein, Introduction to Natural Language Processing, MIT Press, 2019.
- Vaswani et al., "Attention Is All You Need", Advances in Neural Information Processing Systems 30, 2017.