Natural Language Processing and Text Mining

Guide to AI and Data Courses · Part II: Branches of AI
Text preprocessingText representationLanguage modelsTransformersText miningNeighboring courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How does a computer analyze, understand and generate human language?

Natural language processing studies how computers analyze, understand and generate human language; text mining extracts information and patterns from large collections of text. At the heart of the course are text representation, which turns text into numbers, and language models that predict the next word.

At a glance

Approach

linguistics, statistics and deep learning

Input

text

Output

label, translation, summary, answer

Success measure

accuracy, BLEU, human evaluation

Math load

medium

Data needs

large text corpora

Introduction

A Short History

1954

Georgetown-IBM experiment: more than sixty Russian sentences are translated into English automatically.

1966

Weizenbaum's ELIZA imitates a therapist with simple pattern matching.

1990s

Statistical methods replace rule-based systems; IBM's translation models lead the way.

2013

Mikolov et al.'s word2vec embeds words as vectors that reflect meaning.

2017

The transformer architecture is introduced, followed by pretrained models such as BERT and GPT.

2022

Large language models reach a mass audience through chat interfaces.

The Course

Core Concepts

Tokenization

Splitting text into word or subword units.

Morphology

Root and suffix analysis; critical in agglutinative languages like Turkish.

Bag of words

Representing text by word counts, ignoring order.

TF-IDF

High weight for words frequent in a document but rare in the corpus.

Word embeddings

Words with similar meanings lie close in vector space.

Language model

A model that predicts the probability of the next word.

Named entity recognition

Finding names of people, places and organizations in text.

Topic modeling

Finding hidden topics in documents with methods such as LDA.

The Course

A Typical Workflow

CollecttextcorpusPreprocesstokenizing,stemmingRepresentTF-IDF,embeddingsModelclassifier,transformerEvaluateF1, BLEU
Example: An online store collects product reviews, tokenizes the text and analyzes suffixes, turns the reviews into vectors with a pretrained transformer, trains a positive/negative sentiment classifier and measures it with the F1 score. The result shows which topics the complaints cluster around.

The Course

What Does the Course Cover?

  1. Regular expressions, tokenization, normalization
  2. Morphology: stemming and morphological analysis
  3. N-gram language models and smoothing
  4. Text classification: naive Bayes, logistic regression
  5. Vector semantics: TF-IDF and word2vec
  6. Part-of-speech tagging and hidden Markov models
  7. Named entity recognition and sequence labeling (CRF)
  8. Syntactic parsing: dependency trees
  9. Language modeling with recurrent networks
  10. Transformers, BERT and pretraining
  11. Machine translation and evaluation with BLEU
  12. Topic modeling (LDA) and information extraction

Skills you will gain

text processing with spaCy and NLTKcleaning text with regular expressionsTurkish morphology with Zemberekword embeddings with Gensimfine-tuning a Hugging Face modelevaluating with F1 and BLEU

A typical semester; the order and coverage vary between universities.

Try It

Is the Customer Review Positive?

An online store gets thousands of reviews a day; negative ones should go straight to customer service. Pick a sample review or write your own; the model scores the words one by one.

The computer first splits the text into words, then gives each word a weight. A word like “not” or “didn't” flips the meaning of what follows; language processing has to teach the model this structure.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Artificial Intelligencelanguage understanding and generation
Programmingstring handling, file reading and regular expressions in Python
Data Structuresdictionaries, hash tables and tries: for storing a vocabulary

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you process strings in Python and search text with a regex?An intro programming course, then a short regex tutorial
Can you count word frequencies using a dictionary (hash table)?Hash tables from a data structures course
Can you explain conditional probability and Bayes' rule?Probability and Statistics
Can you evaluate a classifier with precision, recall and F1?Machine Learning
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
subfieldThis course is a subfield of Artificial Intelligencelanguage understanding and generation
shares methodsShares methods with Deep Learningword embeddings, transformer architecture
shares methodsShares methods with Probabilistic Graphical Modelshidden Markov model, conditional random field
shares methodsShares methods with Generative Modelstext generation
shares methodsShares methods with Speech Recognition and Audio Processinglanguage model, speech to text
shares methodsShares methods with Data Miningtext classification, topic modeling
shares methodsShares methods with Information Retrieval and Recommender Systemstext representation, TF-IDF, semantic search

Differences

Where It Differs (1/2)

CriterionNLPSpeech and AudioIR & Recommenders
Core questionHow does a computer analyze, understand and generate human language?How are words, speakers and events extracted from a sound wave?How are the most relevant documents or products picked from millions of candidates?
Approachlinguistics, statistics and deep learningacoustic signal and language modelmatching and ranking
Inputtextaudio waveformquery, documents, user history
Outputlabel, translation, summary, answertext, speaker, sound eventranked result list
Success measureaccuracy, BLEU, human evaluationword error rate (WER)precision, recall, NDCG
Math loadmediummedium-highmedium: vector spaces
Data needslarge text corporamany hours of recordingslots of interaction data
  • Speech and Audio: Speech recognition goes from the sound wave to text; NLP analyzes meaning once the text is available.
  • IR & Recommenders: Information retrieval fetches the documents most relevant to a query; NLP analyzes and generates the language inside them in detail.

Differences

Where It Differs (2/2)

CriterionNLPData MiningGenerative Models
Core questionHow does a computer analyze, understand and generate human language?How are hidden, useful patterns found in large piles of data?How does a model generate new samples that look like real data?
Approachlinguistics, statistics and deep learningdiscovery-driven analysislearning the data distribution
Inputtextlarge tables or transaction datahuge, mostly unlabeled data
Outputlabel, translation, summary, answerrules, clusters, outliersnew samples: images, text, audio
Success measureaccuracy, BLEU, human evaluationinterestingness, support, confidencerealism and diversity (FID etc.)
Math loadmediummediumhigh
Data needslarge text corporalargevery large
  • Data Mining: Data mining looks for patterns in any kind of data; this course handles the structure specific to language (words, suffixes, sentences).
  • Generative Models: Generative models produce all kinds of samples, including images and audio; NLP also covers understanding and analysis tasks.

Practice

Applications

Machine translation

Automatic translation between languages.

Sentiment analysis

Finding the attitude in reviews and posts.

Question answering

Finding answers in documents or a knowledge base.

Summarization

Making a long text short and to the point.

Chat assistants

Systems that talk with users in natural language.

Document classification

Sorting news, petitions or emails by topic.

Practice

Risks and Ethics

Hallucination

Language models fluently invent sources that do not exist; in the US, lawyers have filed briefs citing case law a chatbot made up.

Stereotypes in embeddings

Embeddings produce analogies like "man is to programmer as woman is to homemaker"; translation renders neutral pronouns by stereotype.

Low-resource languages

Models work best in English; on Turkish and languages with little data the same system makes noticeably more errors.

Surveillance through text

Emails, messages and posts are personal data under GDPR and similar laws; text mining can also be used to monitor and profile people.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

NLTKspaCyHugging Face TransformersGensimStanzaZemberek (Turkish)

Well-known datasets

Penn TreebankCoNLL-2003SQuADGLUEIMDb movie reviewsWikipedia dumps

Practice

Common Misconceptions

Common beliefIn fact
The model understands text like a human.Language models learn statistical patterns; fluent text is not always correct or consistent.
Any method built for English works for Turkish.In agglutinative Turkish one root takes hundreds of forms; tokenization and morphology need separate care.
Large language models ended classic NLP.TF-IDF, regular expressions and small models are still fast, cheap and enough for most jobs.
Text mining is just counting words.Counting is the start; extracting entities, relations and topics requires language structure.

Next Steps

After This Course

Courses that build on or extend this one

This course is mostly an end point: it builds on the others.

Related careers

NLP engineerComputational linguistData scientistSearch and recommendation engineerResearcher
After NLP you can move to spoken language with speech recognition, to semantic search with information retrieval, or to the inner workings of large language models with generative models. The Explainable AI and AI Ethics course covers the bias and reliability problems of language models.

Next Steps

Further Reading

  • Jurafsky and Martin, Speech and Language Processing, 2nd ed., Prentice Hall, 2009.
  • Manning and Schütze, Foundations of Statistical Natural Language Processing, MIT Press, 1999.
  • Bird, Klein and Loper, Natural Language Processing with Python, O'Reilly, 2009.
  • Eisenstein, Introduction to Natural Language Processing, MIT Press, 2019.
  • Vaswani et al., "Attention Is All You Need", Advances in Neural Information Processing Systems 30, 2017.

The End

Chapter 6: Natural Language Processing and Text Mining
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.