Speech Recognition and Audio Processing
Introduction
What Question Does It Ask?
Speech recognition and audio processing study how to analyze the sound wave from a microphone and extract words, the speaker's identity or events in the environment. Sound is first converted to time-frequency features; the acoustic model and the language model together find the most likely text.
At a glance
acoustic signal and language model
audio waveform
text, speaker, sound event
word error rate (WER)
medium-high
many hours of recordings
Introduction
A Short History
Bell Labs' Audrey system recognizes digits spoken by a single speaker.
IBM Shoebox is demonstrated, recognizing sixteen English words and the digits.
In DARPA-funded projects, Carnegie Mellon's Harpy system recognizes about a thousand words.
Hidden Markov models (HMMs) become the standard method of speech recognition.
Deep neural networks begin replacing Gaussian mixtures in acoustic modeling.
OpenAI's Whisper is trained end to end on about 680,000 hours of multilingual data.
The Course
Core Concepts
Measuring the sound wave thousands of times per second.
A time-frequency image from the short-time Fourier transform.
Features adapted to how the human ear perceives frequency.
Predicts the sound unit (phoneme) from a stretch of audio.
Says which word sequence is more likely.
Finding the most likely word sequence with beam search.
Verifying or identifying whose voice it is.
WER: the rate of insertion, deletion and substitution errors.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Speech production and perception; phonemes
- Digital audio: sampling, framing, windowing
- Spectrograms, the mel scale and MFCCs
- Hidden Markov models and Gaussian mixtures
- Decoding with Viterbi and beam search
- N-gram and neural language models
- Deep acoustic models
- End-to-end recognition: CTC, attention, transducers
- Evaluation: word error rate
- Speaker recognition and verification
- Noise robustness and speech enhancement
- Speech synthesis (TTS)
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
A Smart Home Recognizes Commands
A smart home assistant must recognize four voice commands: yes, no, stop, start. Pick a command, change the speaker, raise the noise; the system compares the sound with stored templates.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Digital Signal Processing | spectrogram, MFCC, filters |
| Calculus | complex numbers, integrals and an introduction to Fourier series |
| Programming | array operations and reading audio from files in Python |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you explain how the Fourier transform splits a signal into frequencies? | Digital Signal Processing |
| Can you use conditional probability and Bayes' rule? | Probability and Statistics |
| Can you explain how a simple neural network is trained? | Artificial Neural Networks |
| Can you read an audio file in Python and plot its waveform? | An intro programming course, then a short NumPy and librosa tutorial |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| builds on | This course builds on Digital Signal Processing | spectrogram, MFCC, filters |
| shares methods | Shares methods with Deep Learning | end-to-end speech recognition |
| shares methods | Shares methods with Natural Language Processing and Text Mining | language model, speech to text |
Differences
Where It Differs (1/2)
| Criterion | Speech and Audio | Signal Processing | NLP |
|---|---|---|---|
| Core question | How are words, speakers and events extracted from a sound wave? | How are sampled signals analyzed, filtered and transformed? | How does a computer analyze, understand and generate human language? |
| Approach | acoustic signal and language model | mathematical transforms and filters | linguistics, statistics and deep learning |
| Input | audio waveform | sampled signal | text |
| Output | text, speaker, sound event | filtered signal, spectrum | label, translation, summary, answer |
| Success measure | word error rate (WER) | signal-to-noise ratio, frequency response | accuracy, BLEU, human evaluation |
| Math load | medium-high | high: complex numbers, Fourier | medium |
| Data needs | many hours of recordings | no learning; the signal suffices | large text corpora |
- Signal Processing: Signal processing filters and transforms any kind of signal; this course focuses on features and models specific to audio and speech.
- NLP: NLP assumes it already has text; this course obtains that text from the sound wave and also handles audio events.
Differences
Where It Differs (2/2)
| Criterion | Speech and Audio | Deep Learning | Biometrics |
|---|---|---|---|
| Core question | How are words, speakers and events extracted from a sound wave? | How do many-layered networks learn useful representations from raw data? | How is a person recognized from their face, fingerprint or voice? |
| Approach | acoustic signal and language model | multilayer representation learning | matching and verification |
| Input | audio waveform | raw data: images, text, audio | face, fingerprint, iris, voice |
| Output | text, speaker, sound event | learned representation and prediction | identity decision |
| Success measure | word error rate (WER) | accuracy, loss, scaling | FAR, FRR, EER |
| Math load | medium-high | high: derivatives, matrices, probability | medium |
| Data needs | many hours of recordings | large; needs GPUs | few samples per person |
- Deep Learning: Deep learning offers general network architectures; this course adapts them to the time-frequency structure of sound and the alignment problem.
- Biometrics: Biometrics uses the voice as proof of identity, like a face or fingerprint; this course also analyzes what is said.
Practice
Applications
Understanding commands to control a phone or home.
Turning video, meeting and lecture recordings into text.
Analyzing and routing calls.
Voice-based banking and access control.
Detecting breaking glass, sirens, coughs.
Turning text into natural-sounding speech.
Practice
Risks and Ethics
A voice imitated from a few seconds of recording is used in fraud and to defeat voice authentication.
Devices waiting for a wake word can record by mistake, and a voice also carries health and emotion information.
Error rates are clearly higher for some accents and dialects, so these speakers gain less from captions and assistants.
End-to-end models can write sentences nobody said during silence or noise, which is dangerous in medical transcription.
In depth: Explainable AI and AI Ethics.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Speech recognition maps sound directly to words. | The same word sounds very different across accents, speeds and noise; the language model resolves the ambiguity. |
| The louder the recording, the better. | Clipping and reverberation hurt recognition; a clean, balanced recording is what matters. |
| A model that works well in English works in Turkish too. | Each language needs enough data; agglutinative Turkish also has a vocabulary problem to solve. |
| Audio processing is only about speech. | Music, environmental sounds and medical sounds (heart, cough) are also part of the course. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Rabiner and Juang, Fundamentals of Speech Recognition, Prentice Hall, 1993.
- Huang, Acero and Hon, Spoken Language Processing, Prentice Hall, 2001.
- Jurafsky and Martin, Speech and Language Processing, 2nd ed., Prentice Hall, 2009.
- Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition", Proceedings of the IEEE 77(2), 1989.