Speech Recognition and Audio Processing

Guide to AI and Data Courses · Part III: Signals, Images and Audio
Digitizing soundSpectrogram and MFCCAcoustic modelLanguage modelSpeakers and eventsRelated courses
Sercan KÜLCÜ · Giresun University · Computer Engineering

Introduction

What Question Does It Ask?

How are words, speakers and events extracted from a sound wave?

Speech recognition and audio processing study how to analyze the sound wave from a microphone and extract words, the speaker's identity or events in the environment. Sound is first converted to time-frequency features; the acoustic model and the language model together find the most likely text.

At a glance

Approach

acoustic signal and language model

Input

audio waveform

Output

text, speaker, sound event

Success measure

word error rate (WER)

Math load

medium-high

Data needs

many hours of recordings

Introduction

A Short History

1952

Bell Labs' Audrey system recognizes digits spoken by a single speaker.

1962

IBM Shoebox is demonstrated, recognizing sixteen English words and the digits.

1970s

In DARPA-funded projects, Carnegie Mellon's Harpy system recognizes about a thousand words.

1980s

Hidden Markov models (HMMs) become the standard method of speech recognition.

2012

Deep neural networks begin replacing Gaussian mixtures in acoustic modeling.

2022

OpenAI's Whisper is trained end to end on about 680,000 hours of multilingual data.

The Course

Core Concepts

Sampling

Measuring the sound wave thousands of times per second.

Spectrogram

A time-frequency image from the short-time Fourier transform.

MFCC

Features adapted to how the human ear perceives frequency.

Acoustic model

Predicts the sound unit (phoneme) from a stretch of audio.

Language model

Says which word sequence is more likely.

Decoding

Finding the most likely word sequence with beam search.

Speaker recognition

Verifying or identifying whose voice it is.

Word error rate

WER: the rate of insertion, deletion and substitution errors.

The Course

A Typical Workflow

Recordmicrophone,16 kHzExtractfeaturesspectrogram,MFCCAcousticmodelphonemeprobabilityDecodelang. model,beam searchTextmeasuredby WER
Example: A voice assistant records the question "what's the weather tomorrow" at 16 kHz, computes a spectrogram from 25 ms frames, the acoustic model gives phoneme probabilities for each frame, and the language model turns them into the most likely sentence. The text is passed to NLP for meaning.

The Course

What Does the Course Cover?

  1. Speech production and perception; phonemes
  2. Digital audio: sampling, framing, windowing
  3. Spectrograms, the mel scale and MFCCs
  4. Hidden Markov models and Gaussian mixtures
  5. Decoding with Viterbi and beam search
  6. N-gram and neural language models
  7. Deep acoustic models
  8. End-to-end recognition: CTC, attention, transducers
  9. Evaluation: word error rate
  10. Speaker recognition and verification
  11. Noise robustness and speech enhancement
  12. Speech synthesis (TTS)

Skills you will gain

extracting features with librosatranscribing and fine-tuning with Whispertraining models with torchaudioanalysing speech in Praatcomparing systems by WER

A typical semester; the order and coverage vary between universities.

Try It

A Smart Home Recognizes Commands

A smart home assistant must recognize four voice commands: yes, no, stop, start. Pick a command, change the speaker, raise the noise; the system compares the sound with stored templates.

Sound is first turned into a spectrogram, a time–frequency picture. The same word can be said fast or slow; DTW matches them by stretching the time axis. As noise grows, confidence drops.

Context

Prerequisites

Courses worth knowing before this one, and why they are needed.

CourseWhy it is needed
Digital Signal Processingspectrogram, MFCC, filters
Calculuscomplex numbers, integrals and an introduction to Fourier series
Programmingarray operations and reading audio from files in Python

Context

Are You Ready for This Course?

Ask yourself; for every "no", the right-hand column shows where to start.

Ask yourselfIf not, start with
Can you explain how the Fourier transform splits a signal into frequencies?Digital Signal Processing
Can you use conditional probability and Bayes' rule?Probability and Statistics
Can you explain how a simple neural network is trained?Artificial Neural Networks
Can you read an audio file in Python and plot its waveform?An intro programming course, then a short NumPy and librosa tutorial
If you can say "yes" to most of these, you are ready; one or two gaps can be closed during the semester.

Context

Neighbouring Courses

subfieldbuilds onshares methodsapplication

Overlap

Overlapping Courses

RelationCourseShared ground
builds onThis course builds on Digital Signal Processingspectrogram, MFCC, filters
shares methodsShares methods with Deep Learningend-to-end speech recognition
shares methodsShares methods with Natural Language Processing and Text Mininglanguage model, speech to text

Differences

Where It Differs (1/2)

CriterionSpeech and AudioSignal ProcessingNLP
Core questionHow are words, speakers and events extracted from a sound wave?How are sampled signals analyzed, filtered and transformed?How does a computer analyze, understand and generate human language?
Approachacoustic signal and language modelmathematical transforms and filterslinguistics, statistics and deep learning
Inputaudio waveformsampled signaltext
Outputtext, speaker, sound eventfiltered signal, spectrumlabel, translation, summary, answer
Success measureword error rate (WER)signal-to-noise ratio, frequency responseaccuracy, BLEU, human evaluation
Math loadmedium-highhigh: complex numbers, Fouriermedium
Data needsmany hours of recordingsno learning; the signal sufficeslarge text corpora
  • Signal Processing: Signal processing filters and transforms any kind of signal; this course focuses on features and models specific to audio and speech.
  • NLP: NLP assumes it already has text; this course obtains that text from the sound wave and also handles audio events.

Differences

Where It Differs (2/2)

CriterionSpeech and AudioDeep LearningBiometrics
Core questionHow are words, speakers and events extracted from a sound wave?How do many-layered networks learn useful representations from raw data?How is a person recognized from their face, fingerprint or voice?
Approachacoustic signal and language modelmultilayer representation learningmatching and verification
Inputaudio waveformraw data: images, text, audioface, fingerprint, iris, voice
Outputtext, speaker, sound eventlearned representation and predictionidentity decision
Success measureword error rate (WER)accuracy, loss, scalingFAR, FRR, EER
Math loadmedium-highhigh: derivatives, matrices, probabilitymedium
Data needsmany hours of recordingslarge; needs GPUsfew samples per person
  • Deep Learning: Deep learning offers general network architectures; this course adapts them to the time-frequency structure of sound and the alignment problem.
  • Biometrics: Biometrics uses the voice as proof of identity, like a face or fingerprint; this course also analyzes what is said.

Practice

Applications

Voice assistants

Understanding commands to control a phone or home.

Captioning

Turning video, meeting and lecture recordings into text.

Call centers

Analyzing and routing calls.

Speaker verification

Voice-based banking and access control.

Sound event detection

Detecting breaking glass, sirens, coughs.

Speech synthesis

Turning text into natural-sounding speech.

Practice

Risks and Ethics

Voice cloning

A voice imitated from a few seconds of recording is used in fraud and to defeat voice authentication.

Always-on listening

Devices waiting for a wake word can record by mistake, and a voice also carries health and emotion information.

Accent and dialect gaps

Error rates are clearly higher for some accents and dialects, so these speakers gain less from captions and assistants.

Hallucinated transcripts

End-to-end models can write sentences nobody said during silence or noise, which is dangerous in medical transcription.

In depth: Explainable AI and AI Ethics.

Practice

Tools and Datasets

Tools and libraries

librosaKaldiPraattorchaudioESPnetSpeechBrainWhisper

Well-known datasets

TIMITLibriSpeechMozilla Common VoiceSwitchboardVoxCelebAudioSet

Practice

Common Misconceptions

Common beliefIn fact
Speech recognition maps sound directly to words.The same word sounds very different across accents, speeds and noise; the language model resolves the ambiguity.
The louder the recording, the better.Clipping and reverberation hurt recognition; a clean, balanced recording is what matters.
A model that works well in English works in Turkish too.Each language needs enough data; agglutinative Turkish also has a vocabulary problem to solve.
Audio processing is only about speech.Music, environmental sounds and medical sounds (heart, cough) are also part of the course.

Next Steps

After This Course

Courses that build on or extend this one

This course is mostly an end point: it builds on the others.

Related careers

Speech recognition engineerAudio processing engineerResearcherEmbedded audio developer
After this course you can move on to natural language processing for the meaning of the text, to embedded AI for on-device voice assistants, or to biometric systems for voice authentication. Digital signal processing deepens the mathematics behind the subject.

Next Steps

Further Reading

  • Rabiner and Juang, Fundamentals of Speech Recognition, Prentice Hall, 1993.
  • Huang, Acero and Hon, Spoken Language Processing, Prentice Hall, 2001.
  • Jurafsky and Martin, Speech and Language Processing, 2nd ed., Prentice Hall, 2009.
  • Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition", Proceedings of the IEEE 77(2), 1989.

The End

Chapter 19: Speech Recognition and Audio Processing
← Previous chapter · Map of the field · Next chapter →
1 / 1 Sercan KÜLCÜ. All rights reserved.