Review Microsoft's VALL-E 2 (Achieving Human Parity in Zero-shot TTS)
Zero-shot TTS just crossed a line many thought was still years away: Microsoft's VALL-E 2 is the first neural codec language model to hit human parity on both …
Understand Microsoft's VALL-E 2 in 3 Minutes (Achieving Human Parity in Zero-shot TTS)
Short on time but need to know why VALL-E 2 is suddenly the reference point for zero-shot TTS?
From OpenAI Whisper to Your In-House ASR Service: Recognizing Name Entities & Domain-Specific Terms
Whisper out of the box is impressive, but anyone who's actually shipped it into production knows the pain: it fumbles proper nouns, product names, medical term…
Google Researcher's In-Depth Analysis on End-to-End Speech Recognition, Part 1: Overview & Modeling
A decade of deep learning cut ASR word error rates by more than 50% relative and, along the way, made classical HMM pipelines look like a relic.
Deduct OpenAI GPT-4o's Neural Network Architecture
OpenAI hasn't published a paper on GPT-4o, so this video does the next best thing: it reverse-engineers what the neural network architecture almost certainly l…
Google's Universal Speech Model for 100+ languages beats OpenAI's Whisper Model
Google's Universal Speech Model quietly did something remarkable: matched or beat OpenAI's Whisper on ASR across 100+ languages while using roughly one-seventh…
Long Review: Apple's MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Apple doesn't publish much, so when the MM1 paper landed with a full ablation study on multimodal LLM pre-training, it was one of the more useful data releases…
A Quick Review of Apple's SOTA Multimodal LLM: MM1
If you don't have time for the full MM1 breakdown, this quick review gets you the essentials in a few minutes: what Apple built, what they learned from the abl…
The Secret That Made Claude 3 Trump GPT-4
When Claude 3 Opus edged out GPT-4 on MMLU, GPQA, GSM8K, and most of the frontier evaluation benchmarks, the interesting question wasn't whether Anthropic had …
A Review of Microsoft+OpenAI, Google, Meta, and Nvidia's Open Source Large Speech Models for ASR
Every major AI lab has now shipped an open-source large speech model for ASR, and picking the right one for production has become a real decision.
From OpenAI's Whisper Model to Your Own In-House ASR Service: Postprocessing and Language Modeling
Raw Whisper output is impressive out of the box, but anyone who has shipped it to real users knows the gap between a demo transcript and something a downstream…
From OpenAI's Whisper Model to Your Own In-House ASR Service: Long Audio and Streaming (Part 3)
Whisper was trained on 30-second chunks, and it shows the moment you feed it a two-hour podcast or try to wire it into a live captioning pipeline.
From OpenAI's Whisper Model to Your Own In-House ASR Service: ROI (Return-on-Investment) (Part 2)
The build-versus-buy question for ASR usually gets framed as accuracy versus convenience, but the real deciding factor is ROI over the lifetime of the product.
From OpenAI's Whisper Model to Your Own In-House ASR Service: Overview (Part 1)
When OpenAI open-sourced Whisper, it quietly reset the baseline for what an in-house ASR service looks like.
Why word timestamps generated by OpenAI Whisper are not accurate? How to make them accurate again?
Ask Whisper for word timestamps and you'll get numbers back — but if you've ever tried to line them up with the actual audio for karaoke, captioning, or dubbin…
Speech Generative AI: VoiceBox by Meta AI (also Flow Matching and Neural ODE)
For years, speech generation lagged text and image generation in both scale and generality — every task got its own bespoke model.
A Review of SpeechT5: Introducing Google's T5 into Speech (ASR, TTS, SID, ...) Tasks
T5 unified text-to-text tasks under a single encoder-decoder pretraining recipe and became a workhorse of NLP.
![[Olewave's Review] AudioLM: a Language Modeling Approach to Audio Generation](/media/video/2023/03/olewaves-review-audiolm-a-language-modeling-approach-to-audio-generation/preview.jpg)
[Olewave's Review] AudioLM: a Language Modeling Approach to Audio Generation
AudioLM is the paper that convinced a lot of people audio generation should look more like language modeling than signal processing.
![[10 mins] Explain Why OpenAI's Whisper API Isn't As Good As ChatGPT](/media/video/2023/03/10-mins-explain-why-openais-whisper-api-isnt-as-good-as-chatgpt/preview.jpg)
[10 mins] Explain Why OpenAI's Whisper API Isn't As Good As ChatGPT
Whisper landed with a bang, but it did not reshape ASR the way GPT-3 reshaped NLP, and this ten-minute review argues the reasons are baked into the paper itsel…
In-depth review of OpenAI's GPT-3 : Language Models are Few-Shot Learners (Part 1/3: Intro&Approach)
The opening installment of this three-part GPT-3 review sets up the paper that turned scaling from a research bet into industry gospel.
In-depth Review of VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Zero-shot TTS from a three-second reference clip sounded like a stretch until VALL-E shipped, and this in-depth review unpacks exactly how Microsoft pulled it …
Understand Microsoft's VALL-E in 3 Minutes (SOTA Zero-shot TTS)
VALL-E is the moment TTS stopped looking like signal regression and started looking like language modeling, and this quick explainer gets you up to speed fast.
![[Olewave's Review] OpenAI's Whisper ASR: Robust Speech Recognition via Large-Scale Weak Supervision](/media/video/2022/10/olewaves-review-openais-whisper-asr-robust-speech-recognition-via-large-scale-weak/preview.jpg)
[Olewave's Review] OpenAI's Whisper ASR: Robust Speech Recognition via Large-Scale Weak Supervision
When OpenAI dropped Whisper, the ASR community had to reckon with a system trained on 680,000 hours of multilingual, multitask, web-scraped audio that just wor…
Olewave's most detailed illustration of RNN-T: Sequence Transduction with Recurrent Neural Networks
Alex Graves' RNN-T paper is the quiet ancestor of nearly every streaming ASR system shipping today, from Google's on-device recognizer to countless open-source…
![[Olewave's Long Review] Efficient Training of Neural Transducer for Speech Recognition](/media/video/2022/07/olewaves-long-review-efficient-training-of-neural-transducer-for-speech-recognition/preview.jpg)
[Olewave's Long Review] Efficient Training of Neural Transducer for Speech Recognition
Neural transducers are the workhorse of streaming ASR, but training them efficiently is famously painful: the RNN-T loss has cubic memory in sequence length, t…
![[Long Review] Conformer: Convolution-augmented Transformer for Speech Recognition](/media/video/2022/06/long-review-conformer-convolution-augmented-transformer-for-speech-recognition/preview.jpg)
[Long Review] Conformer: Convolution-augmented Transformer for Speech Recognition
Conformer is the model that quietly took over end-to-end ASR, and its recipe, bolt a convolution module onto every Transformer block, turned out to be one of t…
![[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs](/media/video/2022/04/long-review-fully-sharded-data-parallel-faster-ai-training-with-fewer-gpus/preview.jpg)
[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, …
![[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs](/media/video/2022/04/short-review-fully-sharded-data-parallel-faster-ai-training-with-fewer-gpus/preview.jpg)
[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Fully Sharded Data Parallel is Meta's answer to a question every speech and language team eventually asks: how do we train models an order of magnitude bigger …
From Breaking Bad to Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
What do Walter White, Jesse Pinkman, and a batch of unlabeled audio have in common?
Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition
Speech emotion recognition has long been stuck with tiny labeled datasets and hand-crafted acoustic features.