Why word timestamps generated by OpenAI Whisper are not accurate? How to make them accurate again?
Ask Whisper for word timestamps and you'll get numbers back — but if you've ever tried to line them up with the actual audio for karaoke, captioning, or dubbin…
Speech Generative AI: VoiceBox by Meta AI (also Flow Matching and Neural ODE)
For years, speech generation lagged text and image generation in both scale and generality — every task got its own bespoke model.
I-JEPA: Yannn LeCun's First 'World Model' for Computer Vision
Yann LeCun has been arguing for years that generative pretraining is the wrong bet for building machines that understand the world.
LoRA: allow a high school student to train Large Language Model (GPT-3) with a gaming graphics card
Full fine-tuning of a 175B-parameter GPT-3 is a non-starter for almost everyone — the storage alone rules it out, never mind the GPUs.
A Review of SpeechT5: Introducing Google's T5 into Speech (ASR, TTS, SID, ...) Tasks
T5 unified text-to-text tasks under a single encoder-decoder pretraining recipe and became a workhorse of NLP.
A Review of Deepmind's WaveNet for TTS/Audio Synthesis (Does it look like GPT to you?)
Before Tacotron, before VALL-E, before neural codecs turned audio into tokens, DeepMind's WaveNet was the paper that showed neural networks could generate raw …
Review of HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
For a long time, neural vocoders forced you to pick a side: autoregressive models like WaveNet gave you gorgeous audio but were painfully slow, while GAN-based…
In-depth Review of Google's SoundStream: An End-to-End Neural Audio Codec
Neural codecs quietly became the foundation layer of modern generative audio, and SoundStream is one of the papers that got the ball rolling.
Disclosing OpenAI GPT-4's vision+text model, data, and cost to train (speculated)
OpenAI famously refused to disclose GPT-4's architecture, parameter count, training data, or compute budget, which turned the tech report into a Rorschach test…
A Review of GPT-4's Technical Report (GPT-4 in a Nutshell)
GPT-4's technical report is famously thin on architecture details and thick on capability claims, which makes a clear-headed review essential reading for anyon…
[Olewave's Review] AudioLM: a Language Modeling Approach to Audio Generation
AudioLM is the paper that convinced a lot of people audio generation should look more like language modeling than signal processing.
In-depth review of OpenAI's GPT-3 : Language Models are Few-Shot Learners (Part 3/3: Results&Rest)
The final installment of this three-part GPT-3 deep dive gets to the payoff: what actually happens when you throw 175 billion parameters at a benchmark suite a…
[10 mins] Explain Why OpenAI's Whisper API Isn't As Good As ChatGPT
Whisper landed with a bang, but it did not reshape ASR the way GPT-3 reshaped NLP, and this ten-minute review argues the reasons are baked into the paper itsel…
In-depth review of OpenAI's GPT-3 : Language Models are Few-Shot Learners (Part 2/3: Results)
Part two of this GPT-3 deep dive moves past the setup and into the results that actually made people rethink NLP.
WSJ Made a Mistake in Translating China's Foreign Minister's Speech; What has the Diplomat Said?
Machine and human translation errors are more than a language-nerd curiosity when they land on the front page of the Wall Street Journal and reshape geopolitic…
In-depth review of OpenAI's GPT-3 : Language Models are Few-Shot Learners (Part 1/3: Intro&Approach)
The opening installment of this three-part GPT-3 review sets up the paper that turned scaling from a research bet into industry gospel.
In-depth Review of VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Zero-shot TTS from a three-second reference clip sounded like a stretch until VALL-E shipped, and this in-depth review unpacks exactly how Microsoft pulled it …
Understand Microsoft's VALL-E in 3 Minutes (SOTA Zero-shot TTS)
VALL-E is the moment TTS stopped looking like signal regression and started looking like language modeling, and this quick explainer gets you up to speed fast.
ChatGPT/ChatGPT Plus/InstructGPT:Training language models to follow instructions with human feedback
Behind the ChatGPT product sits the InstructGPT paper, and this detailed review pulls apart the pipeline that turned a raw GPT-3 into something people actually…
Explain how ChatGPT/ChatGPT Plus works in 3 minutes
ChatGPT looks like magic from the outside, but the recipe is well-documented if you know where to look, and this three-minute explainer condenses the core idea…
[Olewave's Review] CLIP (3/3): Learning Transferable Visual Models From Natural Language Supervision
The final part of this CLIP review lands on the results section, which is where OpenAI's contrastive image-text model shifted from an interesting idea to a fou…
[Olewave's Review] CLIP (2/3): Learning Transferable Visual Models From Natural Language Supervision
OpenAI's CLIP flipped the script on computer vision by tossing out fixed label sets and instead training on 400 million (image, text) pairs scraped from the in…
[Olewave's Review] CLIP (1/3): Learning Transferable Visual Models From Natural Language Supervision
Before there was BLIP, LLaVA, or any speech-LLM worth its salt, there was CLIP, and this is where the story begins.
[Olewave's Review] Token-level Sequence Labeling for SLU using Compositional E2E Models
End-to-end SLU is having a moment, but treating sequence labeling as sequence prediction quietly throws away decades of well-understood token-level tagging mac…
[Olewave's Review] Branchformer: Parallel MLP-Attention Architectures, and E-Branchformer
Conformer has been the ASR encoder to beat for years, but Branchformer and its follow-up E-Branchformer make a compelling case that sequential conv-plus-attent…
Non Collision Mispronunciation Addition (NCMA) for Accented ASR
Accented and non-native speech breaks pronunciation dictionaries in ways that quietly poison your acoustic model alignments, and this Interspeech 2021 work ask…
[Olewave's Review] OpenAI's Whisper ASR: Robust Speech Recognition via Large-Scale Weak Supervision
When OpenAI dropped Whisper, the ASR community had to reckon with a system trained on 680,000 hours of multilingual, multitask, web-scraped audio that just wor…
One-Edit-Distance FSA/Network-based (OEDN) in Mispronunciation Detection and Accented ASR
Non-native acoustic modeling has a chicken-and-egg problem: bad phone alignments produce bad models, and bad models produce bad alignments.
Olewave's most detailed illustration of RNN-T: Sequence Transduction with Recurrent Neural Networks
Alex Graves' RNN-T paper is the quiet ancestor of nearly every streaming ASR system shipping today, from Google's on-device recognizer to countless open-source…
Google fired Blake Lemoine for saying AI bot is sentient? Does LaMDA or ChatGPT think like human?
When Google fired Blake Lemoine for going public with claims that LaMDA had become sentient, the story went viral, but the underlying question stuck around: do…