Review Microsoft's VALL-E 2 (Achieving Human Parity in Zero-shot TTS)
Zero-shot TTS just crossed a line many thought was still years away: Microsoft's VALL-E 2 is the first neural codec language model to hit human parity on both …
Understand Microsoft's VALL-E 2 in 3 Minutes (Achieving Human Parity in Zero-shot TTS)
Short on time but need to know why VALL-E 2 is suddenly the reference point for zero-shot TTS?
Review DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation ~ Seed-TTS
The Seed-TTS team is back with DiTAR, and it's a clean answer to one of the more awkward tradeoffs in modern speech generation: how do you keep the flexibility…
Review ByteDance/Tiktok's Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
ByteDance's Seed-TTS is one of the strongest signals yet that TTS is graduating from a component into a genuine foundation model.
From OpenAI Whisper to Your In-House ASR Service: Recognizing Name Entities & Domain-Specific Terms
Whisper out of the box is impressive, but anyone who's actually shipped it into production knows the pain: it fumbles proper nouns, product names, medical term…
Meta's Movie Gen vs. OpenAI's Sora: a Detailed Review
Meta finally showed its hand against Sora, and Movie Gen is more than a video generator — it's a cast of foundation models that produces 1080p HD video across …
Google Researcher's In-Depth Analysis on End-to-End Speech Recognition, Part 1: Overview & Modeling
A decade of deep learning cut ASR word error rates by more than 50% relative and, along the way, made classical HMM pipelines look like a relic.
Deduct OpenAI GPT-4o's Neural Network Architecture
OpenAI hasn't published a paper on GPT-4o, so this video does the next best thing: it reverse-engineers what the neural network architecture almost certainly l…
Google's Universal Speech Model for 100+ languages beats OpenAI's Whisper Model
Google's Universal Speech Model quietly did something remarkable: matched or beat OpenAI's Whisper on ASR across 100+ languages while using roughly one-seventh…
Long Review: Apple's MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Apple doesn't publish much, so when the MM1 paper landed with a full ablation study on multimodal LLM pre-training, it was one of the more useful data releases…
A Quick Review of Apple's SOTA Multimodal LLM: MM1
If you don't have time for the full MM1 breakdown, this quick review gets you the essentials in a few minutes: what Apple built, what they learned from the abl…
The Secret That Made Claude 3 Trump GPT-4
When Claude 3 Opus edged out GPT-4 on MMLU, GPQA, GSM8K, and most of the frontier evaluation benchmarks, the interesting question wasn't whether Anthropic had …
Generative AI Models Related to Sora: Normalizing Flows
Everyone's talking about diffusion, but if you actually want to understand the family tree that produced Sora and modern generative video, you need to spend so…
Variational Autoencoder (VAE) and Reparameterization Trick - Revisiting the Classic Generative Model
Before diffusion, before normalizing flows, before latent diffusion models quietly built VAE encoders into every serious generative stack — there was Kingma an…
A Review of Microsoft+OpenAI, Google, Meta, and Nvidia's Open Source Large Speech Models for ASR
Every major AI lab has now shipped an open-source large speech model for ASR, and picking the right one for production has become a real decision.
![[Detailed Paper Reading] Zipformer: A faster and better encoder for automatic speech recognition](/media/video/2024/02/detailed-paper-reading-zipformer-a-faster-and-better-encoder-for-automatic-speech/preview.jpg)
[Detailed Paper Reading] Zipformer: A faster and better encoder for automatic speech recognition
The Conformer has been the default ASR encoder for years, but the k2/icefall team's Zipformer quietly took the WER crown on LibriSpeech, Aishell-1, and WenetSp…
Tycho:a tookit for building high-ROI in-house speech-related services (ASR/TTS/Translation):Overview
Most teams that want an in-house ASR, TTS, or speech translation service face the same messy choice: glue together fragments of ESPnet, NeMo, k2, and HuggingFa…
From OpenAI's Whisper Model to Your Own In-House ASR Service: Postprocessing and Language Modeling
Raw Whisper output is impressive out of the box, but anyone who has shipped it to real users knows the gap between a demo transcript and something a downstream…
From OpenAI's Whisper Model to Your Own In-House ASR Service: Long Audio and Streaming (Part 3)
Whisper was trained on 30-second chunks, and it shows the moment you feed it a two-hour podcast or try to wire it into a live captioning pipeline.
From OpenAI's Whisper Model to Your Own In-House ASR Service: ROI (Return-on-Investment) (Part 2)
The build-versus-buy question for ASR usually gets framed as accuracy versus convenience, but the real deciding factor is ROI over the lifetime of the product.
From OpenAI's Whisper Model to Your Own In-House ASR Service: Overview (Part 1)
When OpenAI open-sourced Whisper, it quietly reset the baseline for what an in-house ASR service looks like.
Why word timestamps generated by OpenAI Whisper are not accurate? How to make them accurate again?
Ask Whisper for word timestamps and you'll get numbers back — but if you've ever tried to line them up with the actual audio for karaoke, captioning, or dubbin…
Speech Generative AI: VoiceBox by Meta AI (also Flow Matching and Neural ODE)
For years, speech generation lagged text and image generation in both scale and generality — every task got its own bespoke model.
I-JEPA: Yannn LeCun's First 'World Model' for Computer Vision
Yann LeCun has been arguing for years that generative pretraining is the wrong bet for building machines that understand the world.
LoRA: allow a high school student to train Large Language Model (GPT-3) with a gaming graphics card
Full fine-tuning of a 175B-parameter GPT-3 is a non-starter for almost everyone — the storage alone rules it out, never mind the GPUs.
A Review of SpeechT5: Introducing Google's T5 into Speech (ASR, TTS, SID, ...) Tasks
T5 unified text-to-text tasks under a single encoder-decoder pretraining recipe and became a workhorse of NLP.
A Review of Deepmind's WaveNet for TTS/Audio Synthesis (Does it look like GPT to you?)
Before Tacotron, before VALL-E, before neural codecs turned audio into tokens, DeepMind's WaveNet was the paper that showed neural networks could generate raw …
Review of HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
For a long time, neural vocoders forced you to pick a side: autoregressive models like WaveNet gave you gorgeous audio but were painfully slow, while GAN-based…
In-depth Review of Google's SoundStream: An End-to-End Neural Audio Codec
Neural codecs quietly became the foundation layer of modern generative audio, and SoundStream is one of the papers that got the ball rolling.
Disclosing OpenAI GPT-4's vision+text model, data, and cost to train (speculated)
OpenAI famously refused to disclose GPT-4's architecture, parameter count, training data, or compute budget, which turned the tech report into a Rorschach test…