[Olewave's Long Review] Efficient Training of Neural Transducer for Speech Recognition
Neural transducers are the workhorse of streaming ASR, but training them efficiently is famously painful: the RNN-T loss has cubic memory in sequence length, t…
Boris Johnson’s Rise and Fall - an analysis of the mics
Political news is not usually what voice AI engineers cue up for a Friday afternoon, but Boris Johnson's resignation announcement, delivered in front of a bank…
[Olewave's Long Review] Xception: Deep Learning with Depthwise Separable Convolutions
Xception took the Inception hypothesis to its logical extreme by pushing cross-channel and spatial correlations into fully separate operations, and the depthwi…
[Olewave's Short Review] Xception: Deep Learning with Depthwise Separable Convolutions
If you just want the core Xception idea without the long-form deep dive, this short review is the express version.
Phased Array Radar on China's Aircraft Carrier Fujian 003 and Its Connection with Speech Beamforming
The phased array radar mounted on China's newest aircraft carrier, Fujian 003, and the microphone arrays that let your smart speaker hear you across a noisy ro…
How Does the All-New Dictation in iOS 16 Work? Reveal Apple's Secret Sauce by a Speech Researcher!
Apple's iOS 16 dictation runs fully on-device, and that single design choice has cascading consequences for latency, privacy, and what kinds of ASR architectur…
[Long Review] Conformer: Convolution-augmented Transformer for Speech Recognition
Conformer is the model that quietly took over end-to-end ASR, and its recipe, bolt a convolution module onto every Transformer block, turned out to be one of t…
[Short Review] Conformer: Convolution-augmented Transformer for Speech Recognition
Conformer took the ASR world by storm by doing something almost boring: gluing a convolution module into each Transformer block so the model captures both glob…
博士大叔使用计算机作弊降维打击2022高考数学压轴大题 Ph.D. uses computer cheating to solve 2022 college entrance math exam
What happens when you turn a research-grade computer algebra toolkit loose on the hardest problem of the 2022 Chinese gaokao math exam?
[Long Review] Cascaded Diffusion Models for High Fidelity Image Generation
Before Imagen and DALL-E 2 made diffusion the default for generative imagery, this Google Brain paper laid out the cascaded diffusion recipe that a lot of late…
[Short Review] Cascaded Diffusion Models for High Fidelity Image Generation
Cascaded diffusion is the two-stage-plus recipe underneath a lot of modern generative work: generate a small image with one diffusion model, then hand it to su…
[Long Review] Axial Attention in Multidimensional Transformers
Full self-attention over an image or a video is prohibitively expensive, so how do you keep the expressiveness of a Transformer without the quadratic blowup?
[Short Review] Axial Attention in Multidimensional Transformers
Axial attention is one of those elegant ideas that keeps Transformers tractable on high-dimensional data: instead of attending over every pixel or every time-f…
[Long Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
This is the paper that made zero-shot voice cloning practical: train a speaker encoder on a discriminative verification task with thousands of noisy, transcrip…
[Short Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
The Google paper that kicked off the modern zero-shot voice cloning wave has a beautifully clean architecture: a speaker encoder trained on verification, a Tac…
[Long Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
Most speech pretraining has focused on the encoder side, wav2vec, HuBERT, WavLM, but sequence-to-sequence ASR needs a pretrained decoder too.
[Short Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
Self-supervised pretraining like wav2vec 2.0 gave us great speech encoders, but if you want a full encoder-decoder ASR system, the decoder was still starting f…
[Long Review] Towards Zero-Label Language Learning
What if you could train task-specific NLP models without a single human-labeled example? "Towards Zero-Label Language Learning" leans hard into that question.
[Short Review] Towards Zero-Label Language Learning
What if the labeled data you spent months annotating wasn't necessary at all?
[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, …
[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Fully Sharded Data Parallel is Meta's answer to a question every speech and language team eventually asks: how do we train models an order of magnitude bigger …
[Long Review] Deduplicating Training Data Makes Language Models Better
Deduplication sounds boring until you learn that a single 61-word sentence appears more than 60,000 times inside C4, and that language models trained on such c…
[Short Review] Deduplicating Training Data Makes Language Models Better
How much of your language model's "generalization" is actually just regurgitation?
[Long Review] CLAS: Deep context: end-to-end contextual speech recognition
Anyone who has shipped an ASR product knows the pain of proper nouns: contact names, song titles, medication names, obscure place names.
[Long Review] Hurdles to Progress in Long Form Question Answering
Long-form question answering is one of those benchmarks that looks impressive on the leaderboard and disturbing under a microscope.
[Long Review] Finetuned Language Models Are Zero-Shot Learners
Before InstructGPT made instruction tuning a household phrase, Google's FLAN paper made the case that a modest amount of multi-task fine-tuning on natural-lang…
[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding
Scaling a Transformer past a hundred billion parameters gets philosophical fast: do you activate the whole network for every token, or route each token to the …
Triplets-like Russian Figure Skaters: Can Kullback-Leibler Divergence be Used Tell Their Difference?
Speaker adaptation is one of those ASR problems that sounds solved until you actually try to adapt a sequence-to-sequence model without wrecking the encoder's …
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models
Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…