Triplets-like Russian Figure Skaters: Can Kullback-Leibler Divergence be Used Tell Their Difference?
Speaker adaptation is one of those ASR problems that sounds solved until you actually try to adapt a sequence-to-sequence model without wrecking the encoder's …
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models
Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 2: GLaM:Efficient Scaling of LMs with MoE
Part 2 of the MoE mini-series turns to GLaM, Google's 1.2-trillion-parameter Mixture-of-Experts language model that matches or beats GPT-3 on zero-, one-, and …
BERT Paper Reviewed from a Speech Perspective
BERT hardly needs an introduction in NLP, but reading it through a speech engineer's eyes uncovers a different set of lessons than the standard "masked languag…
Transformer: Attention is All You Need and Listen, Attend and Spell -- from a Speech Perspective
Two papers, one story. "Attention Is All You Need" gave the world the Transformer and reset how sequence modeling gets done, while "Listen, Attend and Spell" (…
From Breaking Bad to Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
What do Walter White, Jesse Pinkman, and a batch of unlabeled audio have in common?