Tutorials
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models
Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…
Tutorials
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 2: GLaM:Efficient Scaling of LMs with MoE
Part 2 of the MoE mini-series turns to GLaM, Google's 1.2-trillion-parameter Mixture-of-Experts language model that matches or beats GPT-3 on zero-, one-, and …
Tutorials
BERT Paper Reviewed from a Speech Perspective
BERT hardly needs an introduction in NLP, but reading it through a speech engineer's eyes uncovers a different set of lessons than the standard "masked languag…
Tutorials
Transformer: Attention is All You Need and Listen, Attend and Spell -- from a Speech Perspective
Two papers, one story. "Attention Is All You Need" gave the world the Transformer and reset how sequence modeling gets done, while "Listen, Attend and Spell" (…