![[Detailed Paper Reading] Zipformer: A faster and better encoder for automatic speech recognition](/media/video/2024/02/detailed-paper-reading-zipformer-a-faster-and-better-encoder-for-automatic-speech/preview.jpg)
[Detailed Paper Reading] Zipformer: A faster and better encoder for automatic speech recognition
The Conformer has been the default ASR encoder for years, but the k2/icefall team's Zipformer quietly took the WER crown on LibriSpeech, Aishell-1, and WenetSp…
I-JEPA: Yannn LeCun's First 'World Model' for Computer Vision
Yann LeCun has been arguing for years that generative pretraining is the wrong bet for building machines that understand the world.
LoRA: allow a high school student to train Large Language Model (GPT-3) with a gaming graphics card
Full fine-tuning of a 175B-parameter GPT-3 is a non-starter for almost everyone — the storage alone rules it out, never mind the GPUs.
![[Short Review] Conformer: Convolution-augmented Transformer for Speech Recognition](/media/video/2022/06/short-review-conformer-convolution-augmented-transformer-for-speech-recognition/preview.jpg)
[Short Review] Conformer: Convolution-augmented Transformer for Speech Recognition
Conformer took the ASR world by storm by doing something almost boring: gluing a convolution module into each Transformer block so the model captures both glob…
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models
Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 2: GLaM:Efficient Scaling of LMs with MoE
Part 2 of the MoE mini-series turns to GLaM, Google's 1.2-trillion-parameter Mixture-of-Experts language model that matches or beats GPT-3 on zero-, one-, and …
BERT Paper Reviewed from a Speech Perspective
BERT hardly needs an introduction in NLP, but reading it through a speech engineer's eyes uncovers a different set of lessons than the standard "masked languag…
Transformer: Attention is All You Need and Listen, Attend and Spell -- from a Speech Perspective
Two papers, one story. "Attention Is All You Need" gave the world the Transformer and reset how sequence modeling gets done, while "Listen, Attend and Spell" (…