博士大叔使用计算机作弊降维打击2022高考数学压轴大题 Ph.D. uses computer cheating to solve 2022 college entrance math exam
What happens when you turn a research-grade computer algebra toolkit loose on the hardest problem of the 2022 Chinese gaokao math exam?
![[Long Review] Cascaded Diffusion Models for High Fidelity Image Generation](/media/video/2022/06/long-review-cascaded-diffusion-models-for-high-fidelity-image-generation/preview.jpg)
[Long Review] Cascaded Diffusion Models for High Fidelity Image Generation
Before Imagen and DALL-E 2 made diffusion the default for generative imagery, this Google Brain paper laid out the cascaded diffusion recipe that a lot of late…
![[Short Review] Cascaded Diffusion Models for High Fidelity Image Generation](/media/video/2022/06/short-review-cascaded-diffusion-models-for-high-fidelity-image-generation/preview.jpg)
[Short Review] Cascaded Diffusion Models for High Fidelity Image Generation
Cascaded diffusion is the two-stage-plus recipe underneath a lot of modern generative work: generate a small image with one diffusion model, then hand it to su…
![[Long Review] Axial Attention in Multidimensional Transformers](/media/video/2022/05/long-review-axial-attention-in-multidimensional-transformers/preview.jpg)
[Long Review] Axial Attention in Multidimensional Transformers
Full self-attention over an image or a video is prohibitively expensive, so how do you keep the expressiveness of a Transformer without the quadratic blowup?
![[Short Review] Axial Attention in Multidimensional Transformers](/media/video/2022/05/short-review-axial-attention-in-multidimensional-transformers/preview.jpg)
[Short Review] Axial Attention in Multidimensional Transformers
Axial attention is one of those elegant ideas that keeps Transformers tractable on high-dimensional data: instead of attending over every pixel or every time-f…
![[Long Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](/media/video/2022/05/long-review-transfer-learning-from-speaker-verification-to-multispeaker-text-to-speech/preview.jpg)
[Long Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
This is the paper that made zero-shot voice cloning practical: train a speaker encoder on a discriminative verification task with thousands of noisy, transcrip…
![[Short Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](/media/video/2022/05/short-review-transfer-learning-from-speaker-verification-to-multispeaker-text-to-speech/preview.jpg)
[Short Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
The Google paper that kicked off the modern zero-shot voice cloning wave has a beautifully clean architecture: a speaker encoder trained on verification, a Tac…
![[Long Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](/media/video/2022/05/long-review-wav2seq-pre-training-speech-to-text-encoder-decoder-models-using-pseudo/preview.jpg)
[Long Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
Most speech pretraining has focused on the encoder side, wav2vec, HuBERT, WavLM, but sequence-to-sequence ASR needs a pretrained decoder too.
![[Short Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](/media/video/2022/05/short-review-wav2seq-pre-training-speech-to-text-encoder-decoder-models-using-pseudo/preview.jpg)
[Short Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
Self-supervised pretraining like wav2vec 2.0 gave us great speech encoders, but if you want a full encoder-decoder ASR system, the decoder was still starting f…
![[Long Review] Towards Zero-Label Language Learning](/media/video/2022/05/long-review-towards-zero-label-language-learning/preview.jpg)
[Long Review] Towards Zero-Label Language Learning
What if you could train task-specific NLP models without a single human-labeled example? "Towards Zero-Label Language Learning" leans hard into that question.
![[Short Review] Towards Zero-Label Language Learning](/media/video/2022/05/short-review-towards-zero-label-language-learning/preview.jpg)
[Short Review] Towards Zero-Label Language Learning
What if the labeled data you spent months annotating wasn't necessary at all?
![[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs](/media/video/2022/04/long-review-fully-sharded-data-parallel-faster-ai-training-with-fewer-gpus/preview.jpg)
[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, …
![[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs](/media/video/2022/04/short-review-fully-sharded-data-parallel-faster-ai-training-with-fewer-gpus/preview.jpg)
[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
Fully Sharded Data Parallel is Meta's answer to a question every speech and language team eventually asks: how do we train models an order of magnitude bigger …
![[Long Review] Deduplicating Training Data Makes Language Models Better](/media/video/2022/04/long-review-deduplicating-training-data-makes-language-models-better/preview.jpg)
[Long Review] Deduplicating Training Data Makes Language Models Better
Deduplication sounds boring until you learn that a single 61-word sentence appears more than 60,000 times inside C4, and that language models trained on such c…
![[Short Review] Deduplicating Training Data Makes Language Models Better](/media/video/2022/04/short-review-deduplicating-training-data-makes-language-models-better/preview.jpg)
[Short Review] Deduplicating Training Data Makes Language Models Better
How much of your language model's "generalization" is actually just regurgitation?
![[Long Review] CLAS: Deep context: end-to-end contextual speech recognition](/media/video/2022/04/long-review-clas-deep-context-end-to-end-contextual-speech-recognition/preview.jpg)
[Long Review] CLAS: Deep context: end-to-end contextual speech recognition
Anyone who has shipped an ASR product knows the pain of proper nouns: contact names, song titles, medication names, obscure place names.
![[Long Review] Hurdles to Progress in Long Form Question Answering](/media/video/2022/04/long-review-hurdles-to-progress-in-long-form-question-answering/preview.jpg)
[Long Review] Hurdles to Progress in Long Form Question Answering
Long-form question answering is one of those benchmarks that looks impressive on the leaderboard and disturbing under a microscope.
![[Long Review] Finetuned Language Models Are Zero-Shot Learners](/media/video/2022/03/long-review-finetuned-language-models-are-zero-shot-learners/preview.jpg)
[Long Review] Finetuned Language Models Are Zero-Shot Learners
Before InstructGPT made instruction tuning a household phrase, Google's FLAN paper made the case that a modest amount of multi-task fine-tuning on natural-lang…
![[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding](/media/video/2022/02/long-review-gshard-scaling-giant-models-with-conditional-computation-and-automatic/preview.jpg)
[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding
Scaling a Transformer past a hundred billion parameters gets philosophical fast: do you activate the whole network for every token, or route each token to the …
Triplets-like Russian Figure Skaters: Can Kullback-Leibler Divergence be Used Tell Their Difference?
Speaker adaptation is one of those ASR problems that sounds solved until you actually try to adapt a sequence-to-sequence model without wrecking the encoder's …
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models
Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…
Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 2: GLaM:Efficient Scaling of LMs with MoE
Part 2 of the MoE mini-series turns to GLaM, Google's 1.2-trillion-parameter Mixture-of-Experts language model that matches or beats GPT-3 on zero-, one-, and …
BERT Paper Reviewed from a Speech Perspective
BERT hardly needs an introduction in NLP, but reading it through a speech engineer's eyes uncovers a different set of lessons than the standard "masked languag…
Transformer: Attention is All You Need and Listen, Attend and Spell -- from a Speech Perspective
Two papers, one story. "Attention Is All You Need" gave the world the Transformer and reset how sequence modeling gets done, while "Listen, Attend and Spell" (…
From Breaking Bad to Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
What do Walter White, Jesse Pinkman, and a batch of unlabeled audio have in common?
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
wav2vec 2.0 changed the game with contrastive self-supervision, but HuBERT asked a sharper question: what if we skip the contrastive trick entirely and just do…
W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self Supervise
Why choose between contrastive learning and masked language modeling for speech pretraining when you can wire both into the same network?
Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition
Speech emotion recognition has long been stuck with tiny labeled datasets and hand-crafted acoustic features.
Joint Unsupervised and Supervised Training for Multilingual ASR
Pretrain-then-finetune has become the default recipe for multilingual ASR, but it leaves a lot on the table: the two stages don't share losses, and the supervi…
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Most speech SSL models optimize for ASR and hope everything else works out.