![[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding](/media/video/2022/02/long-review-gshard-scaling-giant-models-with-conditional-computation-and-automatic/preview.jpg)
Tutorials
[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding
Scaling a Transformer past a hundred billion parameters gets philosophical fast: do you activate the whole network for every token, or route each token to the …