Description

Finbarr Timbers delves into the topic of Mixture of Experts (MoE) models in his latest post, 'Papers I’ve read this week, Mixture of Experts edition'. MoE models have been propelled into the limelight due to rumors about their potential use in GPT-4. These innovative models employ a form of model parallelism that allows input tokens to select combinations of parameters for each input. Timbers explains the 'winners get bigger' effect, poor sharding performance, and difficulties comparing MoE performance with dense models. Additionally, he provides insights into specific papers addressing these challenges and shares his thoughts on how MoE models could transform AI and the pursuit of AGI-like capabilities