We want to hear from you! Take our quick AI survey and share your insights on the current state of AI, how you're implementing it, and what you expect to see in the future. learn more
MetaAI researchers have unveiled Mobile LLM, a new approach to building efficient language models designed for smartphones and other resource-constrained devices. Published on June 27, 2024, this work challenges assumptions about the necessary size of effective AI models.
The research team, comprised of members from MetaReality Labs, PTorch and MetaAI Research (FAIR), focused on optimizing models with fewer than 1 billion parameters. This is a fraction of the size of models like GPT-4, which is estimated at over a trillion parameters.
Meta's Chief AI Scientist, Yann LeCun, highlighted key aspects of the research on X (formerly known as Twitter):
Countdown to VB Transform 2024
Join enterprise leaders in San Francisco July 9-11 for our flagship AI event. Connect with peers, explore the opportunities and challenges of generative AI, and learn how to integrate AI applications into your industry. Register now
Key innovations in MobileLLM include:
- Prioritizing model depth over width
- Implementing embedding sharing and grouped query focus
- Using a novel immediate block-wise weight distribution technique
These design choices allowed MobileLLM to outperform previous models of the same size on common benchmark tasks by 2.7% to 4.3%. While these single-digit improvements may seem small, they represent meaningful progress in the highly competitive field of language model development.
In particular, the 350 million parameter version of MobileLLM demonstrated comparable accuracy to the 7 billion parameter LLaMA-2 model on certain API calling tasks. This suggests that for certain applications, more compact models can offer similar functionality while using significantly fewer computational resources.
The development of MobileLLM coincides with the growing interest in more efficient AI models. As progress in very large language models shows signs of slowing, researchers are increasingly exploring the potential of more compact, specialized designs. The focus on performance and device deployment places Mobile LLM in the same category as what some researchers call Small Language Models (SLMs), despite the “LLM” in its name.
While MobileLLM is not yet available for public use, Meta has open-sourced the pre-training code, allowing other researchers to build on their work. As the technology develops, it could enable more advanced AI features on personal devices, though the timeline and exact capabilities are uncertain.
The development of MobileLLM represents an important step towards making advanced AI more accessible and sustainable. This challenges the notion that efficient language models must be abundant, potentially opening new avenues for AI applications on personal devices.