The Rise of Transformer-Based Language Models with 100 Million to 5 Billion Parameters
Researchers in the field of artificial intelligence have been focusing on transformer-based, decoder-only language models with 100 million to 5 billion parameters. A recent survey examined 59 cutting-edge open-source models, exploring innovations in architecture, training datasets, and algorithms.
The study also delved into the model abilities in areas such as common-sense reasoning, in-context learning, math, and coding. Researchers benchmarked latency and memory usage during inference to assess model performance on devices.
The term “small” when referring to language models is subjective and relative, with its meaning evolving as device memory capacity increases. The study identified 5 billion parameters as the upper limit for Small Language Models (SLMs), while Large Language Models (LLMs) with 7 billion parameters are predominantly deployed in the cloud.
Small Language Models (SLMs) are specifically designed for efficient deployment on devices like desktops, smartphones, and wearables. The aim is to make advanced machine intelligence accessible and affordable, akin to the universal nature of human cognition.
SLMs have already seen widespread integration into commercial devices. For instance, the latest Google and Samsung smartphones include built-in Large Language Model (LLM) services, allowing third-party apps to leverage LLM capabilities through prompts and modular integrations.