Small language models are compact enough to run on a single device, usually below about ten billion parameters and often between one and four billion. They matter for embedded and edge systems because a quantised model of this size fits in a few gigabytes of memory, runs without a network connection, keeps data on the device, and answers with low latency. They do not replace the largest models for every task, but for many focused jobs they are accurate enough, far cheaper to run, and easier to keep private.
Small language models, often written SLMs, are the practical answer to a real problem: the largest language models are expensive to serve, need constant network access, and send user data to a remote service. For an embedded product, those three properties are often unacceptable. A small language model is compact enough to run on the device itself, which changes the cost, the privacy, and the latency of adding language capability to a product. This article explains what qualifies as a small language model, why the size fits edge and embedded systems, what the evidence says about capability, and how to deploy one on embedded Linux.
What counts as a small language model
There is no single official cut-off, but a useful working definition comes from recent NVIDIA research, which defines a small language model as one that can run on a common consumer device with latency low enough to serve a single user, and adds that as of 2025 it is reasonable to treat most models below ten billion parameters as small. In practice most models deployed on devices today sit between about half a billion and four billion parameters. Current examples include Meta’s Llama 3.2 at 1B and 3B, Google’s Gemma 2 at 2B, Microsoft’s Phi-3-mini at 3.8B, Alibaba’s Qwen2.5 in its smaller sizes from 0.5B to 7B, Hugging Face’s SmolLM2 at 1.7B, TinyLlama at 1.1B, and Apple’s OpenELM family from 270M to 3B.
The parameter count matters because it sets the memory the model needs. A model stored at full 16-bit precision needs about two bytes per parameter, so a 3.8B model would need around 7.6 GB just for weights. Quantisation reduces this substantially by storing weights at lower precision. At 4-bit precision, roughly half a byte per parameter, the same 3.8B model needs about 2 GB for weights, and a 1B model needs under 1 GB. This is what makes on-device deployment realistic on hardware with limited RAM.
Why small language models fit edge and embedded systems
Four properties make this size range suitable for embedded and edge deployment, and each maps to a concrete engineering constraint.
- Memory. A quantised one-to-four-billion parameter model fits in the RAM available on many single-board computers and higher-end embedded platforms, which a seventy-billion parameter model does not.
- Power and cost. Smaller models require far less compute per response, which lowers energy draw on battery-powered devices and removes the recurring cost of calling a hosted service.
- Latency. Running locally removes the network round trip, so responses begin immediately and do not depend on connection quality.
- Privacy. When the model runs on the device, the input never leaves it. For products handling personal, medical, or industrial data, this is often a requirement rather than a preference, and it can simplify compliance in regulated industries.
💡 Key insight: The reason to choose a small language model is rarely that it is the most capable option. It is that on-device constraints — memory, power, offline operation, and data that must not leave the device — rule out a large hosted model before capability is even considered.
What the evidence says about capability
The concern with a smaller model is always accuracy. The useful finding of the last two years is that the gap has narrowed for many tasks. Microsoft’s Phi-3 technical report describes a 3.8B model trained on carefully filtered data that performs, on several standard reasoning and language benchmarks, close to models many times its size. The result is important less for the exact numbers and more for the reason behind them: data quality and training method, not only parameter count, determine how capable a model is.
The second piece of evidence is about where these models are used. The 2025 NVIDIA paper “Small Language Models are the Future of Agentic AI” argues that in agent systems, where the model is called repeatedly to perform narrow, repetitive steps such as parsing, routing, or calling a tool, a small model is often sufficient and is far cheaper to run. The paper estimates that serving a seven-billion parameter model is on the order of ten to thirty times cheaper, in latency, energy, and compute, than serving a model in the seventy-to-175-billion range for these repetitive calls.
💡 Key insight: Capability is task-specific. A small language model that is weaker at open-ended reasoning can still match a large one on a focused, well-defined job, and that is exactly the kind of job most embedded products need.
Edge and embedded use cases
Small language models fit best when the model needs to be close to the data source and the task is narrow. Common uses include on-device assistants, equipment diagnostics helpers, command interpreters, log summarisers, and domain-specific natural-language interfaces for a product with a limited command set.
They also pair well with retrieval-augmented generation on embedded systems. In this pattern the model stays small, but it answers questions using locally stored manuals, service documents, or system logs that are retrieved and supplied as context. This gives an assistant that understands a narrow technical domain without the cost and infrastructure of a large cloud model, and without sending internal documents off the device.
Running a small language model on embedded Linux
On embedded Linux the common path is a runtime that loads a quantised model file and runs inference on the CPU or an available accelerator. The most widely used runtime is llama.cpp, which loads models in the GGUF format and runs efficiently on Arm and x86 CPUs. Ollama wraps llama.cpp with a simpler interface, and ONNX Runtime is a common choice where a model is exported to the ONNX format for use with a vendor accelerator.
A minimal test with Ollama on a Linux machine looks like this:
raghu@techveda.org:~$ ollama run llama3.2:1b
>>> Summarise this sensor log in one sentence: temp rose from 40 to 95 C in 3 minutes then the device reset.
The device overheated rapidly and reset after its temperature climbed from 40 to 95 C within three minutes.
The practical limits on a board are memory and inference speed, and both depend on the match between the model, the quantisation level, the runtime, and any hardware accelerator. A small single-board computer such as a Raspberry Pi can run the smallest models slowly, while a module with a GPU or NPU such as an NVIDIA Jetson Orin can run larger ones at a usable speed. Measure memory footprint and response time on the target hardware, not on a workstation, because a model that runs well on a laptop can be too slow on a low-power board.
System integration matters as much as the model. On a device that also runs control tasks, the inference service should be scheduled so that it does not interfere with latency-sensitive loops or core application work. A common and effective pattern is cloud-edge collaboration: run a small model locally for private, low-latency, first-pass processing, and escalate only the harder queries to a larger cloud model. This matches the heterogeneous design that the NVIDIA paper recommends, where small models are the default and large ones are called selectively.
For engineers building these systems, the surrounding platform work — the kernel, drivers, memory, and the build system — is what makes on-device inference reliable.
Where small language models make sense, and where they do not
Small language models are a good fit when the task is narrow and well defined, and whenever offline operation or data privacy is required. They are a poor fit when the task needs broad world knowledge, long and complex reasoning, or the highest possible accuracy on open-ended questions. A smaller model has a smaller store of facts and a shorter effective reasoning span, and it will make more mistakes on tasks that a large model handles. The correct decision is to match the model size to the task, not to always reach for the smallest or the largest.
Key takeaways
- Small language models are models compact enough to run on one device, usually below about ten billion parameters and often one to four billion.
- Quantisation to 4-bit precision brings a 3.8B model to roughly 2 GB of weights, which makes on-device deployment realistic.
- They are chosen for memory, power, latency, and privacy reasons that rule out large hosted models on embedded systems.
- Evidence from the Phi-3 report and NVIDIA’s agentic AI paper shows small models match larger ones on many narrow tasks at a fraction of the cost.
- Test on the target board, isolate inference from real-time work, and escalate only hard queries to a cloud model.
Frequently asked questions
What is a small language model?
A small language model is a language model compact enough to run on a single device, usually below about ten billion parameters and often between one and four billion. It trades some general capability for speed, low cost, offline operation, and privacy.
How much memory does a small language model need on a device?
It depends on the parameter count and the quantisation level. At 4-bit precision a 3.8B model needs about 2 GB for weights and a 1B model needs under 1 GB, plus additional memory for the working context, so a few gigabytes of RAM is a realistic target.
Are small language models as accurate as large ones?
Not in general. For broad knowledge and long, complex reasoning a large model is more accurate. For narrow, well-defined tasks such as classification, extraction, or summarising a fixed input, a small model can match a large one, which is why they suit many embedded products.
How do I run a small language model on embedded Linux?
Use a runtime that loads a quantised model, such as llama.cpp with a GGUF file, Ollama, or ONNX Runtime for accelerator use. Confirm the model fits in available RAM and measure inference speed on the target board, not on a workstation.
