Small models.
Exact purpose.

Small describes the deployment as much as the model.

A small language model (SLM) is a language model built or selected to work within a tighter compute budget than a broad, large-scale system. The label does not give one universal parameter limit. In the Phi-3 technical report, Microsoft researchers describe a 3.8-billion-parameter model as Phi-3-mini and test a quantized version on a phone. Google’s current Gemma 4 deployment guide lists some variants for mobile and others for laptops, desktops, and servers. These examples use size in relation to a model family, task, and device—not as a shared quality score.

Parameter count is one useful clue, but it does not tell you how much memory an application needs, how quickly a model answers, whether it handles your language, or whether it fits your task. A compact model can be a poor choice for a workload it has not been tested on. A larger model can be wasteful when a smaller one meets the requirement.

A smaller file is not the whole runtime.

Model size can be reduced in different ways. A team may train a compact architecture, distill a smaller model from a larger teacher, prune selected weights, or quantize its numerical values to use fewer bits. These are different techniques. Quantization can make weights more compact without changing the number of parameters; its effect on quality and speed depends on the model, quantization method, runtime, and device.

Distillation trains a student using selected signals from a teacher. It changes the student through training, while quantization changes the numerical precision used to represent weights or calculations. The two can be combined. Read how teacher and student models differ ↗.

For a closer look at bit width, weight-only formats, and what to measure on the target runtime, read the quantization guide ↗.

As a rough estimate, 4-bit weights for a 4-billion-parameter model represent about 2 GB of raw weight values. That is a floor for the weights, not the memory needed by the complete application. Runtime buffers, temporary calculations, operating-system use, and the context’s key-value (KV) cache all need room too. Longer prompts and outputs can make that cache grow; this guide shows how to estimate it ↗.

Mixture-of-Experts models add another wrinkle: a model may activate only part of its parameters for each token while still needing many or all expert weights available in memory. Google’s Gemma 4 documentation describes this for its 26B A4B variant. Do not size a device from the “active parameters” label alone; check the model’s deployment documentation and measure it in the intended runtime. For a closer look at this architecture and its router, read the guide to Mixture-of-Experts models ↗.

Local execution changes the data path and the operating budget.

A compact model may make it practical to run inference on a phone, laptop, or other constrained machine. Depending on the model and application, local execution can reduce the need for a remote model call, support work without a network connection, or shorten the round trip. It can also put new demands on device memory, battery, heat, and updates.

One concrete example comes from the Phi-3 authors: they report running a 4-bit, 3.8B Phi-3-mini offline on an iPhone 14, with the model occupying 1.8 GB and generating more than 12 tokens per second in their setup. That is a result for their particular model, phone, and software—not a forecast for another model or device.

Local inference is not automatically private. A surrounding app can still sync history, send diagnostics, contact hosted tools, or expose files. Trace the complete path before handling sensitive information; this guide maps what local execution does and does not protect ↗.

Start with work you can describe and check.

Compact models are worth evaluating for bounded tasks such as classifying a request, extracting known fields, drafting a short response, or providing a device-side suggestion. Those are starting points for a test, not promises that a given SLM will succeed. Include messy inputs, uncommon language, missing facts, and examples where the model should stop or ask for help.

For open-ended research, complex multi-step reasoning, code changes, or long and varied documents, the trade-off may favor a larger model, retrieval, tools, or a person in the loop. Retrieval can supply current material to a smaller model, but the system must still find the right evidence and produce a supported answer. Compare the full workflow rather than assuming that model size alone decides the result.

A hybrid system can use a compact model for routine cases and send difficult or uncertain cases to a more capable model or a reviewer. Make that routing visible and test it: a mistaken confidence score can send the hardest request down the cheapest path. A remote fallback also changes where information goes, so include it in the privacy review.

Measure accepted work on the target hardware.

1. Use real examples. Include ordinary requests, edge cases, out-of-scope questions, and examples where a person should take over.
2. Check task quality. Measure correctness, required format, unsupported claims, and recovery from missing or ambiguous input.
3. Measure peak memory. Test the longest realistic context and output in the actual runtime; do not count only the downloadable weight file.
4. Record response time and power. Measure the first-token wait, generation pace, sustained use, heat, and battery on the device people will use.
5. Trace the data path. Check network activity, local storage, logs, permissions, model sources, licenses, and update behavior.
6. Compare cost per accepted result. Include hardware, maintenance, retries, escalation, and human review—not only a model’s listed size or token price.

When comparing candidates, hold the task, inputs, and acceptance rules steady. A public benchmark can help narrow the search, but your own evaluation decides whether a particular model and device combination is fit for use. For the broader decision, read how to choose an AI model for real work ↗ and how to read benchmark results ↗.

Fit the model to the system around it.

Follow the compute through inference and memory ↗, compare models against the actual task ↗, and trace the complete data path ↗.