Open-weight models that fit edge hardware, and their licences

Running a language model locally is first a memory problem and second a licensing one. Mistral Small 3 is 24 billion parameters under Apache 2.0, with the 3.1 release adding vision understanding and a 128,000-token context window. Microsoft's Phi-4 is 14 billion parameters under the MIT licence and aimed at low-latency reasoning. For multimodal work on constrained hardware, Hugging Face's SmolVLM is 2 billion parameters, with a 256 million parameter variant, also Apache 2.0.

Permissive licences are not universal. Meta's Llama 4 models ship under the Llama 4 Community License, a custom agreement rather than an OSI-approved licence, and Google's Gemma models carry Google's own terms of use. If your product embeds the weights, read the licence before you benchmark the model.

On the serving side, three projects cover most deployments. llama.cpp is MIT-licensed C and C++ built on ggml, with CPU, CUDA, Metal and Vulkan backends. Ollama, also MIT, wraps model download and serving behind a CLI and local HTTP API. For phones and embedded targets, Google's LiteRT — Apache 2.0, and the successor to TensorFlow Lite since September 2024 — is the on-device runtime.

Hardware choice follows from memory. A 24-billion-parameter model quantised for local use needs a machine with headroom: DGX Spark publishes 128 GB of unified memory at 273 GB/s, and Ryzen AI Max+ 395 allows up to 112 GB of its 128 GB to be allocated to the GPU. Accelerators with their own small memory pool, such as the 4 GB or 8 GB on Hailo-10H, target smaller models rather than these.

Whatever the datasheet says, measure tokens per second with your own prompt lengths on the exact quantised build you intend to deploy.