What on-device AI means, and why a small model can be enough

On-device AI means the model runs on your own computer or phone, so what you give it is processed there instead of being sent to a company's server. Whether that is practical depends on the job. A general chatbot needs billions of parameters and gigabytes of memory. A narrow job, such as judging whether a paragraph is likely AI-written, can be done by a model small enough to ship inside a browser extension.

The worked example here is Botfilter, a browser extension that marks likely AI-written text and whose model is one 23.6 MB file bundled inside the extension. It runs on your device, needs no account, and sends no page text, scores, or usage data anywhere.

On-device AI means the model runs on your hardware

Every AI feature has two parts: the model (a file of learned numbers, called weights or parameters) and the code that feeds input through it, called inference. “On-device,” “local,” and “offline” AI all describe where inference happens. When it happens on your machine, the input never has to travel.

ONNX Runtime, the engine Botfilter uses, lists the benefits of running models in the browser: it can be faster, it works offline, it costs the developer less, and it helps privacy because “the data never leaves the device for inferencing.” Google makes a similar case for the model built into Chrome, citing low latency, lower server costs, and user privacy.

The words are often used loosely, so it helps to separate three claims:

• Local inference. The model runs on your hardware, which is a fact about where the computing happens.
• No data leaves. The product also sends nothing else: no copies of input, no results, no usage logs. This is a separate promise about the rest of the code.
• No account. You can use it without signing in or identifying yourself, which is a third, separate promise.

A product can do the first without the other two. A local model can still report what you did to an analytics service, or require a login. When you evaluate a “private AI” tool, check all three.

What leaves your device in each setup

The practical question is where your input goes. Here are five common setups, from the most data sent to the least.

Cloud inference is not careless by default. Providers can encrypt traffic, limit retention, and sign contracts about data use. But those are promises you have to trust. Local inference removes the question for the model step, because there is no copy to retain. AI detectors are a clear case: most of them are web services that receive the text you check. For how Pangram, GPTZero, and Originality.ai handle it, see where your text goes with each AI detector.

Ways to run AI on your own machine today

If you want a chatbot or another model that runs locally, there are four common routes. Each project below describes itself in its own documentation; this is not a ranking.

• Local LLM apps. llama.cpp is an open source engine for “LLM inference in C/C++” that runs on a wide range of hardware. Ollama runs models behind a small server on your own computer, which other apps reach at a localhost address. Before you rely on either for privacy, confirm that the model you chose is stored on your disk.
• Libraries that run models in a web page. Transformers.js runs Hugging Face models in the browser “with no need for a server,” using ONNX Runtime on the CPU by default and WebGPU when a developer turns it on. WebLLM runs chat models in the browser on WebGPU and says everything runs “with no server support.”
• Models built into the browser. Chrome's built-in Gemini Nano asks for at least 22 GB of free space on the drive that holds your Chrome profile, and either a GPU with more than 4 GB of video memory or 16 GB of RAM with 4 CPU cores.
• On-device first, with a cloud step. Apple calls on-device processing the cornerstone of its approach to privacy, and says Apple Intelligence needs larger models in the cloud for some requests. Those go to its Private Cloud Compute servers, which Apple says keep no user data after the response is returned.

The last route is the one to read closely. “On-device” in a product description can mean on-device first. For each feature, ask whether it can ever send your input to a server, and what happens to it there.

A private LLM and a small classifier are different tools

People searching for a “private LLM” usually want a chatbot that runs locally. A large language model (LLM) generates text one token at a time and can take on almost any task you describe in a prompt. That generality costs size. Microsoft describes Phi-4-mini, one of the smaller current LLMs, as a 3.8 billion parameter model with a 128,000-token context. At 16 bits per parameter, its weights alone come to about 7.6 GB; at 8 bits, about 3.8 GB.

A classifier does something narrower. It reads text and returns a score for a fixed set of labels. Botfilter's model has two labels, human-written and AI-written, and cannot generate a word. It is not an LLM, and it does not send your text to one.

Phi-4-mini has about 167 times as many parameters. That gap is why Botfilter can load in the background of a browser without asking for a graphics card or gigabytes of disk.

Why a small model can be enough for one narrow task

A general model has to be ready for any request. A classifier only has to be good at one decision, so it can start from a compact language model and learn that decision from labeled examples.

Botfilter starts from MiniLM, a compressed Transformer from Microsoft Research built by training a small “student” model to mimic the self-attention of a larger “teacher” model's last layer. Botfilter uses a 6-layer version derived from Microsoft's 12-layer MiniLM, with a hidden size of 384. It was then fine-tuned on 697,474 labeled texts, 671,247 human-written and 26,227 AI-written, drawn from sources such as news, Reddit posts from before 2017, work email, and generations from 15 AI models. The final training run took about 8 minutes on one GPU.

On a sealed test set the model never saw during training or tuning, at the default Balanced setting, it produced these results:

Those numbers are for one task on the kinds of text it was trained and tested on. They do not mean a small model can do what an LLM does. They show that for this one fixed job, a 22.7 million parameter model reaches these rates on the text it was tested on. The full data sources, training settings, and per-model results are in how the Botfilter detector was trained.

How a browser runs a model: WebAssembly, WebGPU, and ONNX

Three open technologies make in-browser inference work.

• ONNX is “an open format built to represent machine learning models.” A model trained in one framework, such as PyTorch, can be exported to a single .onnx file and run by any compatible engine.
• WebAssembly is a compact binary format browsers run on the CPU. The WebAssembly project describes it as a memory-safe, sandboxed environment that follows the browser's same-origin and permission rules.
• WebGPU is the browser API for the graphics card. Unlike its predecessor WebGL, it supports general-purpose computation, which MDN notes matters for machine learning. MDN also lists it as limited availability: not every browser supports it yet.

ONNX Runtime Web ties these together. It can run a model on the GPU through WebGPU, or on the CPU through WebAssembly, and its documentation notes that WebAssembly supports every ONNX operator while the GPU paths support a subset. That is why Botfilter keeps WebAssembly as its fallback.

In Botfilter, a content script reads the text of posts, emails, and paragraphs on the page, starting with what is on screen, and skips anything you are typing. A worker loaded from inside the extension scores it, on WebGPU when the graphics card runs a test batch fast enough and on WebAssembly otherwise. After about 10 minutes with nothing to score, the worker closes and the model's memory is freed. The full path is in how Botfilter runs AI detection on your device.

The model is stored in 8-bit integers rather than 32-bit floats, which is how 22.7 million parameters fit in 23.6 MB. An automated test runs the shipped file through ONNX Runtime Web on the CPU path and checks that its scores match the training code's to within a small fixed tolerance, listed in the training article.

How to check what a local AI tool sends

You do not have to take a privacy claim on trust when the software runs in your browser. These steps work for any Chrome extension or in-browser AI tool. Botfilter is in development and not yet in the Chrome Web Store, so they are how you can check it once it is available.

1. Open chrome://extensions and the extension's details. Chrome shows its site access there, and other permissions that come with a warning, but not every permission. Look for access you did not expect.
2. Turn on Developer mode. Under “Inspect views,” open the extension's background or offscreen page and select the Network tab, then use the tool on a page. Addresses starting with chrome-extension:// are files inside the extension; any other address is data crossing the network.
3. Set that Network tab to Offline, or turn off Wi-Fi, and use the tool on a page you saved. A tool that runs its model locally keeps working.
4. Under Application, look at IndexedDB and extension storage to see what the tool keeps on your disk.

For Botfilter, the details page will show access to read the sites you visit; its other permissions are storage, offscreen, scripting, and alarms, and it does not use Chrome's sign-in (identity) permission. What each store holds is in Botfilter's own check-it-yourself walkthrough.

What running locally does not fix, and when to choose it

Running locally answers the privacy question. It does not answer the accuracy question. A local model is exactly as right or wrong as it would be on a server, and a small one is trained for one kind of input.

For why no detector should be used to accuse someone, read how accurate AI detectors are, and what false positives mean. To decide between local and cloud AI, four questions settle most cases:

• Is the input private? Email, messages, and anything about other people favor local inference.
• Is the task narrow? Labeling, sorting, and filtering often fit a small local model. Open-ended writing and reasoning still favor a large model.
• Does it run constantly? Scoring every post on every page would mean sending the text of every page you read to a server. Locally, it costs only your own CPU or GPU time.
• Does it need to work offline? Only local inference does.

Botfilter answers yes to all four, which is why it was built as a small local model rather than a cloud service. See what Botfilter does, and read Introducing Botfilter for the three display modes and when it will be available.