The phrase “AI at the edge” covers two very different things. One is a Linux-class SoC with a few TOPS of NPU running a vision model — a real computer in a small box. The other is a microcontroller with a few hundred kilobytes of flash running an inference engine measured in kilobytes. Only the second is genuinely new, and it is the one worth understanding.
What fits on a small part is narrower than the marketing suggests, and more useful than sceptics expect. Anomaly detection from vibration or motor current. Keyword spotting for a handful of words. Simple gesture and presence detection. Classifying a sensor signal that would otherwise need hand-written thresholds nobody can tune reliably across units. These are all small models — often a few tens of kilobytes after quantisation.
What does not fit: anything conversational, general-purpose vision, or a model that has to be updated frequently over a constrained link. If a proposal on a microcontroller involves a language model, it is worth asking what problem that solves that a lookup table does not.
The binding constraint is memory, not arithmetic. Even with a hardware accelerator, flash and RAM set a hard ceiling, so most of the engineering effort goes into feature selection, quantisation to eight bits or less, and resisting the temptation to grow the model. That work happens before any accelerator matters — which is why the cheapest way to make edge AI succeed is usually to spend more time on the sensor and the signal path than on the silicon.