How to Review an Edge LLM Benchmark Without Fooling Yourself

A Jetson Nano and Ollama benchmark review appeared among DEV's top articles on 2026-07-12. Edge inference experiments are valuable because they make constraints physical: memory is finite, heat accumulates, and a product cannot hide behind average cloud latency. They are also easy to overgeneralize. Before comparing numbers, capture the test envelope. device_model: "<exact board and memory>" power_mode: "<mode and watt limit>" cooling: "<passive/fan, ambient temperature>" os_image: "<name and version>" runtime: "<Ollama version or other runtime>" model: "<repository, tag, parameter count>" quantization: "<exact format>" context_length: 0 prompt_tokens: 0 generated_tokens: 0 network: "offline after model download" “Jetson Nano” and a model family name are not enough to reproduce a result. Model tag changes, quantization, context length, and power mode can move the outcome. For a mobile or edge product, I would run at least these cases: Case Why it matters cold start after reboot packaging and model-load cost first request time to first useful output ten repeated requests cache and thermal behavior long context memory pressure and degradation background/foreground cycle lifecycle recovery network unavailable true offline behavior cancellation mid-generation cleanup and user control Report median, p95, and worst observed time to first token; generation rate; peak resident memory; device power; temperature; throttling; and task success. A single tokens-per-second number misses the experience. Use language such as: Observed: the declared model completed 10/10 prompts under the stated setup. Inferred: this may support a short offline assistance flow. Not established: battery impact on a phone, performance on another board, or accuracy on production tasks. If accuracy matters, keep a fixed task set and score outputs independently of performance. Faster wrong answers do not improve the product. Define the acceptance envelope before testing: p95 first useful output <= target AND peak memory <= device budget AND no thermal throttle during repeated run AND task pass rate >= product threshold AND cancellation releases resources <= target Replace each target with a number derived from the actual user flow. If the device misses the envelope, choose a smaller model, shorter context, remote fallback, or a narrower feature. The public MonkeyCode repository documents native mobile support with synchronized PC/mobile data while tasks run in server-side environments. That is a different placement choice from on-device inference, but the same measurement discipline applies to mobile continuity, network use, latency, and recovery. This article does not report a MonkeyCode device test. Disclosure: I contribute to the MonkeyCode project. The MonkeyCode description comes from public documentation; the benchmark protocol is independent and has not been executed here. Mobile and edge developers can discuss supported workflows in the MonkeyCode Discord. The team can also confirm current free model-credit availability, eligibility, and usage limits for hosted evaluation. The best benchmark is not the largest number. It is the one that lets another person reproduce the product decision.