Getting 40% more token throughput from our local product-enrichment pipeline
At Yuh Gettin’ Tru, we use a local model to help turn retailer product data and images into descriptions for semantic search. I recently switched the model build running under oMLX on my MacBook M4 Pro. Observed throughput went from 18.7 to 26.2 tokens per second, with near-identical results for our

workload. That’s about a 40% improvement, using the same base model. The newer build is also smaller. This seemed worth writing up because the improvement came from revisiting the model and inference software we were already using, rather than changing the application architecture.
Yuh Gettin’ Tru helps people find products across retailers in Barbados. One of the challenges is that different stores describe the same kinds of products very differently. A retailer might list a “refrigerator”, while someone searches for a “fridge”. Product titles can be abbreviated, descriptions can be sparse, and useful information might only be visible in the product images.
Our web-processing pipeline has three independently runnable stages: Crawl product pages ↓ Extract structured product data ↓ Enhance descriptions and regenerate embeddings The crawler captures HTML and stores it for later processing. The extraction worker then uses store-specific profiles to pull out fields such as the title, price, description, brand, SKU and images. That extraction step does not call an LLM. It uses structured data, metadata and configured HTML selectors.
The model comes in at the enhancement stage, once we already have a product record. The enhancement worker sends the existing product data and product images to a local Gemma model through oMLX’s chat-completions API. The input includes the title, available metadata, the first 500 characters of the existing description, and up to five product images. Where available, we use our stored thumbnail for the primary image.
We ask the model to produce a compact semantic description, targeting 300 to 350 characters, containing: The product’s identity, brand, type and important specifications. Concrete details about its use, composition, dimensions or compatibility. Likely search terms, including common synonyms. For example, our prompt explicitly mentions pairs such as “sofa/couch” and “TV/television”.
Someone should not need to guess which word a retailer used. The response is requested as schema-constrained JSON, with a product_name, a semantic_vector_description, and an empty attributes object reserved for future use. We then generate a BGE-large embedding from the semantic description and store both. So the task is quite specific: produce short, useful retrieval text from product data and images.
We are not asking the model to write a long sales description or answer an open-ended question. Here is what changed: Previous build New build Model gemma-4-E4B-it-qat-mxfp4 gemma-4-E4B-it-oQ4e-mtp Observed token throughput 18.7 tok/s 26.2 tok/s Model size 6.2 GB 5.1 GB The throughput figures come from real workloads across several million tokens and thousands of requests, rather than a quick benchmark with a handful of prompts. Both builds use the same base model at roughly comparable quantisation levels. The newer oQ4e build uses mixed precision, allocating more bits where they matter rather than applying the same precision everywhere.
It also includes a multi-token prediction, or MTP, head, which oMLX can now use. The size figures above describe the model files, not peak memory consumption while serving requests. A useful way to think about the MTP head is as a small “advance scout”. It proposes likely upcoming tokens ahead of the main model.
The main model then verifies those proposals and determines what gets accepted. When several proposed tokens can be accepted together, generation can progress with fewer sequential decoding steps. There is additional work involved in making and checking the proposals. Whether that pays off depends on the workload and how well those proposals match what the main model would produce.
There are also two changes bundled into this particular comparison: the quantisation and the MTP support. These measurements compare the complete builds. They do not isolate how much of the improvement comes from each. An important caveat from our workload checks: the MTP build was slower on a text-only prompt.
The improvement here is on the multimodal product-enhancement path, where we supply images alongside the product data. I would not assume that the same switch will improve every application. The increase from 18.7 to 26.2 tok/s is about 40%. That does not mean the entire pipeline processes 40% more products per hour.
A product enhancement also involves downloading images, preparing the request, processing the input, generating an embedding and committing the result. Crawling and extraction have their own constraints. For this workload, we care about the time taken to produce and persist a usable enhancement, as well as the model’s token rate. The repository includes a comparison script that runs both models through the actual enhancement code path.
It uses real products with images, the production prompt and the same response schema. Application-level response caching is disabled for the comparison. It reports wall-clock timings and product throughput, alongside basic output checks such as whether a parsed result contains a product name and description, and how closely the description follows the length target. Those checks are useful, but they are not a complete factuality or search-relevance evaluation.
In the workloads I’ve been running, the results from the two builds have been near-identical. That is an observation about this task, not a claim that the models are interchangeable for everything. A faster model helps, but we also try to avoid generating descriptions unnecessarily. The pipeline computes a hash of the enhancement-relevant inputs: the title, brand, category, SKU, image hash, description excerpt and additional image references.
Price and currency are deliberately excluded. A price change should not require another model call to describe what the product is. There is also a version check when an enhancement is committed. If extraction has changed the relevant product inputs while the model was working, the stale enhancement is not written over the newer version.
That lets the stages run independently without treating a slow model response as the latest truth. The prompt is also split into static system instructions and per-product content, allowing oMLX to reuse the shared prefix where supported. These details remain useful regardless of which model build we run. The other part of this is easy to overlook: the inference software needs to support the features included in the model build.
Downloading a model with an MTP head is not enough if the runtime cannot use it. In our case, updating oMLX was part of making that capability available. Our enhancement worker selects its model through OMLX_MODEL: export OMLX_MODEL=gemma-4-E4B-it-oQ4e-mtp uv run ygt enhance-worker The model is resolved when the worker starts, so changing the environment also means restarting the worker. Checking the model ID in the startup log is worthwhile before collecting comparison numbers.
If you have a local model doing useful background work, revisit it occasionally. Check the available builds, update the runtime, and compare them using your actual inputs and output requirements. For our product-enrichment workload, that meant moving from 18.7 to 26.2 tok/s on the same MacBook, with a smaller model and no meaningful difference in the descriptions I was getting back.
Key Takeaways
- •That’s about a 40% improvement, using the same base model.
- •This story was reported by Dev.to, covering developments in the dev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article:
Read Full Article on Dev.to →


