Evidence, Not Answers: Why Our VLM Never Made the Final Decision

There is an obvious way to build shelf recognition today: take a photo of a store shelf, send it to a large vision-language model, and ask, "Which products are on this shelf?"

We deliberately did not build it that way.

Over the past year I built a shelf-recognition system for a regional dairy producer and its distribution company. Merchandisers photograph shelves in stores; the system turns each photo into share of shelf, product presence and price data. Everything ran self-hosted on a single NVIDIA H200.

The most important architectural decision we made was this: the vision-language model extracts evidence, but it never makes the final decision. This article explains why, and what we built instead.

Why we did not "just ask the VLM"

Our catalog had 1,383 products: 156 of our own and 1,227 from other brands. The hard cases were never "is this yogurt or milk?". The hard cases were siblings:

the same kefir in 450 g and 900 g packs;
the same product in a canister and in a PET bottle;
a 930 g canister and a 1900 g canister that look almost identical in a photo;
our product next to a competitor product with similar colors.

Ask a general-purpose model to name the product, and you will get a name. What you will not get is a reliable signal that it is guessing. That is the problem. For shelf analytics, a confident wrong answer is worse than no answer. If the system counts a competitor product as ours, share-of-shelf numbers go to management and they are wrong. Nobody sees the error, because the number looks fine.

So we designed the system around one rule: every decision must be backed by evidence, and when evidence is weak, the answer is "unknown".

The pipeline

Each shelf photo comes from an offline-first Android app and goes through seven steps:

Detection. GroundingDINO finds the product boxes.
Reading. Qwen-VL, used as an OCR engine, reads the text on each package: brand, weight, fat percentage, flavor.
Catalog search. Qwen3-Embedding and Qdrant search the product catalog by text.
Visual search. DINOv2 and ArcFace search a gallery of confirmed product crops by image.
Fusion. A deterministic layer combines the signals by fixed rules.
Guardrails. Weak or contradictory evidence is rejected.
Result. An exact SKU, a brand only, or unknown.

In parallel, price tags are detected, read by a VLM and linked to the nearest product box.

A note on terms: below, "OCR" means the text a vision-language model reads from each product crop. We used Qwen-VL as our OCR engine — first Qwen2.5-VL-72B, later Qwen3-VL-32B in FP8 — with EasyOCR as a fallback. Reading text is exactly the kind of narrow task where a VLM is excellent. Deciding which product it is was a different story.

Analysis is asynchronous. A photo takes roughly 10–30 seconds to process, but the merchandiser never waits: the app stores photos locally and uploads them when the network is available. Capture is decoupled from inference.

Principle 1: The model extracts, the rules decide

The fusion layer is a short ladder of rules. The rules are checked from top to bottom, and the first one that matches wins:

OCR + visual. OCR read a brand, and the nearest visual neighbor has the same brand.
OCR + catalog. OCR read a brand, and the catalog search found a product of that brand.
Brand only, no SKU. OCR read a brand, but nothing confirms it.
Rejected. OCR read only generic words, such as "milk" or "fresh".
Catalog. The catalog search score is 0.55 or higher.
Catalog + visual. Catalog search and visual search agree on the brand.
Visual. Only the visual search is confident.
Weak catalog. There is only a weak catalog match.
Unknown. Nothing matched.

Two details matter here.

First, rule 3: if OCR reads a brand but neither search confirms it, the system returns the brand without a product code. It never returns a product code of a different brand than the one printed on the package.

Second, the VLM's own confidence is not used anywhere. In one of our audits we wrote down a blunt note: the model's self-reported confidence was "always high." It was useless as a signal. What the model reads from the package, such as a weight or a brand, can be checked against the catalog. How confident the model feels cannot.

On our golden set — hundreds of field photos, labelled across stores and re-labelled over six months, matched at IoU 0.5 — the full pipeline reached 95.8% brand precision and 73.1% SKU precision end to end. We measured brand and SKU separately because they serve different reports: share of shelf needs the brand, assortment analysis needs the exact SKU.

Principle 2: Solve siblings with specific evidence, not a bigger model

Every sibling problem got its own small, measurable resolver. Each one uses one piece of evidence the generic pipeline ignores.

Weight from the package. The same product family comes in several weights. We read the weight from the crop and "snap" it to the only family member with that weight, within ±5%. On a stratified golden set, the high-confidence gate was correct in 27 of 28 cases (96%). The rule deliberately works only inside one product family: weight cannot tell two different brands apart.

Shape of the box. Two canisters, 930 g and 1900 g, look the same in a photo. Our first attempt used the relative height of the box and was no better than a coin flip (AUC 0.53). Then we looked at the width-to-height ratio of the bounding box: the small canister is narrow (median 0.35), the large one is wide (0.49). That single feature gave AUC 0.89–0.90 and 92.4% / 90.8% accuracy on two product families (n = 181). No model training was needed.

Package form from the VLM. We asked the VLM one narrow question: what is the package type? On audited results it was correct in 92.7% of cases. But we measured it per package type and kept it only where it was reliable: PET bottles and buckets were at 93% on 401 crops. One package type was switched off after an audit of 313 decisions, and families that include canisters were excluded completely, because the model confused canisters with PET bottles.

The price tag: useful, but never a vote on identity. A price tag carries the product name, weight, fat percentage and price, so it is tempting to use it to identify the product. We deliberately did not. In real stores tags are often outdated, or belong to the product that stood there last week. So the tag answers one question only — what does this product cost — and never changes which product the system thinks it is.

The pattern is the same every time: a narrow question, a measured answer, and a scope limited to where the measurement holds.

Principle 3: "Unknown" needs a reason, and the reason needs a fix

Returning "unknown" is only useful if you know why. Every unknown box in our system got one of 19 reason codes, and every code pointed to a lever — the thing that would actually fix it. The main groups:

OCR empty or unreadable → improve photo quality.
Product not in the catalog → add it to the catalog.
No close visual neighbor → add product crops to the gallery.
Brand known, exact SKU not → needs text evidence, such as weight or flavor.
Guardrail rejected a contradiction → nothing to fix; this is a healthy reject.

This mapping saved us from at least one expensive mistake. Our intuition was that visual unknowns could be fixed by adding more product crops to the gallery. We measured it. When the nearest visual neighbor was very close (distance ≤ 0.20), it gave the right brand 83% of the time, but the right SKU only 49% of the time. At distance 0.20–0.34, it was 42% for the brand and 15% for the SKU.

Siblings look alike; that is the whole problem. More images of similar packages do not separate them. The real lever was text: weight and flavor from OCR. We stopped investing in gallery seeding for this class of errors.

We also learned that a reason code can hide problems. One code, "brand mismatch — healthy reject," looked like the system working correctly. An audit showed that it was also hiding catalog gaps and false rejections. We split it into subclasses so each one pointed to a real fix.

Unknown boxes also went to a review inbox. When a reviewer confirmed a product, the crop was added to the confirmed visual gallery. The system improved on its own failure cases without retraining any model.

Principle 4: Guardrails can hurt too — ship them in shadow mode first

Guardrails sound safe. They are not automatically safe.

Our brand-verification guardrail rejects a decision when the brand read by OCR contradicts the assigned product. When we tested its second iteration on every box, it would have removed 93% of correct recognitions of our own products. Our own brand name was often misread by OCR, so the guardrail kept "catching" contradictions that were not there. The third iteration added a trusted-skip rule for these cases and reduced the loss to 1%.

We caught this before it touched production data, because every risky mechanism in the system had three modes: off, shadow and active. In shadow mode, the new logic runs on real traffic and writes its decision into a separate audit table, next to the live result, without changing anything the business sees. Each mechanism had its own audit table, and rollback was a single switch back to off.

The decision to go from shadow to active was never a feeling. It was a numeric gate, announced before the measurement and recorded in the code with a date.

Principle 5: Every decision must be explainable

For every box, the system stored its full decision trail, not only the final SKU:

the box, detector confidence and the OCR text;
the brand and attributes read from the package;
the catalog search score and the visual distance;
the match source — which rule made the decision;
guardrail verdicts and the reason for any unknown;
the thresholds that were active at that moment.

When someone asked "why did the system say this is product X?", we could answer in one minute: which signal decided, which rule matched, and which evidence supported it. When the answer was wrong, the box went into the golden set, so the same error could not come back silently.

What the business got

Recognition was never the product. The product was data for merchandising and sales management:

Share of shelf by brand and store, with low-share stores flagged, plus shelf level (top, eye level, bottom).
Product presence: in how many stores each of our products was actually on the shelf.
Price intelligence from price tags: our price versus competitors by category, and the markup each store put on our products. Each tag was linked to the nearest product box, preferring a tag directly below the product within a fixed radius. Prices then went through an outlier filter, because OCR sometimes read a barcode or a date as a price. One known limitation: if a store puts the wrong tag under a product, the system repeats the store's mistake. Price tags are small, so we cut photos into a 3×3 grid and ran detection on each tile. Recall on price tags went from 16% to 65%, at the cost of nine extra detector calls per photo.
Drift-based compliance: each store visit was compared with that store's own history, and stores with a sudden drop in our share or missing key products were flagged first.

When would I let a VLM decide?

I am not against VLMs. We used one in production every day. But the question is not "is the model good?". The question is "what happens when it is wrong, and will anyone notice?".

I would let a VLM make the final decision when errors are cheap, visible and easy to correct — for example, a draft description that a human reviews. I would not let it decide when errors are silent and flow into numbers that people trust, like share of shelf, prices or stock.

In those systems, the architecture is the product: narrow questions to the model, evidence you can check, rules you can read, "unknown" with a reason, and every change measured in shadow mode before it goes live.

The technical case study, with the architecture and runnable examples, is on GitHub: github.com/swd07/retail-shelf-detection

I'm Eduard Kharaev, an applied AI engineer. I build production AI systems: retrieval, computer vision, self-hosted LLMs and agents with strict boundaries. I'm based in Tbilisi and open to new roles. You can reach me at haraev87@gmail.com or on Telegram: @Edharaev.