Visa’s shopping benchmark tests understanding before payment
The research exposes inconsistent buying decisions; checkout and payment testing are planned extensions.
Visa’s November 5 benchmark evaluates whether models discover a shopper’s requirements and choose appropriately, including deciding against a purchase. It tested 22 models from 12 labs across 15 shopping scenarios.
The overall pass rate was 57.2%, and all three attempts passed in only 34.2% of model-scenario combinations. Visa plans to extend testing to checkout, payment and order placement. The present results therefore measure shopping decisions, not end-to-end payment reliability.
For teams building agent commerce, the implication is to validate intent before authorizing execution. Spending limits address how much an agent may spend; requirement discovery and a valid no-purchase outcome address whether it should buy at all.