Checkout audit methodology

The audit answers one question: can an AI shopping agent find, understand, trust, buy and follow up a product in this store, on behalf of a person? It does not measure design, brand or conversion for human visitors.

The store is judged in seven dimensions, in the order an agent meets them.

Every finding names its source and how many sources agree.

1. Hard checks (facts): deterministic code over what the store publishes (pages, structured data, platform JSON, robots.txt, manifests). Each check returns pass, fail, warn or skipped, with evidence. Skipped means "not probed". It never counts against the store.
2. Conformance (facts): when the store declares UCP or ACP, we create one probe session, read it and cancel it. Every response is validated against the published schemas (UCP 2026-08-25, ACP 2026-04-17, AP2 0.2.0) and the flow rules (totals arithmetic, state order). A session is never completed.
3. Model review: one or more reviewer models answer the model checks with a bounded, read-only tool loop. Store content reaches them marked as untrusted data. A finding needs evidence: a URL on the store and a quote. Findings without evidence, for unknown checks, or citing other sites are rejected. Results are merged per check and product:
   • agreed: every model that judged it raised it (two or more models);
   • majority: more than half raised it;
   • single: one model, or a minority. Models that disagree are listed as dissent.
4. Live runs: a computer-use agent performs scripted shopping tasks in a real browser. We grade the final state as the store reports it (the cart or the order), never the agent's own words.

Scoring is provisional. Per dimension, start at 100 and subtract for each counted failing finding.

• Counted findings: hard, conformance and live findings, and model findings with agreed or majority agreement. A single model finding is shown but scores nothing.
• Cap: a dimension with a counted critical failure scores at most 20; with a high failure, at most 50.
• The floor is 0. Skipped findings never count.
• No overall score is defined yet. The report shows the plain average of the seven dimensions as a guide.

• Sampling: we read the catalog up to three times the sample size, then sample round-robin across product types or categories with a seed fixed per audit. A rerun of an audit judges the same products.
• Coverage (products sampled of total, models completed of requested, live runs completed of requested) is always reported. A report stopped by the budget cap is marked partial.

Rules no setting can change:

• Every fetch goes through a guard that refuses private and internal addresses. The browser agent respects robots.txt.
• An order is placed only for owner or reference audits that ask for it and have an owner test setup (a hidden product and, optionally, a discount code; never a payment credential). Every other run stops at the final review step without pressing the order button.
• The harness, not the model, refuses: clicks that place an order, logins and account creation, typing into payment or password fields, and discount codes the owner did not supply. A bot wall stops the run and becomes a Reach finding.
• Screenshots mask personal fields when they are taken.

• Tasks come from five templates: one exact variant, two items, cheapest under a budget, check the returns policy then buy, and change the cart. Tasks are filled from the sample, seeded per audit. A deep audit runs three tasks.
• Caps: each task has a step cap and a time cap. The audit's budget cap stops further runs.
• Grading order: no overshoot (an order placed when it should not be), then the harness stop, reached the stop point, items, variants, quantities, and price (subtotal within 1% of the expected total, same currency). The first failing check names the failure class.
• Intent match is a small model's 0 to 1 opinion of whether the cart serves the request. It is shown beside the verdict and never changes it.
• A failed run becomes a live finding on the check it reproduces (for example, a wrong variant on the check for unambiguous variant options).

• The index lists reference audits only: the newest finished audit per store and day, with the dimension scores, effort, platform and live-run pass counts.
• MishiPay audits the stores in the index through the same pipeline as everyone else, at the same effort for every store in a round.
• An index entry is a measurement at one date, not a certification.

No reference audits have been published yet.

Last 90 days.

Every report records the spec versions it used. A change to a check, a weight or a cap is a new methodology version.

• May anonymous users get the model review, and with which models?
• Live runs before domain verification, in a no-purchase mode?
• Commercial placement of the audit.
• Shopify connected mode with more scopes (legal policies, shipping)?
• Third-party model providers, and the privacy wording that goes with them.
• Scoring: the weights, the caps and an overall score are provisional until these are decided.