Research method

A reproducible protocol for measuring AI recommendations

This is a research protocol, not a published customer or market benchmark. Use it to specify observation collection before seeing the answers.

Sivlo · Updated

1. Register the question and sampling plan

Define the market, language, product category and bounded set of buying situations. Retain exact question wording and the selection rationale. Separate branded checks from unaided discovery questions. Give a changed panel a new version.

2. Retain the observation and environment

For each attempt retain question ID, timestamp, service and model, API or consumer surface, search setting, country, language, repeat and panel version. Preserve the original answer and visible sources. Record errors, timeouts and observed cost rather than replacing missing observations with zero.

Do not send personal customer messages or confidential material into a public benchmark. Use market questions and obtain permission before publishing identifiable customer evidence.

3. Define counting rules in advance

Specify when a name counts as a mention and when the business is actually recommended. Treat negative and ambiguous recommendations consistently. Manually review a classification sample and report disagreements. A visible source is an observation, not access to a model’s internal trust score.

4. Repeat and assess comparability

Repeat the same panel at different times. Keep service settings and market fixed. Report model or collection-method changes. Calculate numerators, denominators and uncertainty separately by service and market. A before-and-after change alone does not identify an intervention effect.

5. Publish the method alongside results

Publish collection dates, sampling, coverage, exclusions, repeats, counting rules and limitations. Share answer examples only when permitted. Disclose the publisher’s commercial relationship to compared businesses. If observations have not yet been collected, publish the protocol without invented results.

Measure your own market

See who AI recommends in your market, why, and what to change.

Request a demo