robots.txt and AI crawlers: the setting to check
Blocking AI assistants’ crawlers means disappearing from their answers. The check takes two minutes.
Visibility, position, share of voice, share of citations, sentiment, technical score: what each one is for.
Because AI visibility has several sides: being cited, being cited early in the answer, carrying weight against competitors, being backed by good sources, being described in the right tone, and having readable pages. A single score flattens all of them.
Each of the six metrics below answers a different question. Their definitions are those of iPand’s methodology; other tools use similar names, but the logic is the same.
One rule applies to all six: read them across many answers, never just one. SparkToro had the same questions put to ChatGPT, Claude and Google nearly three thousand times, and found less than a one-in-a-hundred chance of getting the same list of brands twice[1]. A metric only means something as a trend.
| Metric | The question | How it is calculated | The trap |
|---|---|---|---|
| Visibility | Am I cited? | Answers that mention your brand ÷ answers collected | Reading it on too few questions |
| Position | Where does my name appear? | Average mention rank, on answers where you are cited | Treating it as a ranking |
| Share of voice | How much weight do I carry? | Your mentions ÷ mentions of the same type | Forgetting who the others are |
| Citation share | Which sources do AI engines rely on? | Citations of a domain ÷ all citations | Only looking at your own site |
| Sentiment | What tone is used about me? | Average tone of the passages about you | Scoring the whole answer instead |
| Technical score | Are my pages readable? | Average of the criteria, weighted by importance | Thinking it is enough to get cited |
Definitions from iPand’s methodology[2].
Visibility is the share of collected answers that mention your brand. It is the baseline: if you are not cited, the other five have little to say.
It is calculated over a set of questions, engines and days: for instance, every ChatGPT and Google AI Overviews answer on your tracked questions over the last month. For Meeple Junction, our fictional store in Denver, being cited often on advice questions and never on buying questions would say far more than one overall score.
It is also the metric SparkToro considers the most sensible: a presence rate calculated over dozens or hundreds of repeated questions holds up, where the detail of any single answer does not[1].
Position is the average rank at which your brand is mentioned, calculated only on the answers where it appears. Being named first does not carry the same weight as being named fifth.
Read it with care. SparkToro’s study shows that the order of brands changes from one answer to the next, to the point that its author calls any “ranking position in AI” meaningless[1]. An average position over many answers is still useful to spot a trend; a position taken from a single answer proves nothing.
Sentiment sums up the tone of the passages about your brand, not of the whole answer. Being described as “pricey but reliable” is not the same as “best avoided”.
Scored on the whole answer, it would mislead you: a positive answer about your sector can still contain one negative line about you. Also look at the recurring themes, such as price, welcome, selection or delays: they are often the themes of your customer reviews, and our article on reviews explains how to shift them.
The technical score, which iPand’s methodology calls the optimization score, is the average of the criteria that apply to a page, weighted by importance, across the 24 criteria of the framework. It tells you whether an engine can read, understand and reuse your page.
It is the only one of the six you fully control, and the only one that does not measure answers. A good score guarantees no citation; a poor score, on the other hand, can prevent one: a page blocked by robots.txt or invisible without JavaScript will not be cited. Our article on robots.txt and AI crawlers covers the most common case.
Often enough to smooth out the variability of answers, and for long enough to see a trend: daily collection read weekly or monthly is a good balance. A one-off monthly check confuses noise and signal.
Keep the same questions from one period to the next, or you are comparing two different things. Also write down the date of every action, such as a new page, a review campaign or a technical fix, so you can link a change in the curve to a cause.
For a first snapshot, the free test asks ChatGPT, Gemini, and Perplexity a few of your customers’ questions. To follow these six metrics over time, iPand’s Visibility module calculates them on your own questions; our comparison of tools covers the other options.
Visibility % across dozens to hundreds of prompts run multiple times is a reasonable metric.
Visibility: the share of answers that cite you. It tells you whether you are part of the conversation. Then add citation share, which tells you where to act.
Because AI answers vary, even for the same question. That is normal: read your metrics weekly or monthly, on the same questions.
Enough to cover your main intents: finding, comparing, buying and asking for advice. A few dozen well-chosen questions beat hundreds picked at random.
Partly. Search Console counts appearances in Google AI Overviews together with the rest of your traffic, without isolating them, and says nothing about ChatGPT or Perplexity. The two measures complement each other.
Test your visibility
Find out for free whether ChatGPT, Gemini, and Perplexity mention your business.
Blocking AI assistants’ crawlers means disappearing from their answers. The check takes two minutes.
Six tools compared on the engines they track, the number of prompts, public prices and countries, with the source and date of every figure.
You can decline: the whole site stays available.
Remembers your theme choice and your cookie choice.
Pages viewed, referrer, device; aggregated statistics.