Are you measuring AI visibility wrong? Tom Capper reveals the 4 biggest mistakes in AI prompt tracking and 4 creative ways to expand your list of prompts.

Click on the whiteboard image above to open a high-resolution version!

Happy Friday, Moz fans. Today I want to talk about prompt tracking. Now this has been a big discussion topic in the industry, particularly the last few months, I feel actually, but actually the last year in general, and it's something that we have built out in both Moz Pro and STAT, and we're including for all our customers.

But I think a lot of people have their reservations about prompt tracking. So I'm not going to focus on talking about our products today. This isn't an advert. But instead, I want to talk about four mistakes that I see people making in any tool, really, that I think are leading to some of these reservations and misunderstandings.

So the first one, and probably the biggest one, actually is citations.

So I've got this example, a beautiful prompt here. This is the kind of thing that you might see as like a screenshot in your prompt tracking tool. There's a question at the top, which in this case is probably something about a phone recommendation or what phone is best for certain needs or something like this.

Then you get a response below, which mentions various phone brands and sub-brands, and then has a citation. In this case, I've taken the example of pcmag.com.

Now say you're Apple. You're Apple in this example, and you're trying to gauge your own performance. What you could do is look for apple.com being mentioned. You say, "Oh, no, the citation is actually from an external site. It's from PCMag. Therefore, this is a failure.

I think that would be ill-conceived. You don't really care where this information comes from. In fact, if anything, it's more authoritative if this information is coming from someone else's site, right? You are not necessarily looking to be cited.

Obviously, you might be one of the cited sources here. But the most important thing is that you are mentioned. 

So I think this is mentions over citations, and also not necessarily just mentions of your domain or even of your main brand. Obviously, some of your other brands, like iPhone, in this case, for example, or iTunes, or whatever it might be, those are more important than who is being cited.

We know now that these kinds of AI responses, in most cases, are not very likely to generate a click or referral traffic through to a website.

This is a different place in your marketing funnel from traditional search, where you might have been focused on getting that click. This is much more about awareness. This is about being in the conversation. So I don't think you should be overfocused on who is here, except insofar as this is the opportunity.

You might see this, say you're not happy with how you're mentioned in this snippet. You might see this and think, "Okay, fine, I need to do some outreach to PCMag and make sure I'm featured in their roundups or whatever it might be." So this is your opportunity, not your ranking or something like that.

Speaking of which, I think the second big mistake that people make is thinking about this in traditional SEO metrics, like ranking.

So here, I could say, "Oh, fine, I'm mentioned first. Therefore, that's great. I've won." But actually, the Samsung brands have been mentioned more times.

Over a large number of prompts, I might be far more interested in how many times I'm mentioned at all, what percentage of responses mentioned me at all, how I'm mentioned, what are the snippets in which I'm mentioned, and what kind of adjectives are used.

There are all sorts of metrics. I think this is quite a young sort of space right now. So this is emerging in terms of what metrics people prefer. But I really think you shouldn't focus on the traditional SEO ones because they're not that helpful in this space.

Sure, it is a piece of contextual information who is mentioned first. It is something you might wish to look at, but it shouldn't be your be-all and end-all in the same way that it might have been traditionally.

I would be looking over a large number of prompts as my sort of bread and butter, my equivalent to ranking, before I get into the more subtle stuff. I would be looking over a large number of prompts and just looking at what percentage of the responses mention me at all as my main sort of gauge to get started, which brings me on to my third big mistake, and that is quantity.

If you are a brand like Apple, like I've just been talking about, you probably have been tracking. I assume that Apple does have an active SEO program, and they probably are tracking thousands, if not tens of thousands, perhaps hundreds of thousands of keywords.

So it wouldn't make sense for a brand of that size to go and say, "Okay, across all of our products and all of our brands and all of our markets, we're going to track 50 prompts."

If anything, there is more variance in prompts. The outputs vary more from day to day. They are less deterministic. They are longer tail. If anything, you would be tracking more prompts than you were doing top-of-funnel keywords.

Now, don't get me wrong. I think if you're a small business, I think if you're a local business, I think if you're a business that does a very specific thing, maybe 50 prompts can be enough in some cases. But this is the same kind of business that would have been tracking a smaller number of keywords as well.

So I think you need to have a scale that is appropriate, and that's what we're trying to support in our products as well, making sure that customers can have a scale that is similar to their keyword tracking without it being sort of astronomically expensive or something like this. 

I'll talk a little bit in a moment about how you might make use of different types of prompts to make sure that you're getting a representative picture.

But this is the same kind of thing you're used to with keyword tracking, basically, where maybe 10 years ago you might have had a similar conversation about keyword tracking, where you say, "There's so much localization now. There's so much personalization now. We need to have a wider variety of keywords in order to get representative data in our tracking." And this is the same kind of thing. 

The last thing I want to speak about is head terms. So when you're deciding what prompts to track, as I mentioned, they're going to be longer tail typically. I don't want you to take whatever your existing keyword list was and just dump it into whatever your prompt tracking tool of choice might be.

Very few people are opening ChatGPT, typing "laptops," and then hitting Enter. I'm sure it's been done. It's probably been done by me for an example on a slide. But this is a weird thing to do. This isn't really how people interact with these tools. So you should make sure you reflect that in the kind of prompts that you're researching and including.

Now, prompt volume, it can be very difficult right now. There are long-standing stats that people like to use, which I think is something like 15% of searches of keywords that Google sees every day are brand new, as in never seen before.

One has to imagine that for prompts, this is 90-something percent. I don't know the exact stat. 

But if I were to guess, it's probably 90-something percent. I think the overwhelming majority of prompts probably have a volume of roughly one. So, really, you just need to identify the topics that are important and then have some realistic prompts. 

We have some tools to help with this. My colleague, Dr. Pete, has written a great guide, which will be referenced below, I hope. People Also Ask could also be a good starting point because those are sort of real long-tail keywords that Google seems to think exist and might be useful. There are lots of different things you can do here to try and get some kind of representative, more specific, longer-tail terms. Those are the four sorts of mistakes I wanted to talk about.

But as a bonus, I just want to talk about four ideas as well for more creative things you could do with your prompt tracking to try and get different insights out of the same tools. 

1. Consider specifying locations or personas

So one of the big ones that you could play with is locations and personas. 

So, in your traditional rank tracking, something that a lot of people do and something that I've done in the past is take the same keyword and then maybe track it from say 50 different postcodes, or something like that, to see if I get different responses. 

For different kinds of business, this may be more or less useful. Or if you're not a business that deals in real-world locations, then maybe this would be more about personas. So instead of saying, "What is the best phone for me?" you might say, "What is the best phone for a university student who has to do document editing on the go?" or something like this. That's a pretty strange example, but for example. Then you might have various other personas, or you might have various other locations, depending on the type of business. 

2. Play around with different languages

The second thing you can do is play around with languages. 

So at the moment, most prompt-tracking tools are somewhat language-ambivalent because most of the interfaces that they deal with are also language-ambivalent. 

ChatGPT basically responds in whatever language you use. You can open ChatGPT. The interface is in English. If you type to it in Italian, it will reply to you in Italian. So you can take advantage of that. If you do exist in multiple markets where you want to see what the responses are like in different languages, you can take advantage of that and just track prompts in whatever languages you see fit. 

Obviously, that could be combined with the location point. 

Then these last two ideas are slightly different. So this is less about sort of prompt expansion in terms of getting the prompt you want to track to get more detail and to be more representative, and more about getting sort of different types of insights rather than just performance.

3. Specify qualities

So, for example, instead of saying what is the best iPhone for blah, blah, blah, you could say what is the cheapest phone, or the most durable phone, or the most trustworthy phone, or the most privacy-respecting phone. You could have these as sort of different campaigns that you track in order to get different types of insights. 

4. Identify and focus on market gaps

Or the flip side of that, this would take a little bit more processing on your part to go through these insights, but you could ask, "What is it that the Samsung phone does better than the iPhone," or something like this, and then get sort of more qualitative insights out that you could then track over time to see whether you're making a difference in your marketing with that perception. 

All right. I hope you like these ideas. I hope you're not making these mistakes. If you are, I hope this was useful. Thank you very much. See you in the next Whiteboard Friday.