How to block Rogerbot From Crawling UTM URLs

[Firestarter-SEO]

I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like:

User-agent: rogerbot
Disallow: /?utm_source*

This does not seem to be working. Any ideas? 

[tawnycase]

Shoot! There may be something else going on. Give us a shout at [email protected] and we'll see if we can figure it out!

[Jenny1]

FYI - I tried this and it did not work. Rogerbot is still picking up URL's we don't need. It's making my crawl report a mess!

[tawnycase]

The only difference there is the * wildchar. The string with that character will limit the crawler from accessing any URL with that string of characters in it.

[Firestarter-SEO]

What is the difference between Disallow: /*?utm_ and Disallow: /?utm_  ?

[tawnycase]

Hi there! Tawny from the Customer Support team here!

You should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:

User-agent: Rogerbot
Disallow: ?utm

etc., until you have blocked all of the parameters that may be causing these duplicate content errors. It looks like the _source* might be what's giving our tools some trouble. It looks like Logan Ray has made an excellent suggestion - give that formatting a try and see if it helps!

You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer. Here is a great resource about the robots.txt file that might be helpful: https://a-moz.groupbuyseo.org/learn/seo/robotstxt We always recommend checking your robots.txt file with a handy Robots Checker Tool once you make changes to avoid any nasty surprises.

[LoganRay]

Skyler,

You're close, give this a shot:

Disallow: /*?utm_

This will be inclusive of all UTM tags regardless of what comes before the tag or what element you have first.