Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear

-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://a-moz.groupbuyseo.org/learn/seo/robotstxt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console not loading some resources
When I check an URL with Search Console it cannot load some page resources, even from other domains (like: ssl.google-analytics.com, www.facebook.com and www.google-analytics.com).
Reporting & Analytics | | TottiataHUN
Have any of you experienced this issue?
Steps to reproduce: open Google Search Console check an URL click "View crawled page" link select "More info" tab click "Page resources" {?}/{?} couldn't be loaded When I check the listed resources, all of them can be loaded from a web browser.
So I do not understand why Google cannot load them.
And there is no additional info why the resources could not be loaded.
Any ideas? google-search-console-other-error-1.png google-search-console-other-error-2.png0 -
How to track google auto search suggestion click?
Hello Guys, In google.co.uk when I search SEL and google gives me option of different different sites and when I click on any one site then that click tracking I need. I have attached the screenshot to understand easily. Is it possible to track such things or possible via server logs etc? TV99h
Reporting & Analytics | | micey1231 -
Redirecting all URLs appended with index.htm or index.html
It has come to my attention with one of my clients (WordPress website) that for some time they have within their Landing Page report (of GA - Google Analytics) URLs that should all be pointing to the one page, example: domain.com/about-us, also has a listing in GA as domain.com/about-us/index.htm Is this some kind of indication of a subdirectory issue? Has anyone had experience with this in such wordpress plugins as Yoast SEO, or other SEO plugin? My thoughts here are to simply redirect any of these non-existent files with a redirect in .htaccess - but what I'm using isn't working. I will insert the redirect here - - and any help would be greatly appreciated. RewriteEngine onRewriteCond %{THE_REQUEST} ^./index.html?
Reporting & Analytics | | cceebar
RewriteRule ^(.)index.html?$ http://www.dupontservicecenter.com/$1 [R=301,L] and this rewrite doesn't work: RewriteEngine on
RewriteRule ^(.+).htm$ http://dupontservicecenter.com/$1.php [R,NC] _Cindy0 -
Referral Traffic from Google
Hello, I have a question about my company's new website. I've worked in SEO and studied Google Analytics results for a few years now but have never really come across something like this. I started in this position in January of this year and when I started breaking down the traffic sources in Google Analytics, I noticed most of the traffic was coming from Google.com as a referral source. I had never seen Google.com as a referral source before so I looked into options for what it could be. It was not a paid ad and our organic traffic was coming through in Analytics, Before I could get any further, our new website was launched (we switched CRM's to WordPress) and the referral traffic from google went from 2,966 in January of 2015 to 22 in February 2015. for more comparison, in February of 2014, the referral traffic from Google was 2,496. I expected a drop when we switched CRM's but we correctly re-directed all pages and created a new sitemap and our organic traffic is up since the switch (not enough to cover drop in referral). I thought at first this had to do with our Google sellers account being de-activated when we made the switch, but I quickly fixed this over a month ago and no change. I'm wondering if anyone has ever seen Google.com come through as a referral source in Google Analytics and if they we're able to figure out what it actually was. This would be a great help! Thank you, Alex
Reporting & Analytics | | RASEO1 -
Switch to www from non www preference negatively hit # pages indexed
I have a client whose site did not use the www preference but rather the non www form of the url. We were having trouble seeing some high quality inlinks and I wondered if the redirect to the non www site from the links was making it hard for us to track. After some reading, it seemed we should be using the www version for better SEO anyway so I made a change on Monday but had a major hit to the number of pages being indexed by Thursday. Freaking me out mildly. What are people's thoughts? I think I should roll back the www change asap - or am I jumping the gun?
Reporting & Analytics | | BrigitteMN0 -
Setting up Google Analytics for Subsites
I currently have one main .com site and am planning on launching geo-location subsites .co.uk, .com.au, .ru, etc... Traffic will flow between both sites and some of the content on the subsites will be duplicate and therefore include a canonical tag to the main site. I want to set up GA to capture who is going to the subsites and vice versa and correctly capture crossover traffic. Any advice on implementing advanced analytics directly (or links to sources that will direct me the right direction for this project)
Reporting & Analytics | | theLotter0 -
What is s.ytimg.com in google analytics?
My clients GA reports 273 visits from s.ytimg.com. I go to the site, it doesn't exist. I googled it, there were some code with s.ytimg.com in it, but nothing I could understand. Anybody have an idea where this comes from?
Reporting & Analytics | | endlessrange0