Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://a-moz.groupbuyseo.org/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Conflicting average position data from Google Search Console?
I'm looking at Google Search Console data in Google Analytics, specifically Average Position as given in the Landing Page report, and the same metric broken out by mobile and desktop in the Devices report. In the Landing Page report, I see an aggregated average position that's much higher/worse than an actual average of what is reported for mobile, desktop and tablet traffic under the Device reporting. For example: Mobile: 5 Desktop: 5 Tablet: 5 So the average still should be roughly 5, right? Why would the Landing Page then show an aggregate Average Position of 8? I wouldn't expect to see a precisely same average given that different device types have different proportions that could render differently when the buckets are combined, but this is a huge swing. In fact, the aggregate Average Position as given in the top level Devices report is closer to 5 than to the 8 shown in the Landing Pages report. (These aren't actual numbers, but are illustrative of what I'm seeing, by the way.) Unless I'm missing some vital difference in the way that Average Position is reporting for the Landing Page report versus the Device reports, it doesn't seem like this should be possible. What am I missing?
Reporting & Analytics | | BradsDeals0 -
In Google Search Console: Total Clicks differ from the sum clicks on search queries
Hi.This may be a google technical question, but I've searched a lot and I couldn't find any certain information about that.The problem appears when you look at two stats in Search Console ( WebMasters Tools, some months ago), particulary in the Search Analysis.1- Total clicks2- Sum of clicks at every search query.I attached an image to make me clear.8IZsxs6.png
Reporting & Analytics | | NachoRetta2 -
Google Search Console (new GWT) - Does a language specific sub folder need its own GSC profile
HI I've got a clients site set which targets 3 language/countries: English via the main site on the domain.com Turkish via a Turkish language site on a subfolder domain.com/tr/ And German via domain.de The devs have set up .com and .de in GSC and is reporting data in both However there's no data in the domain/com/tr GSC profile ! Is that because its on a subfolder so data pertaining to it is being reported in the main domain.com GSC account ? Or does something more need to be done to set up the Turkish subfolder in GSC ? If so what ? All Best Dan
Reporting & Analytics | | Dan-Lawrence0 -
No-indexed pages are still showing up as landing pages in Google Analytics
Hello, My website is a local job board. I de-indexed all of the job listing pages on my site (anything that starts with http://www.localwisejobs.com/job/). When I search site:localwisejobs.com/job/, nothing shows up. So I think that means the pages are not being indexed. When I look in Google Analytics at Acquisition > Search Engine Optimization > Landing Pages, none of the job listing pages show up. But when I look at Acquisition > Channels > Organic and then click Landing Page as the primary dimension, the /job pages show up in there. Why am I seeing this discrepency in Organic Landing pages? And why would the /job pages be showing up as landing pages even though they aren't indexed?
Reporting & Analytics | | mztobias0 -
Is it possible to use Google Tag Manager to pass a user’s text input into a form field to Google analytics?
Hey Everyone, I finally figured out how to use auto event tracking with Google Tag Manager, but didn't get the data I wanted. I want to see what users are typing into the search field on my site (the URL structure of my site isn't set up properly to use GA's built-in site search tracking). So, I set up the form submit event tracking in Google Tag Manager and used the following as my event tracking parameters: Category: Search Action: Search Value When I test and look in Google Analytics I just see: "search" and "search value." I wanted to see the text that I searched on my site. Not just the Action and Category of the event.... Is what I'm trying to do even possible? Do I need to set up a different event tracking parameter? Thanks everyone!
Reporting & Analytics | | DaveGuyMan0 -
Big variation in the number of search results. (person's name)
Hi, I have been noticing a really dramatic variation in the number of results Google is returning for the name "Carolyn Hadlock." Most of the time it seems to be around 2000. But then it will jump up to over 10,000. Does anyone know why there would be such a big jump? And then why it would go back? If tested both logged into Google and then not - as well as having others log is as themselves. That does not seem to be it. Any thoughts would be much appreciated.
Reporting & Analytics | | yandl0 -
Bing Won't Index Site - Help!
For the past few weeks I’ve been trying to figure out why my client's site is not indexed on bing and yahoo search engines. My Google analytics is telling me I’m getting traffic (very little traffic) from Bing almost daily but Bing webmaster tools is telling me I’ve received no traffic and no pages have been indexed into Bing since the beginning of December. At once point I was showing ranking in Bing for only one keyword then all of a sudden none of my pages were being indexed and I now rank for nothing for that website. From Google I’m getting over 1200 visits per month. I have been doing everything I can to possibly find the culprit behind this issue. I feel like the issue could be a redirect problem. In webmaster tools on Bing I’ve used “Fetch as Bingbot” and every time I use it I get a Status of “Redirection limit reached.”. I also checked the CRAWL Information and it’s saying all the URL’s to the site are under 301 redirect. A month or so ago the site was completely revamped and the canonical URL was changed from non www to www. I have tried manually adding pages to be indexed multiple times and Bing will not index any of the sites pages. I have submitted the sitemap to Bing and I am now at a loss. I don’t know what’s going on and why I can’t get the site listed on Bing. Any suggestions would be greatly appreciated. Thanks,
Reporting & Analytics | | VITALBGS
Stephen0 -
Google Analytics: how many visits from country Google domains?
Hello, I manage a site with visitors from many different countries. With Google Analytics, it is normal to see the number of visitors from each search engine. However, I would like to identify the number of visitors from each Google-search contry domain. How many visitors from Google.com? How many from Google.co.uk. And from Google.co.zm? And so on. Anybody knows if this is possible and if yes, how can it be done? Thank you in advance, Dario
Reporting & Analytics | | Darioz0