Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://a-moz.groupbuyseo.org/learn/seo/robotstxt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console not loading some resources
When I check an URL with Search Console it cannot load some page resources, even from other domains (like: ssl.google-analytics.com, www.facebook.com and www.google-analytics.com).
Reporting & Analytics | | TottiataHUN
Have any of you experienced this issue?
Steps to reproduce: open Google Search Console check an URL click "View crawled page" link select "More info" tab click "Page resources" {?}/{?} couldn't be loaded When I check the listed resources, all of them can be loaded from a web browser.
So I do not understand why Google cannot load them.
And there is no additional info why the resources could not be loaded.
Any ideas? google-search-console-other-error-1.png google-search-console-other-error-2.png0 -
Conflicting average position data from Google Search Console?
I'm looking at Google Search Console data in Google Analytics, specifically Average Position as given in the Landing Page report, and the same metric broken out by mobile and desktop in the Devices report. In the Landing Page report, I see an aggregated average position that's much higher/worse than an actual average of what is reported for mobile, desktop and tablet traffic under the Device reporting. For example: Mobile: 5 Desktop: 5 Tablet: 5 So the average still should be roughly 5, right? Why would the Landing Page then show an aggregate Average Position of 8? I wouldn't expect to see a precisely same average given that different device types have different proportions that could render differently when the buckets are combined, but this is a huge swing. In fact, the aggregate Average Position as given in the top level Devices report is closer to 5 than to the 8 shown in the Landing Pages report. (These aren't actual numbers, but are illustrative of what I'm seeing, by the way.) Unless I'm missing some vital difference in the way that Average Position is reporting for the Landing Page report versus the Device reports, it doesn't seem like this should be possible. What am I missing?
Reporting & Analytics | | BradsDeals0 -
Google Analytics - Average Position
Hi Just trying to get some clarity on Google Analytics Average Positions in "Aquistions/Search Engine Optimisation". For a very competitive keyword Google Analytics is saying i am on average position of 6. Is this Page 6? I am assuming position six would be 1.6?
Reporting & Analytics | | Cocoonfxmedia0 -
% Change - Google analytics - how to calculate?
Hi All,
Reporting & Analytics | | JohnPalmer
I have two dates with two different numbers I want to calculate the "% Change" like google analytics, The numbers of June 2015 - 127,931 sessions
The numbers of June 2014 - 90914 sessions please tell me what is the %Change. Best. J0 -
Google Analytics - Organic Search Traffic & Queries -What caused the huge difference?
Our website traffic dropped a little bit during the last month, but it's getting better now, almost the same with previous period. But our conversion rate dropped by 50% for the last three weeks. What could cause this huge drop in conversion rate? In Google Analytics, I compared the Organic Search Traffic with previous period, the result is similar. But the Search Engine Optimization ->Queries shows that the clicks for last month is almost zero. What could be the cause of this huge differnce? e9sJNwD.png k4M8Fa5.png
Reporting & Analytics | | joony0 -
How can I see what Google sees when it crawls my page?
In other words, how can see the text and what not it sees from start to finish on each page. I know there was a site, but I can't remember it.
Reporting & Analytics | | tiffany11030 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
Is it possible to use Google Tag Manager to pass a user’s text input into a form field to Google analytics?
Hey Everyone, I finally figured out how to use auto event tracking with Google Tag Manager, but didn't get the data I wanted. I want to see what users are typing into the search field on my site (the URL structure of my site isn't set up properly to use GA's built-in site search tracking). So, I set up the form submit event tracking in Google Tag Manager and used the following as my event tracking parameters: Category: Search Action: Search Value When I test and look in Google Analytics I just see: "search" and "search value." I wanted to see the text that I searched on my site. Not just the Action and Category of the event.... Is what I'm trying to do even possible? Do I need to set up a different event tracking parameter? Thanks everyone!
Reporting & Analytics | | DaveGuyMan0