Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Cleaning up a Spammy Domain VS Starting Fresh with a New Domain
Hi- Can you give me your opinion please... if you look at murrayroofing.com and see the high SPAM score- and the fact that our domain has been put on some spammy sites over the years- Is it better and faster to place higher in google SERP if we create a fresh new domain? My theory is we will spin our wheels trying to get unlisted from alot of those spammy linking sites. And that it would be faster to see results using a fresh new domain rather than trying to clean up the current spammy doamin. Thanks in advance - You guys have been awesome!!
Intermediate & Advanced SEO | | murraycustomhomescom0 -
Why is my domain authority still 1?
I changed the domain of my website from www.vanillacrush.co.uk to www.carissamay.co.uk at the end of December and yet my DA for carissamay is still 1. As advised, I set up a 301 redirect from VC to CM which seems to be working fine. However when I check on redirect detective it tells me I also have a 302 set up. Could this be confusing things? http://www.vanillacrush.co.uk
Intermediate & Advanced SEO | | Carissamayhttp://www.vanillacrush.co.uk/
http://www.carissamay.co.uk
Any help would be greatly appreciated! Many thanks
0 -
How Can I Redirect an Old Domain to Our New Domain in .htaccess?
There is an old version of http://chesapeakeregional.com still floating around the web here: http://www.dev3.com.php53-24.dfw1-2.websitetestlink.com/component/content/category/20-our-services. Various iterations of this domain pop up when I do certain site:searches and for some queries as well (such as "Diagnostic Center of Chesapeake"). About 3 months ago the websitetestlink site had files and a fully functional navigation but now it mostly returns 404 or 500 errors. I'd like to redirect the site to our newer site, but don't believe I can do that in chesapeakeregional.com's .htaccess file. Is that so and would I need access to the websitetestlink .htaccess to forward the domain? Note* I (nor anyone else in our organization) has the login for the old site. The new site went live about 9 months before I arrived at the organization and I've been slowly putting the pieces together since arriving.
Intermediate & Advanced SEO | | smpomoryCRH0 -
Prevent Google from crawling Ajax
With Google figuring out how to make Ajax and JS more searchable/indexable, I am curious on thoughts or techniques to prevent this. Here's my Situation, we have a page that we do not ever want to be indexed/crawled or other. Currently we have the nofollow/noindex command, but due to technical changes for our site the method in which this information is being implemented if it is ever displayed it will not have the ability to block the content from search. It is also the decision of the business to not list the file in robots.txt due to the sensitivity of the content. Basically, this content doesn't exist unless something super important happens, and even if something super important happens, we do not want Google to know of its existence. Since the Dev team is planning on using Ajax/JS to pull in this content if the business turns it on, the concern is that it will be on the homepage and Google could index it. So the questions that I was asked; if Google can/does index, how long would that piece of content potentially appear in the SERPs? Can we block Google from caring about and indexing this section of content on the homepage? Sorry for the vagueness of this question, it's very sensitive in nature and I am trying to avoid too many specifics. I am able to discuss this in a more private way if necessary. Thanks!
Intermediate & Advanced SEO | | Shawn_Huber0 -
How can I prevent duplicate pages being indexed because of load balancer (hosting)?
The site that I am optimising has a problem with duplicate pages being indexed as a result of the load balancer (which is required and set up by the hosting company). The load balancer passes the site through to 2 different URLs: www.domain.com www2.domain.com Some how, Google have indexed 2 of the same URLs (which I was obviously hoping they wouldn't) - the first on www and the second on www2. The hosting is a mirror image of each other (www and www2), meaning I can't upload a robots.txt to the root of www2.domain.com disallowing all. Also, I can't add a canonical script into the website header of www2.domain.com pointing the individual URLs through to www.domain.com etc. Any suggestions as to how I can resolve this issue would be greatly appreciated!
Intermediate & Advanced SEO | | iam-sold0 -
Changing domain for a magento store
Hi all, wondering if i could gather some views on the best approach for this please... We currently have a magento site up with about 150,000 pages (although only 9k indexed in Google as product pages are set to no index by default until the default manufacturer description has been rewritten). The indexed pages are mainly category pages, filtering options and a few search results. While none of the internal pages have massive DA - seem to average about 18-24 which isn't too bad for internal pages, I guess - I would like to transfer as much of this over to the new domain. My question is, is it really feasible to have an htaccess with about 10,000 301 redirects on the current domain? The server is pretty powerful so could probably serve the file without issue but would Google be happy with that? Would it be better to use the change url option in WMT instead. Ive never used that so not sure how that would work in this cause. Would it redirect users too? As a footnote, the site is changing because of branding reasons and not because of a penalty of the site. Thanks, Carl
Intermediate & Advanced SEO | | daedriccarl0 -
Microsites: Subdomain vs own domains
I am working on a travel site about a specific region, which includes information about lots of different topics, such as weddings, surfing etc. I was wondering whether its a good idea to register domains for each topic since it would enable me to build backlinks. I would basically keep the design more or less the same and implement a nofollow navigation bar to each microsite. e.g.
Intermediate & Advanced SEO | | kinimod
weddingsbarcelona.com
surfingbarcelona.com or should I rather go with one domain and subfolders: barcelona.com/weddings
barcelona.com/surfing I guess the second option is how I would usually do it but I just wanted to see what are the pros/cons of both options. Many thanks!0 -
XML Sitemap index within a XML sitemaps index
We have a similar problem to http://www.seomoz.org/q/can-a-xml-sitemap-index-point-to-other-sitemaps-indexes Can a XML sitemap index point to other sitemaps indexes? According to the "Unique Doll Clothing" example on this link, it seems possible http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic Can someone share an XML Sitemap index within a XML sitemaps index example? We are looking for the format to implement the same on our website.
Intermediate & Advanced SEO | | Lakshdeep0