Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Redesign - What to do with old 301 URLs?
My current site is on wordpress. We are currently designing a new wordpress site, with the same URLs. Our current approach is to go into the server, delete the current website files and ad the new website files. My current site has old urls which are 301 redirected to current urls. Here is my question. In the current redesign process, do i need to create pages for old the 301 redirected urls so that we do not lose them in the launch of the new site? or is the 301 command currently existing outside of our server so this does not matter? Thank you in advance.
Web Design | | CamiloSC0 -
How can a Pincode finder website be SEO optimised?
Guys, I wanted to build a simple Pincode finder website for India. The targeted visitors as is obvious will be from India. Alike other Pincode finder websites, the users in this case too will have to key in the location / area of whose pincode he is looking for and they will get Pincode from that very location / area. Other than this, users will also come to this website when they search for something like " <location name="">pincode</location>" on Google (for instance, users will search for something like "Hiranandani Gardens Powai Pincode") Along with data fethced from our sources via Indian postal departments and other data available in public domain, we shall be using data from Google Maps API too. My question in regards to the same is as follows: What should the page-structure / structure of the website be for ranking well on Google? What should be the URL structure? Other suggestions to rank well on Google in this regards? Competition: (You can search for the term "Hiranandani Gardens Powai Pincode" to know how these sites show data) http://www.getpincode.info http://www.pincode.net.in Pls. help...
Web Design | | ShalinTJ0 -
Learning Center on Subdomain or New Domain
Hello, With so much emphasis for SEO to develop unique, information content I am working to build out a "Learning Center' for one of my clients in the finance industry. As you can guess, this is quite a competitive space and their rankings have become somewhat stagnant so we are looking for new ways to develop original, share-worthy content for the site. So my question is, in order for them to retain the most SEO value should we develop this on a subdomain such as learning.website.com or a new domain altogether. Note: We do not want to develop this under the current site domain as it we want o only post informational content & courses and keep these pieces outside of the "sales" side of their website. Therefore, I thought the subdomain would be the best bet so that we would retain the most value for SEO... however, some of my colleagues disagree. Some vote it should be within a directory under website.com. Some say a new site & domain altogether. My vote is to place this under a subdomain such as http://learning.website.com with the idea that the site's authority will influence the root http://website.com. The argument I was provided against the subdomain was: "essentially adding a subdomain would mean you're sort of starting over again in terms of building authority. adding a folder will ride the coattails off the already established authority." Lastly, for clarification, the current domain is set up as http://website.com so the subdomain would be http://learning.website.com, the directory would be set up as http://website.com/learning/ and a whole new domain would be http://newwebsite.com. This is for a Wordpress site. Thoughts? All feedback is much appreciated!
Web Design | | TinaMumm0 -
Is it possible to redirect the main www. domain - but keep a subdomain active?
Hi Mozzers, Quick question, which I hope one of you can answer... Let's say I have a website (i) www.example.com and on that a subdomain exists, (ii) subdomain.example.com. Let's say I want to change my main domain from www.example.com to www.newwebsite.com. I'd 301 all content, use GWT to notify Google of a change of address etc etc. Having done that, is it still possible to keep the original subdomain active? So, even though www.example.com has been redirected / transferred to www.newwebsite.com, subdomain.example.com would still exist. If that is possible, what is the implication for Domain Authority? On the one hand, I have transferred the main site (so DA from that will transfer to the new site); but part of that root domain is still active. Make sense? Any answers? Thanks everyone...
Web Design | | edlondon0 -
Website Redesign - Will it hurt SERP?
Hi - I am planning to redesign my blog and I was wondering if this will affect my rankings? The new website template (custom designed) is much more user and seo friendly. The content, url structure, internal linking structure, meta tags, and site structure will remain exactly the same, but the visual design will be different (new sidebar widgets, and slightly different layout on inner pages). The current website is ranking very well (mostly top 5), has a healthy backlink profile, strong social media presence, and great traffic. I have heard that switching to a new template will dramatically hurt the rankings. Is this true? Are there any exceptions? Any ways I can prevent the rankings from dropping? Would really appreciate your input. Thanks in advance. Howard
Web Design | | howardd0 -
Need to rebuild client's flash website
I am working with their web designer and need to figure out a way to rebuild their site which is currently all in flash. I was wondering if there was a way to do this without spending a ton of time in completely re-doing the site from scratch.
Web Design | | awalker840 -
URLs with Hashtags - Does Google Index Them?
Hi there, I have a potential issue with a site whereby all pages are dynamically populated using Javascript. Thus, an example of an URL on their site would be www.example.com/#!/category/product. I have read lots of conflicting information on the web - some says Google will ignore everything after the hashtag; other people say that Google will now index everything after the hashtag. Does anybody have any conclusive information about this? Any links to Google or Matt Cutts as confirmation would be brilliant. P.S. I am aware about the potential issue of duplicate content, but I can assure you that has been dealt with. I am only concerned about whether Google will index full URLs that contain hashtags. Thanks all! Mark
Web Design | | markadoi840 -
Best Website Builder - Help Me Choose
I need to built a multi language site (to built a Pilates, Yoga site) and I will use a site builder. After posting questions on wix.com I came to the fact I should continue my research because there are not SEO friendly. Do you have a suggestions? Limited to html knowledge, using a website builder is my only option. Here are some of the features I need: Multilanguage Web Site Mobile version SEO Friendly Nice Template Selections( this is important) HTML customization Twitter, Facebook, Blog... I'm not looking at free website builder, when you want good features, there is a price to paid. Thank you for your help and suggestions, BigBlaze
Web Design | | BigBlaze2050