Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Increase Website Visibility on Google and Bing?
I am working on an e-commerce niche website and I aim to rank higher on Google to drive more traffic to my website. Any suggestions?
Link Building | | digitalenginehub0 -
Redirecting dofollow, high-domain-authority links from one site to another: good idea?
I have two sites that offer help for freelance writers. The first gets almost no search traffic and is very low priority for me. The second is my main priority and gets substantial search traffic. The low-priority site has a significant number of dofollow backlinks from high-domain-authority sites. Is it a good idea to 301 redirect these links to similar/related posts on my high-priority site? Would this potentially boost the SEO of my high-priority site? Thanks for any help!
Link Building | | John88990 -
What is the best way to remove a link that redirects to a spammy site?
I've got a new client and I'm trying to clean up their backlinks. There are several links that all redirect to this spam site http://www.expert-lender.com. All of the websites appear to be real, i.e. http://www.sammorganhomes.com/ but the actual links i.e http://www.sammorganhomes.com/wp-fav/backup/supplement/semitruckleasing.html are in a sub-directory and redirect to the spam site. I don't know if these links were from the previous SEO company or if they paid someone to create these or if these sites have been hacked. Can anyone tell me what is going on here and what should I tell these site owners?
Link Building | | Total-Design-Shop0 -
How to index Backlinks Fast in Google!
I Made few backlinks with my keyword but they are not getting index and it has been for 15 days. Do you have any idea that will help me out to solve this problem and also to index back links quickly.
Link Building | | rohitdeveloper0 -
Google Disavow File Update
Is there any specific format to update the Disavow file? Also if I submitted the file a months ago, and need to update it now... should I leave the old 'excluded domains' or should I remove them? Lets say this is what I have: How would you update it? #explanation from to Google went here... and ended here.
Link Building | | dhidalgo1
"domain:exampledomainalreadysubmitted1.com"
"domain:exampledomainalreadysubmitted2.com"
"domain:exampledomainalreadysubmitted3.com" Thanks for your input0 -
A link with "return false"- OSE sees as a No Followed Link
Hello, I couldn't find a clear answer to the impact on SEO for a link written in this way: [" class="expert_info" onclick="window.open(this.href);return false;">](w</span>ww.yourwebsite.com<span style=) [Does the "return false" act as a "no follow"? I came across this in our link data in Open Site Explorer which lists these links all as "no follows." However, an engineer I spoke to said that it shouldn't impact search engine behavior. Any ideas? Thank you in advance! -Sarah K.](w</span>ww.yourwebsite.com<span style=)
Link Building | | OneMedical0 -
Site stuck on 2nd page on Google - how can it improve?
Hi I am working on a site http://www.harrisbassett.co.uk that for search terms such as 'Chartered Accountants Swansea' appears near the top of the 2nd page of Google. When looking at what is listed on the 1st page for 'Chartered Accountants Swansea' I would appreciate any advice/guidance on what else could be done to give the site a final push to get on the 1st page of search results. Over the past 2 weeks I have been building inbound links from local citation sites yelp, qype etc along with sites from competitor analysis via opensiteexplorer. Looking at the sites that are on the 1st page I cant understand why many of them are ranking higher than me? Any insight and plan of attack would be most appreciated from any search experts on the forum. Many Thanks
Link Building | | idv0 -
Are backlinks to a domain that redirects useless?
I am working on trying to get backlinks. We are currently redirecting a few Domains to our current domain name. One of the Domains that is redirecting is the main Domain, we will be advertising in the near future. Can I send backlinks to the redirected URL or does it only carry link juice if it is to the main Domain?
Link Building | | hfranz0