Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does an external link to an image file on my site help my SEO?
I found a few websites that are displaying images from my site. Curiously, they have also linked these images to the source image file on my website. I found these because Moz showed these sites in my list of backlinks. I assume that means it impacts my Domain Authority. But does this actually help me in search engines since it's not linking to a page on my site?
Link Building | | AlexLenhoff0 -
Nofollow links showing up in Google Search Console
If some nofollow links are showing up in Google Search Console, does that mean they are indeed passing link juice?
Link Building | | BrightonRecoveryCenter0 -
Will Google penalize for backlinks on non-indexed pages?
Have a client who has a bunch of links from spammy sites that aren't even indexed by Google. Will Google penalize (i.e. Penguin) for backlinks from spammy sites that aren't indexed? I'm guessing "Yes, Google still crawls non-indexed sites and may penalize the sites linked to from the banned site." Anyone have any data or experience with this? Anyone have a non-indexed or banned site? Does Google still crawl it?
Link Building | | AdamThompson0 -
How does Google treat multiple backlinks on the same page?
Hello mozCrew Just a quick yay or nay. We have an embeddable code on our site for customers who want to post our link on their website listed here: http://www.accupos.com/etc/embed-code-restaurants.php But in our link, there are actually 2 separate links, one branding link that says the name of our company, AccuPOS, and another link which targets a specific target keyword "Restaurant POS Software". My question is, does Google only give credit for the branding link, since it is listed FIRST? Or will we get double credit? Is this a good or bad practice, and what would you recommend? Thank you! Derek Moore
Link Building | | DerekM880 -
Free Classified Backlinks & Google
Is there a risk that Google will de-index your site or lower your ranking if you have too many backlinks from free classifed sites? I would just use 1 free ad per classified website. Thanks.
Link Building | | finalfrontier0 -
Value of Link Redirect from Google News
Some sites are linking to us with the URL from Google News instead of using a direct link. For example: "http://news.google.com/news/url?sa=t&fd=R&usg=ghjgjhggjhgkjghjg&url=http://www.oursite.com/awesome-news-010123/" Does this pass the same value as a normal link? I asked the publisher to replace the link with a normal link but they pushed back saying that the news URL is better.
Link Building | | ProjectLabs0 -
Should I 301 Redirect The index.html Page?
Hello, As an experiment I am trying to work with an SBI site but am noticing index.html is being forced on me as the homepage. As I understand it's good practice to 301 redirect the index.html page to the to the actual home page. Can someone please confirm that I am right in thinking this? Here is an example: http://www.anguilla-beaches.com/index.html
Link Building | | umtmedia
http://www.anguilla-beaches.com/ According to yahoo 773 links are pointing to the index.html page and 771 links are pointing to the / page. I'm sure Google knows that these are the same pages and does not consider them as duplicate but surely the index.html page should be redirected right? am I going mad here? Now if we look at this site: http://www.jugglenow.com the homepage has 198 links according to yahoo
http://www.jugglenow.com/index.html this page also has 198 links? now I am confused, are these 2 the same page? On this page: http://www.jugglenow.com/juggling-balls.html notice how there is no link to the actual homepage but there is a link to the index.html page. Am I missing something here because I can't figure it out, surely there should be just 1 homepage without a index.html page. Any help would be appreciated0 -
What's the difference between follow and nofollow links?
I understand this may be a really dumb question and from my understanding there is a piece of code in some url's that tell search engines not to follow that link. I am interested in finding out what the purpose of nofollow links are and how they apply to search rankings. Thanks for the help
Link Building | | A2890