Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to know about the Impact of Do-follow backlinks and No-follow backlinks for increasing DA.
Have Do-follow backlinks and No-Follow backlinks same value to increase Domain Authority in Moz? Recently, I have created 290 profile backlinks which was do-follow. But from yesterday I have seen all backlinks are no-follow now according to the decision of sites' owners. My site's DA has already increased up to 35. Will this DA fall gradually from now? Apart from it, if I have created same backlinks for my another site; will my DA be 35? Please try to clarify it. My site is: Homeworkpaper.net Thanks. Waiting for your valuable answer.
Link Building | | darrellpc1 -
How much time does it takes for a backlink to get registered in Google? My basketball anime website has been live since start of this week and Google has not registered any backlink.
So my website https://www.animedhfa.com/basketball-anime/ about basketball Anime has shown no changes in DA or backlinks. It has been 5 days perhaps since it went live.
Link Building | | ahmad74750 -
Sponsored posts against Google guidelines?
I'm a bit confused. Every blog I try to outreach on always give me a quote for a sponsored post. Isn't this against Google guidelines because you paying for a link technically even though your paying for a post? Do you guys buy sponsored posts? Should this be avoided? How do you outreach to a blog that offers sponsored posts? They can smell that you want a link from a mile away and give you a nice fat quote.
Link Building | | The_Kiwi_Man1 -
Links from PRWeb press release violate Google's quality guidelines?
My site has had a manual action performed on it by Google indicating that I have inbound links that fall outside of their quality guidelines. I did my own research, found what I thought was the issue, had the links removed and requested reconsideration. Google's response surprised me in that they highlighted two specific pages with links that were the direct result of valid press releases and a publisher picking up our release off a wire service. Has anyone else seen this occur? Anyone had a case successfully reconsidered? I realize that I don't need to do anything at all as the manual action is in effect and will stay that way, discounting those links, but I would rather a) not have any manual action against my site and b) know for the future so this doesn't happen again. Also, is this applicable for guest blog posts, which effectively create the same type of backlinks? Thanks
Link Building | | barberm1 -
Does Google index a content in iframe?
Hello guys, I´ve done a infografic and i want to share it with some bloggers. But, i want to share an iframe, not an image (beacuse there is some links on it). My question: Can Google consider the links inside an iframe (for link building)? Thank you 🙂
Link Building | | seomobly0 -
A link with "return false"- OSE sees as a No Followed Link
Hello, I couldn't find a clear answer to the impact on SEO for a link written in this way: [" class="expert_info" onclick="window.open(this.href);return false;">](w</span>ww.yourwebsite.com<span style=) [Does the "return false" act as a "no follow"? I came across this in our link data in Open Site Explorer which lists these links all as "no follows." However, an engineer I spoke to said that it shouldn't impact search engine behavior. Any ideas? Thank you in advance! -Sarah K.](w</span>ww.yourwebsite.com<span style=)
Link Building | | OneMedical0 -
How does Google treat multiple backlinks on the same page?
Hello mozCrew Just a quick yay or nay. We have an embeddable code on our site for customers who want to post our link on their website listed here: http://www.accupos.com/etc/embed-code-restaurants.php But in our link, there are actually 2 separate links, one branding link that says the name of our company, AccuPOS, and another link which targets a specific target keyword "Restaurant POS Software". My question is, does Google only give credit for the branding link, since it is listed FIRST? Or will we get double credit? Is this a good or bad practice, and what would you recommend? Thank you! Derek Moore
Link Building | | DerekM880 -
Free Classified Backlinks & Google
Is there a risk that Google will de-index your site or lower your ranking if you have too many backlinks from free classifed sites? I would just use 1 free ad per classified website. Thanks.
Link Building | | finalfrontier0