HiI know there are two crawlers that Moz uses Roger bot and open site Explorer uses dotbot

Make sure there is no forward slash "/" after e.g. (Disallow:/)

Mozused to have an awesome writeup on it but it just forwards to Moz.com/help now it could be that they have another great write up but the URL changed. For now here's the information

User-agent: rogerbot
Disallow: 
User-agent: dotbot
Disallow:

User Agent Analyser
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])

http://www.useragentstring.com/Dotbot1.1_id_16014.php

https://udger.com/resources/ua-list/bot-detail?bot=rogerbot

http://www.botopedia.org/user-agent-list/crawlers/item/369-rogerbot-seomoz

hope this helps,

Thomas

Please forgive my 1st comment I the button too early and use the dictation software so I save it to one page then paste to another I am sincerely sorry I got this part on there without the entire thing.

Send me the domain either privately if you can or through this chat I would be more than happy to look into it for you. I can tell you I have made the no follow no index mistake myself showing a intern something on our own site and talk about it here below.

However if you are still getting problems you may want to download

screaming frog SEO spider

it only will check for 500 links for free however it gives you invaluable insight

It is a download and works on Mac, Windows and Linux

http://www.screamingfrog.co.uk/seo-spider/

if you want to try something web-based

http://www.internetmarketingninjas.com/tools/

http://www.internetmarketingninjas.com/broken-links-tool/

http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/

http://www.internetmarketingninjas.com/seo-tools/google-sitemap-generator/

I would also not hesitate to use their DNS tool to check that everything there is okay

Another tool or tools I would strongly recommend and you can access for free are the excellent Internet marketing ninjas

The words used in the metadata tags, in body text and in anchor text in external and internal links all play important roles in on page search engine optimization (SEO). The On-Page Optimization Analysis Free SEO Tool lets you quickly see the important SEO content on your webpage URL the same way a search engine spider views your data. This free SEO onpage optimization tool is multiple onpage SEO tools in one, helpful for reviewing the following onpage optimization information in the source code on the page:

• Metadata tool: Displays text in title tags and meta elements
• Keyword density tool: Reveals onpage SEO keyword statistics for linked and unlinked content
• Keyword optimization tool: Analyzes on page optimization by showing the number of words used in the content, including anchor text of internal and external links
• Link Accounting tool: Displays the number and types of links used
• Header check tool: Shows HTTP Status Response codes for links
• Source code tool: Provides quick access to on-page HTML source code

if you are talking about just the index and no follow

I can now happily say I have done this identical thing.

I have done the exact same thing. I can tell you I was showing somebody how to use the WordPress SEO plug-in when I got distracted and simply did not change the settings back to follow and index. So approximately 2 to 3 days later I noticed a huge loss in ranking year for the company brand name.

(Luckily this was mine not a clients)

It took approximately two days after I changed the settings back to normal follow and index them submitted my entire website to Google's Webmaster tools even clicking yes when asked the index all large change
before I knew it all the rankings had returned back to normal literally the keywords I was tracking returned within the normal fluctuation I see as they were in many cases sometimes better & sometimes little bit worse what I had feared they never would come back at all.

Sincerely,

Thomas

Believe me when I say I was extremely thankful for this and don't see why you will not get the same results with your site.

I hope this is a simple a mistake of just that one problem like mine that's the only thing I can give you a testimony of. I would say you have nothing to worry about. But remember to tell Google Webmaster tools I also did tell Bing but that's up to you

Hi Nick,

My hart goes out to you and your company for having to deal with this. However I do believe that you should use a 302  redirect along with a no flow  just to be certain. Here's some more information.

http://blog.hubspot.com/blog/tabid/6307/bid/11285/SEO-Are-Nofollow-Links-Still-Valuable.aspx

http://a-moz.groupbuyseo.org/blog/nofollow-is-dying-the-impact-of-microblogging-and-nofollow-on-seo

Google lists the three main intended uses of nofollow as:

1. Linking to untrusted content
2. Paid links
3. Crawl prioritisation (typically linking to yourself with nofollow)

Leaving aside for a second the ability / likelihood of webmasters using nofollow correctly (which means that the search engines need to work even with broken implementations just as they often rank HTML code that doesn't validate), there are two big uses of nofollow that are breaking the model:

1. Complete "silo-isation" of large sites
2. Domain owner not trusting trusted content authors' links

"Silo-isation"

Disregarding the fact that I just made that word up, there is a very real trend of powerful sites nofollowing all (or nearly all) outbound links even though they are the very definition of editorial links. The site owners have presumably seen the ranking power achieved by Wikipedia nofollowing all outbound links and are trying to form their very own black hole.

Lack of trust

My understanding of the original intent of proffering nofollow as a solution to the problem of linking to untrusted places was that it was mainly intended for situations like blog comments, profile links, etc., where users of your site could create links to wherever they pleased.

This is definitely valuable (as anyone who has ever had to moderate blog comments can attest) but what about once you do trust the commenter? Since so many sites have no mechanism whereby that nofollow is ever removed, we end up in a situation where people are creating huge amounts of really valuable content and the links they create are nofollow.

Sincerely, Thomas

You can do that using http://totally.awe.sm/ , bit.ly pro or Bilty free if you don't mind not having control of what appears after the /  you will need different TLD and the same name for your domain. if example.com is your brand domain then example.co is your shared domain.

I would recommend against using your websites domain name to create a lot of shared links that will not be part of your website. Even if they will look like example.com/fbshare or example.com/679a43054

You will need a new domain of your choice of course.

Sorry if I was not as clear in the beginning.

I misread your question.

Sincerely,

Thomas

Hi Dan

"if i have multilingual sub sites (Wordpress Multi User envrionment) then I should enter the language specific social accounts into the appropriate language sub site Yoast OR as per the sentence at the top of Yoast page: "...pick the main account associated with this site and please enter them below", just put the main social accounts in here. Im trying to work out if when they say main here they mean the original main English social account ? "

When using multilingual or geo-targeting a.k.a. Hreflang there is an additional plug-in you will want to use ( I like wpml)

See: http://wplang.org/hreflang-attributes-wordpress/

I like: https://wpml.org/documentation/plugins-compatibility/using-wordpress-seo-with-wpml/?

Yes you also want to add a notice of these alternative pages via OpenGraph:

• The locale these tags are marked up in. Note that OpenGraph _does _use an underscore.
• An array of other locales this page is available in.

More on that for instance on the Facebook developers pages.

&

https://yoast.com/hreflang/

this is the accounts page you are discussing?

http://i.imgur.com/fxsPr5H.gifv

https://neonbrand.com/seo/ultimate-guide-setting-wordpress-seo-yoast-plugin-version-221/

http://kb.yoast.com/article/219-getting-open-graph-for-your-articles

https://yoast.com/social-media-optimization-with-wordpress-seo-by-yoast/

Social – WordPress SEO by Yoast

This is a great new section to version 2.0 of the Yoast SEO plugin. He’s really brought together all the social requirements together in one place. It’s really self explanatory, just make sure you fill out as much of each tab as possible.

Accounts

Get your social accounts setup! Then come back here and plug in the URL’s (or in Twitter’s case, the username… come on Yoast-onians, be consistent!

CbVowPG.png fxsPr5H.gif