Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?

[Dan-Lawrence]

Hi

Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ?

Is there an option i havnt seen to add logins/passwords for crawl software to access ?

cheers

dan

[Dan-Lawrence]

ok thanks Chiaryn

is that the actual name of the moz crawler (to allow in Robots) simply rogerbot ? or any other characters etc ?

Also is it not the case that even when blocked by robots.txt G can still crawl/index it once password removed, think i read few comments somewhere on Moz that can still happen somehow ?

Please advise asap ?

Many Thanks

Dan

[ChiarynMiranda]

Hey Dan,

Unfortunately, our crawler is not able to access password protected content on your site. If you create a staging subdomain that is not password protected, you could use the robots.txt file to allow rogerbot and block other crawlers, but I'm afraid our crawler will not crawl anything that a normal search engine crawl would not be able to crawl so we cannot crawl password protected pages.

I hope this helps.

Chiaryn

[Dan-Lawrence]

i dont suppose either of you are able to help at all with this related question:

http://a-moz.groupbuyseo.org/community/q/site-crawl-errors-download-list-of-all-urls

[Dan-Lawrence]

i dont suppose either of you are able to help at all with this related question:

http://a-moz.groupbuyseo.org/community/q/site-crawl-errors-download-list-of-all-urls

[Dan-Lawrence]

Hi Andy

Screaming Frog does have password access feature for your info i have just tried it

All Best

Dan

[Dan-Lawrence]

Thanks Matt

I have got screaming frog and can confirm that it has password access feature, but i really want Moz to be able to access too, i would have thought they should have this option somewhere. Are you saying Moz crawls have more info than SF (re 'moz level' analysis) ?

Dev site better password prtected than robots arnt they i think ?

Cheers

Dan

[Andy-Halliday]

Hi Dan

I was about to ask the exact same question, so will keep an eye out for an answer.

I hope it is possible, but I couldn't work it out.

[MattAntonino]

I don't know if there's a way to do this in Moz but you could always get Screaming Frog & tell it to ignore robots.txt - that will definitely crawl it. You can check titles, descriptions, canonicals, H1s, etc. that way.  It doesn't give the Moz level analysis but it's a start that def works. You can also see if you have parameter issues that way.