Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into [email protected] as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Why Moz DA of my website not increasing?
Hi, I have two queries which are related to Moz DA and Moz Spam Score. 1. I have created my website 6 months ago and also do proper SEO of my site. Now according to Ahrefs my DR is 47 and also shows me some organic traffic, Organic Keyword of my site. But when I've checked my site in Moz.com it shows me the DA is only 1 and also there is no organic keywords and no backlinks. Why my DA is not increasing and stuck in 1 position? When will DA increase and please guide me how I can Improve DA of my website? URL: https://www.fasterapk.com 2. My second query is, what is the Moz Spam Score? Is it important for a website and which factors Moz used to create a Spam Score of any website? Experts guide needed. Thanks, overview?site=https%3A%2F%2Ffasterapk.com⌖=domain
Getting Started | | dekho870 -
How I can increase DA of my site?
Hi, I have my 4 month old blog, PA of site is 17 but DA is still 5. I don't know how to increase DA of site. Please suggest me how to increase DA of Site https://myeasygrader.com/ . Thanks
Getting Started | | markwillson0 -
Has anyone purchased the MOZ SEO courses? Are they good?
I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?
Getting Started | | Stevepair0 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up.
Getting Started | | bluerinsevintage
Adam
0 -
MOZ for initial keyword research?
Can I use MOZ for initial keyword research? Can't create an AdWords campaign (no website yet to register) to get use of Google's Keyword Planner & my subscription to Wordtracker is showing me too many unrelated keywords... Peter
Getting Started | | nodiffrei0 -
How long does it usually take Moz to populate information for a new Web site?
We recently launched (9/13/2013) an e-commerce Website and added the campaign to SEO MOZ. Week after week the Domain Rank is 1 and none of our keyword stats or link stats are populated. We have another Moz campaign that posts weekly updates and is doing extremely well. I'm just wondering how long it usually takes Moz to start populating all the analysis stats? I'm also wondering if there might be a campaign setting buried somewhere that I need to enable or maybe it just takes more than 5 weeks? Any insights would be much appreciated. Here's the new URL we need to track with MOZ: http://www.imsportshq.com
Getting Started | | Tripper0
