Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler is not able to crawl my website
-
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe -
@jasontorney Hi, I had the same problem after moving several of my websites to a Virtual Private Server that had enhanced security features.
One of these features was specifically to stop the Moz bot from crawling websites, and the hosting engineers advised they had done this because it was particularly aggressive in nature.
With my VPS control panel I have found the switch that allows me to disable bot blocking, and I occasionally do this if I'm grading a page with Moz, but advice from hosting support was to otherwise leave it active to protect the websites from attached (which means I don't get feedback from Moz crawls).
If you check with your hosting company you may find that they have a similar bot blocker configured for security purposes.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website ranking declined after connecting to CDN
Hi! We are trying to rank https://windowmart.ca for various local search terms. Our head office is in Edmonton where we try to rank https://windowmart.ca/edmonton-windows-doors/ for such terms as "windows Edmonton", "replacement windows Edmonton", "windows and doors Edmonton" as well as others. The website was the leader in its niche for around 2 years. Then we've got some server related issues, moved to a new server and connected CDN Nitropack that really improved our google speed test results. Recently we noticed that our rankings started to drop. Do you know if Nitropack can negatively effect local SEO rankings? Thank you!
Technical SEO | | vaskrupp0 -
Any SEO benefits of adding a Glossary to our website?
Hi all, I manage a website for a software company. Many terms can be quite tricky so it would be nice to add a Glossary page. Other than that, I have 2 questions: 1. What would be the SEO benefits? 2. How would you suggest to implement this glossary so we can get as much SEO benefit as possible (for example how would we link, where would we place the glossary in the terms of the sitemap, etc.). Any advice appreciated! Katarina
Technical SEO | | Katarina-Borovska2 -
Why did my website DA fell down?
Hello, Could you please let me know why might my website's DA have fallen down in merely a week? What might be a reason? I also noticed traffic from google dropped down at the very same week. Will be very thankful for any advise!
Technical SEO | | kirupa0 -
I have multiple URLs that redirect to the same website. Is this an issue?
I have multiple URLs that all lead to the same website. Years ago they were purchased and were sitting dormant. Currently they are 301 redirects and each of the URLs feed to different areas of my website. Should I be worried about losing authority? And if so, is there a better way to do this?
Technical SEO | | undrdog990 -
Websites being hacked & duplicated, what should we do?
Hi, please help! Our website was hacked and being totally duplicated. They even injected codes to intercept our orders. Although the codes issue had been solved, still there're two mirror sites out there. When search for some of our key words, they even have good ranks. What exactly can we do to let Google ban those two sites. Thanks in advance!
Technical SEO | | Squall3150 -
Does redesigning the website effects the SEO?
What are the precautions to be taken in redesigning the website ? do it effect on link building? I am planing to re design my website, most of the Keywords are already optimized by Google, and i have given many back links to it . After redesigning my website will it get effected? Kindly answer my question
Technical SEO | | PrasanthMohanachandran0 -
If a redirecting URL has more value than the website should I move it?
Client has two website addresses: Website A is a redirect to Website B. It has one indexed page. But this is the URL being used in collateral. It has the majority of back links, and citations everywhere list Website A as the URL. Website B is where the actual website lives. Google recognizes and indexes the 80+ pages. This website has very few backlinks going to it. This setup does not seem good for SEO. Moreover, the analytics data is completely messed up because Website B shows that the biggest referral source is... you guessed it Website A. I'm thinking going forward, I should: Move all the content from Website B to Website A. Setup Website B to permanently 301 Redirect to Website A. Is that the best course of action?
Technical SEO | | flowsimple0
