Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
-
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this:
User-Agent: rogerbot Crawl-Delay: 5
Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop.
Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
-
Hello Dave. Thanks for your reply. We are aware this is not affecting us being temporary and exclusive to the MOZ bot so that's why we are worried about the data-set issues.
As I mentioned most of our excluded content are products, we can't be certain that MOZ has every keyword and that the ones discovered are being weighted correctly.
Understandably Shopify might never make robots.txt available so it would be nice for MOZ to identify the web as a shop hosted on Shopify (a moz.txt file) and apply a rate limiting, at the very least allow the user to control the crawl parameters from our control panels for those SaaS apps that block these core functions.
Hope MOZ and Shopify one day have a coffee and find a way to figure this out. But meanwhile, Is there any way to request crawls in specific folders? something like "domain.com/products/*****"
-
hey, Dave from the Help Team here.
The 430 error seems to be a result of shopify blocking our bot from accessing those pages temporarily. We have seen instances where this clears up after the second crawl, so keep your eye out for your weekly campaign update email in the meantime.
The good news is, that your human visitors will still be able to access your pages to do their shopping, phew!
Thanks so much for letting us know. We'll track this issue and look into a fix. I'm sorry I don't have better news for you at this time.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Factors
Hello Folks, I just wanna know that on which factors does Moz calculates DA ...? and also wanna know my site DA can anyone tell me Helmet Best
Moz Bar | | Alisa8844880 -
Moz not updating the spam score metrics
Hi Experts, I have done everything to come over the following metrics flagged as spam on my website by Moz, few months ago. But, Moz has not updated the spam score yet. ✓Low MozTrust or MozRank Score - improved from 2 to 4.5
Moz Bar | | jamesh.rich01
✓Large Site with Few Links - My website have more than 6K backlinks
✓Small Proportion of Branded Links - My website have a good amount of branded backlinks
✓Thin Content - Every webpage on website has more than 500 words content
✓External Links in Navigation - There is no external link in navigation other than social media links
✓No Contact Info - The proper address has alreay been placed on website footer
✓Low Number of Pages Found - I am wondering if there are any standard score or number of links to reach to remove these flag?
Also, please suggest some ways that will help me improve moz spam score at faster rate. Thanks for your help in advance!"0 -
What does the Bold/ Strong mean in Moz bar?
Under On-Page Elements in the Moz bar there is a Tag/ Location called Bold/ Strong. What does that mean?
Moz Bar | | TiffanyatElite0 -
Has using Moz got me banned from Google search?
Dear fellow Mozzers, For the last couple of weeks when ever I do a google search I have to enter a captcha code. Now I hardly use google anymore which is annoying ! Then I went onto support.google.com and saw this: https://support.google.com/websearch/answer/86640?hl=en&ref_topic=3378866 "Automated traffic includes: Sending searches from a robot, computer program, automated service, or search scraper Using software that sends searches to Google to see how a website or webpage ranks on Google" Is Moz alone doing this? Thanks in advance for your response. Ash
Moz Bar | | -Ash-0 -
Moz reporting for C-Blocking
Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330. Does this mean 330 Ip addresses came from the same location in the month of november?
Moz Bar | | rpaiva1 -
I want to uninstall the Moz SEO toolbar. How do I do this?
I installed the Moz toolbar and I don't understand it and it covers up important parts of websites and makes them inaccessible. I want to get it off my computer. I installed it in chrome. How do I get it off?
Moz Bar | | Bonnie761 -
How Does Moz Shoot Whiteboard Fridays?
Okay, sort of a meta question here... My company, a global test preparation company, is looking into new ways to deliver online course content (both pre-recorded and live streaming). Around the office we have looked at dozens of examples of companies that do video presenting or teaching, and then I realized that I watch a great example of high-quality online video every week: Whiteboard Fridays! They look and sound fantastic! Moz team, what are you willing to share re: the setup you use? I'm looking for specific direction about lighting, sound, and equipment. Anything you share would be VERY much appreciated. Thanks!
Moz Bar | | ScottShrum
Scott 3-methods-fueled-by-data-and-tools-to-earn-more-and-better-links-whiteboard-friday0
