Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
-
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this:
User-Agent: rogerbot Crawl-Delay: 5
Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop.
Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
-
Hello Dave. Thanks for your reply. We are aware this is not affecting us being temporary and exclusive to the MOZ bot so that's why we are worried about the data-set issues.
As I mentioned most of our excluded content are products, we can't be certain that MOZ has every keyword and that the ones discovered are being weighted correctly.
Understandably Shopify might never make robots.txt available so it would be nice for MOZ to identify the web as a shop hosted on Shopify (a moz.txt file) and apply a rate limiting, at the very least allow the user to control the crawl parameters from our control panels for those SaaS apps that block these core functions.
Hope MOZ and Shopify one day have a coffee and find a way to figure this out. But meanwhile, Is there any way to request crawls in specific folders? something like "domain.com/products/*****"
-
hey, Dave from the Help Team here.
The 430 error seems to be a result of shopify blocking our bot from accessing those pages temporarily. We have seen instances where this clears up after the second crawl, so keep your eye out for your weekly campaign update email in the meantime.
The good news is, that your human visitors will still be able to access your pages to do their shopping, phew!
Thanks so much for letting us know. We'll track this issue and look into a fix. I'm sorry I don't have better news for you at this time.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain Analysis Limit
Hello, even though we are a Moz Pro user we have a 'part-timer' who keeps the client metrics scoreboard up to date. Rather than have him running wild in Moz Pro we bookmarked the 'Domain Analysis' page and he knows that only 3 can be reported on per day. If it was an immediate need then we would use Moz-Pro. The issue was reported to me that he is receiving 'Search Limit reached' and do a moz-pro trial. So, has the 3 per day limit been changed? Thanks, Dave
Moz Bar | | 7thPowerInc0 -
How do you use Moz to research related topics?
Like most of the folks here I'm a pretty big fan of the content that comes out through Whiteboard Fridays, and I try to apply the things I learn, but one of the WBF videos that I'm following along with does not do a stellar job of detailing execution using Moz KW Explorer. https://a-moz.groupbuyseo.org/blog/related-topics-in-seo-whiteboard-friday Now granted, this came out in 2016, but I still feel the core principle and strategy results in a higher quality piece of content and is still relevant to discovering and understanding searcher task completion requirements, and drafting content that fulfills those requirements. Towards the end Rand sort of mentions that you'll be able to do this with KW explorer, but I'm not really seeing the functionality. The steps I followed were to enter in the keyword in kw explorer, went to keyword suggestions, and selected "based on closely related topics" and ran it, but received no suggestions - came up blank. I then selected "based on broadly related topics" and the same thing happened. I tried this out with the keyword r22, keeping it very broad to start but that didn't seem to work. So what do you all do to perform this sort of research within Moz? Or do you even feel it's relevant in today's Rank Brain driven world?
Moz Bar | | brettmandoes0 -
Moz says "Title Too Long", Yoast says title is the perfect length. Who's right?
For a bunch of my pages, the MOZ Crawl Report says "Title Too Long". Yoast on my site tells me that the titles are the correct length. How can these two things be at odds with each other? Which one is right?
Moz Bar | | TeamViviRealEstate0 -
Are we actually getting accurate data on keyword volumes from Moz (or other sources)?
I have a client who does patio furniture repair and restoration. When performing keyword research in Moz for terms like "patio furniture repair" I see that only 11-50 people in the entire US are searching for this term according to the Moz data. However, running an Adwords campaign currently and our top keyword is the phrase match for "patio furniture repair" which has generated over 100 clicks in just a couple of months in ONE county. Is there a better way to research more accurate results on search volume estimates? This makes organic SEO and keyword targeting hard! Thanks, Ricky
Moz Bar | | RickyShockley1 -
Learn how to use MozBar to analyze competitors' schema markup. Get you Daily SEO Fix!
Schema markup helps Google and Bing identify what your (and your competitors') website pages are all about and as a result, helps search engines better index them. Better indexing can lead to better rankings for relevant keywords and phrases and herein lies the opportunity. In today's Daily SEO FiX, Jordan shows you how to use MozBar to analyze the schema markup of the competition and optimize your own site and pages for rich snippets. If you don't have MozBar, no worries! You may download it for free here. This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Moz Bar | | kellyjcoop1 -
Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader
I just signed up for MOZ Pro for the first time today. Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg. I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?
Moz Bar | | spinoki0 -
Weird back link showed in moz crawl
Some time ago somebody from this site: http://dianibeach.com created a weird link to our site which had on the end db. Later we have realized that the link was coming from every footer on each page. I believe that the back links from footer does not have realy value and even the more of them the less value. We have asked the guy to remove that links as I thought it might harm our site more then help. Now I I was very surprised to find this link in moz crawl error as second top page on our site in current index??? Can somebody explain how is this possible?? The most ridiculous thing is that when I click on that link it realy opens our site! How is that possible, what is it? This is the link: http://villasdiani.com/?db Thank you very much for any help with this
Moz Bar | | Rebeca10 -
How Does Moz Shoot Whiteboard Fridays?
Okay, sort of a meta question here... My company, a global test preparation company, is looking into new ways to deliver online course content (both pre-recorded and live streaming). Around the office we have looked at dozens of examples of companies that do video presenting or teaching, and then I realized that I watch a great example of high-quality online video every week: Whiteboard Fridays! They look and sound fantastic! Moz team, what are you willing to share re: the setup you use? I'm looking for specific direction about lighting, sound, and equipment. Anything you share would be VERY much appreciated. Thanks!
Moz Bar | | ScottShrum
Scott 3-methods-fueled-by-data-and-tools-to-earn-more-and-better-links-whiteboard-friday0