Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
-
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this:
User-Agent: rogerbot Crawl-Delay: 5
Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop.
Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
-
Hello Dave. Thanks for your reply. We are aware this is not affecting us being temporary and exclusive to the MOZ bot so that's why we are worried about the data-set issues.
As I mentioned most of our excluded content are products, we can't be certain that MOZ has every keyword and that the ones discovered are being weighted correctly.
Understandably Shopify might never make robots.txt available so it would be nice for MOZ to identify the web as a shop hosted on Shopify (a moz.txt file) and apply a rate limiting, at the very least allow the user to control the crawl parameters from our control panels for those SaaS apps that block these core functions.
Hope MOZ and Shopify one day have a coffee and find a way to figure this out. But meanwhile, Is there any way to request crawls in specific folders? something like "domain.com/products/*****"
-
hey, Dave from the Help Team here.
The 430 error seems to be a result of shopify blocking our bot from accessing those pages temporarily. We have seen instances where this clears up after the second crawl, so keep your eye out for your weekly campaign update email in the meantime.
The good news is, that your human visitors will still be able to access your pages to do their shopping, phew!
Thanks so much for letting us know. We'll track this issue and look into a fix. I'm sorry I don't have better news for you at this time.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Factors
Hello Folks, I just wanna know that on which factors does Moz calculates DA ...? and also wanna know my site DA can anyone tell me Helmet Best
Moz Bar | | Alisa8844880 -
How do you use Moz to research related topics?
Like most of the folks here I'm a pretty big fan of the content that comes out through Whiteboard Fridays, and I try to apply the things I learn, but one of the WBF videos that I'm following along with does not do a stellar job of detailing execution using Moz KW Explorer. https://a-moz.groupbuyseo.org/blog/related-topics-in-seo-whiteboard-friday Now granted, this came out in 2016, but I still feel the core principle and strategy results in a higher quality piece of content and is still relevant to discovering and understanding searcher task completion requirements, and drafting content that fulfills those requirements. Towards the end Rand sort of mentions that you'll be able to do this with KW explorer, but I'm not really seeing the functionality. The steps I followed were to enter in the keyword in kw explorer, went to keyword suggestions, and selected "based on closely related topics" and ran it, but received no suggestions - came up blank. I then selected "based on broadly related topics" and the same thing happened. I tried this out with the keyword r22, keeping it very broad to start but that didn't seem to work. So what do you all do to perform this sort of research within Moz? Or do you even feel it's relevant in today's Rank Brain driven world?
Moz Bar | | brettmandoes0 -
Moz Bar doesn't show any data and keeps asking me to log in when actually I'm logged in.
Hi all, I've been using Moz Bar for years. It ran well until about three weeks ago. It suddenly failed to show the DA and PA of sites that I open after I log in. And it keeps asking me to log in when I did. I tried to uninstall the Mozbar extension and reinstalled it several times. Nothing worked. I also tried to uninstall Chrome and clear the cookies, still, nothing changed. Did anyone experience this? How do you solve it and make it run on the track? Any information will be appreciated. [admin edited support category]
Moz Bar | | Bennie223310 -
Crawling Problem : No-data Of My Site
Hey, Folks!! So It's Been More than a Month.And Moz is not crawling my website http://www.trickypedia.com/.I have seen Moz Last update I thought they will crawl my site but No results Disappointing. My Sites Domain Authority and Page Authority is still 1. But In other Seo Tools, they are Perfectly Crawling My Website. What Would Be the reason? Can anyone Please Explain.
Moz Bar | | seothatworks010 -
Moz Keyword Tool Monthly Volume
Ive recently put together a Keyword List of about 100 keywords on the Moz Keyword Explorer tool. One keyword, aerial filming, stood out as very low search volume of 51 - 100. I took the same 100 keywords and passed them through the Google Keyword Planner by Google AdWords. Aerial Filming has an average search volume of 1k - 10k according to the Keyword Planner. Even though Keyword Planner gives me a range of 1k - 10k, the lowest number is still 10 times higher than what the Moz Keyword Explorer was indicating. This drastic difference of volume was consistent across all 100 keywords. All of the Monthly Volume numbers were divided by 10. Why does Moz Keyword Tool display a search volume that is 10x less than what Google Keyword Planner is suggesting?
Moz Bar | | fictionarts0 -
Moz Bar Not Showing DA?
Hello all, This could be something to do with our site or the Moz bar on Chrome, I just need to know which it is so if it is our website we can look into it further. On certain sections of our website the Moz bar doesn't display any Domain Authority, not even zero, the bar just isn't present. These types of pages are php which pull in data through a feed daily. Speaking to an SEO expert they said it could be where the page is being updated so frequently, or it could be something more sinister and technically not quite right. Does anyone have any ideas? Is the Moz bar just not working for these types os pages or is it more likely something to do with my site? Ironically it's these pages which I'm having trouble with that are not showing in SERPs! Thanks! 3Foorka
Moz Bar | | HB177 -
Should I exclude prepositions in tracked keywords of moz analytics?
I'm new to Moz. Just set up my trial campaign, and it had suggested many keywords. Many of the phrases that were suggested do not contain prepositions. For example, instead of something like "sporting good stores in Chicago" it suggested "sporting good stores Chicago" Today, I looked at the on-page optimization suggestions, which are (of course) suggesting that I remove prepositions from my page to rank well. Well, as you know, that is unnatural to the reader. But I suspect people are searching in higher volume, leaving the prepositions out. I know that if I were to search for a sporting goods store in Chicago, I would probably leave out "in." What should I do? Should I remove all the suggested keywords, and make them readable (which people are less like using in their search?) Do I go back to all my pages and try to optimize it for a keyword that is natural, but does not include a preposition (such as Chicago sporting goods stores) or should I be doing something else?
Moz Bar | | osaka731 -
Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?
Hi Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ? Is there an option i havnt seen to add logins/passwords for crawl software to access ? cheers dan
Moz Bar | | Dan-Lawrence0