Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://a-moz.groupbuyseo.org/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my website Backlinks not getting Crawl by Moz?
Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain
Link Explorer | | KOidue0 -
Ranking Keywords are not showing on Moz
i just check my Apk Finds on moz site explorer and check its ranking keywords but it does not show it please help me my site age is 3 months
Link Explorer | | malik12378670 -
Does Moz register backlinks from Twitter/Facebook/LinkedIn?
I've noticed that on sites with decent social presence and regular content posts that Moz isn't showing/recording any inbound links from Twitter, Facebook or LinkedIn. Posts are regularly shared via these three channels but no inbound links ever seem to show up. Does Moz not record such inbound links, or are they all disguised using JavaScript or some other trick?
Link Explorer | | JCN-SBWD0 -
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
Moz's new Link Explorer, including our revamped index and DA/PA scores is now open to everyone!
Hey Moz Community, Link Explorer is now open to the public! Everyone can access it via a subscription or a free Moz ‘Community’ account. As you may know by now, the brand-new Link Explorer tool is primed to replace Open Site Explorer as Moz’s link building and analysis tool. The Link Explorer project is the result of an incredible amount of perseverance and hard work by the team, and we’re proud to be able to finally share it with you — we know it’s going to revolutionize how you approach link building and make your job easier. You can read more about the tool here in Sarah Bird’s announcement post. Because Link Explorer improves on almost every aspect of Open Site Explorer, the metrics have improved, too. That means you’re likely going to see some Domain Authority and Page Authority discrepancies between OSE’s index and Link Explorer’s index. We definitely suggest you use the new DA/PA from Link Explorer, as they’re more accurate and refresh daily rather than monthly, as was the case with OSE’s index. However, we also realize that many of you use these metrics to report to your clients and colleagues, and a sudden change or fluctuation could potentially make your job harder. Which DA is the real DA? The new DA is based on a much larger index that has many improvements, several of which are designed to make the index more like Google’s than ever before. You should consider moving towards the new DA (and the old DA won’t be updated after April 26th 2018, so the sooner the better). While there will be fluctuations as we improve the model and add features to the index, we expect it to remain largely stable and to be a far more accurate picture of a site’s authority according to how it’s seen by Google. Why is Link Explorer’s DA/PA considered better than OSE’s, and which should I trust? The larger link index with improved crawl selection allows us to produce a stronger model that includes a much larger proportion of the web. That being said, DA and PA should always be considered in the context of your competitors. A drop in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Is Domain Authority/Page Authority an absolute score or a relative one? Both DA and PA are relative to the Internet as a whole. If Facebook acquired a billion new links, everyone’s PA and DA would drop relative to Facebook. Because of this, it’s always best to look at PA and DA in comparison to your competitors. What does a drop/raise in DA mean in Link Explorer vs OSE? How can I explain this to my clients when I’m reporting it? DA and PA should always be considered in the context of your competitors. A drop or raise in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Reporting that your site has moved from a DA of 45 to a DA of 42 doesn’t tell the whole story, but reporting that your site has a DA of 42 while your main competitor moved from a 43 to a 37 shows that, relative to the sites you’re competing against in the SERPs, your site has significantly more authority and ranking power. What’s happening to MozTrust and MozRank and why, and what should I replace those with? The improvements to our DA/PA and Spam Score metrics now now account for more important nuances in helping you determine one site’s ability to rank higher than another. Because they no longer correlate with Google’s ranking model as well as they used to, MozRank and MozTrust are being deprecated for better metrics. Users should rely on Page Authority, Domain Authority, and Spam Score to determine the importance and quality of pages, domains, and links. I have historical data I use to help my clients benchmark their progress. What do I do now that DA is calculated differently? You should annotate any KPI changes referencing the change in DA and PA. However, most importantly, you should compare those changes to your competitors, as this will best show how strong your site’s authority is relative to the sites you’re competing against in the SERPs. We take updating our metrics very seriously, and our last major update to the model was 7 years ago. Users of Domain Authority and Page Authority can expect us to continue to produce steady, reliable metrics for the long haul, and only make changes to these metrics when we believe the benefits dramatically outweigh the stability of the metric. Do you have any questions about the new metrics? Anticipating a tough time reporting changes to clients or bosses? Metrics, features or functionality missing that you would want to see? Let us know in the thread, and we’ll work to find a good answer for you. Hope you enjoy the new Link Explorer product and the amazing new link index powering it. We are very excited to provide this valuable data to our community and customers.
Link Explorer | | IanWatson9 -
Does using Sucuri block Moz?
I am using Opensite explorer on my site which uses Sucuri and it shows 0 for everything. Does Sucuri stop MOZ from reading the link? I also suspect that using Sucuri has made my SEO suffer because the first page is always saying "redirecting" . Anyone with experience to this? Thanks
Link Explorer | | seoprojecter0 -
The difference between the statistics MOZ & Ahrefs
Hello dear. I have a website for a year. I made 10K links in one year(from 45 Root Domains).the Ahrefs.com has show full list of link root domains and pages. the ahrefs rank is 49. but MOZ said my Established Links are 5! So I have some questions, please help me: 1- what is difference between MOZ rank and ahrefs rank? 2-When Just-Discovered Link Calculate for my site? 3-why my moz trust is 0.00!?How to improve it? URL: https://esta-register.org
Link Explorer | | allyunit0
