• ramc-7JcUnB

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Turn SEO data into actionable Content Briefs
          Moz Pro

          Turn SEO data into actionable Content Briefs

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Access 20 years of data with flexible pricing
          Moz API

          Access 20 years of data with flexible pricing

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. Moz Tools
        3. Moz Pro
        4. Link Explorer
        5. Moz crawling bot

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Moz crawling bot

        Link Explorer
        2
        3
        4701
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • superseopl
          superseopl last edited by

          Hi guys,

          in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".

          I tried to find out what the ID of Moz crawling bot is, and on this page: http://a-moz.groupbuyseo.org/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).

          However, my robots.txt doesn't have such entry. Take a look:

          Automatically banned scanners and crawlers section

          User-agent: 008

          Disallow: /

          user-agent: AhrefsBot

          Disallow: /

          User-agent: MJ12bot

          Disallow: /

          User-agent: metajobbot

          Disallow: /

          User-agent: Exabot

          Disallow: /

          User-agent: Ezooms

          Disallow: /

          User-agent: fyberspider

          Disallow: /

          User-agent: dotbot

          Disallow: /

          User-agent: MojeekBot

          Disallow: /

          Section end

          What could be the problem here, then? Why does the Moz bot think I'm blocking it?

          1 Reply Last reply Reply Quote 0
          • superseopl
            superseopl @KeriMorgret last edited by

            Thank you Keri, problem solved.

            1 Reply Last reply Reply Quote 0
            • KeriMorgret
              KeriMorgret last edited by

              If you pull out dotbot from your exclusions, you should be fine.

              http://www.opensiteexplorer.org/dotbot

              superseopl 1 Reply Last reply Reply Quote 2
              • 1 / 1
              • First post
                Last post

              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • dlwnsh0510

                Unsolved Google redirect backlink created. Do I have to ping this for Moz to recognize it?

                backlinks

                Google redirect backlink created. Do I have to ping this for Moz to recognize it?

                Link Explorer | | dlwnsh0510
                0
              • KOidue

                Why my website Backlinks not getting Crawl by Moz?

                Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain

                Link Explorer | | KOidue
                0
              • jessicapremier

                How is Moz DA affected by spam links? Disavow file?

                So it does not appear that moz let's you upload your disavow file.  So when moz calculates your DA how do spammy links factor in? After digging through our GA it appears our site was hit with the 2016 penguin update and never recovered.  Our weekly visitors were 2k, then dropped to 500 and have stayed close to that level for a while. We've used the disavow tool, without success over the past 3 years.  During that time we have done link out reach and built around 10 legit good quality DA links since.  But we have not recovered.  At this point i'm thinking I should just remove the disavow file. Moz says our spam score for our domain is 5%.

                Link Explorer | | jessicapremier
                0
              • TaylorRHawkins

                Is Moz's backlink checker.... just... not good?

                Hey everyone! Can somebody explain to me why this keeps happening: Whenever I'm trying to backlink my competitors, I typically use RavenTools. Every time, without fail, if I put that same URL into Moz's Open Site Explorer - It gives me about 1/20th of what RavenTools shows me. Sometimes it literally comes up with 2 or 3 links total. Unfortunately, RavenTools has a cap on how many backlink checks you can perform in a month - so once I've used those up, I have to start using OSE... But, it just doesn't work. Does anyone else have this issue? Thanks!

                Link Explorer | | TaylorRHawkins
                1
              • NHARRIS1

                Fastest way to get new backlinks recognised by Moz?

                Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links.  I am not convinced that these techniques are working. On a related note, my website is very large.  I have around 2.5m pages.  Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page).  Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again?  If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly?  Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.

                Link Explorer | | NHARRIS1
                0
              • jakebellacera

                Moz Pro: Linking RDs to Page much lower than Google Search Console

                I'm trying to use the Analyze Keyword tool in Moz Pro, and in the SERP Analysis table, my page has a PA of one, and zero root domains linking to it. If I look at the page in Google Search console, it says I have 229 root domains linking to the page from well known domains like github.com, meetup.com, stackoverflow.com, etc. This particular keyword has been tracked in Moz for the last 6 months, but I just noticed that it was extremely low. I am relatively new to Moz, so forgive me if I sound confused, but can someone explain to me how the numbers can be so low?

                Link Explorer | | jakebellacera
                0
              • wearehappymedia

                Crawling and Links - Showing Links

                Hello I have a campaign in Moz that shows a new site as having no domain authority and no links. The site does have links to it, but they don't show up on OSE. Does Moz count links in the comments section of blogs (or is this counted as social, when the commenting system is one that requires facebook login?) I know blog comments are not great for SEO, I just use them when the post/link is 100% genuinely relevant. The site does have actual links on websites not just comment links, so it isn't showing any type of link. What is the best way to see the links the site has? Thanks!

                Link Explorer | | wearehappymedia
                1
              • SWD.Advertising

                Does Moz have anything similar to ahrefs batch analysis tool?

                I am wanting to do analysis of a whole bunch of URLs at once - i know ahrefs already has a very good tool for this - but I don't really want to have to pay the $79 a month if moz has something similar? I know it has the OSE. Can I do batch analysis in this? Thanks

                Link Explorer | | SWD.Advertising
                1

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.