• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. Moz Tools
        3. Moz Pro
        4. Link Explorer
        5. Moz crawling bot

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Moz crawling bot

        Link Explorer
        2
        3
        4632
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • superseopl
          superseopl last edited by

          Hi guys,

          in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".

          I tried to find out what the ID of Moz crawling bot is, and on this page: http://a-moz.groupbuyseo.org/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).

          However, my robots.txt doesn't have such entry. Take a look:

          Automatically banned scanners and crawlers section

          User-agent: 008

          Disallow: /

          user-agent: AhrefsBot

          Disallow: /

          User-agent: MJ12bot

          Disallow: /

          User-agent: metajobbot

          Disallow: /

          User-agent: Exabot

          Disallow: /

          User-agent: Ezooms

          Disallow: /

          User-agent: fyberspider

          Disallow: /

          User-agent: dotbot

          Disallow: /

          User-agent: MojeekBot

          Disallow: /

          Section end

          What could be the problem here, then? Why does the Moz bot think I'm blocking it?

          1 Reply Last reply Reply Quote 0
          • superseopl
            superseopl @KeriMorgret last edited by

            Thank you Keri, problem solved.

            1 Reply Last reply Reply Quote 0
            • KeriMorgret
              KeriMorgret last edited by

              If you pull out dotbot from your exclusions, you should be fine.

              http://www.opensiteexplorer.org/dotbot

              superseopl 1 Reply Last reply Reply Quote 2
              • 1 / 1
              • First post
                Last post

              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • dlwnsh0510

                Unsolved Google redirect backlink created. Do I have to ping this for Moz to recognize it?

                backlinks

                Google redirect backlink created. Do I have to ping this for Moz to recognize it?

                Link Explorer | | dlwnsh0510
                0
              • Niujbuj76

                Problem with third party cookies while using MOZ?

                Hello, I am using Moz Pro and I am happy to have this tool for my SEO work. But I am having a little problem while working on my blog site https://thepoolparadise.com/ some times the Moz does not work and asks to allow third-party cookies, wherein the settings third-party cookies are always allowed. Please guide me in this matter. Thank you.

                Link Explorer | | Niujbuj76
                0
              • KOidue

                Why my website Backlinks not getting Crawl by Moz?

                Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain

                Link Explorer | | KOidue
                0
              • amzonestep

                My Website's Links Are Not Showing In Moz Link Explorer

                Hello Team, I was analyzing my domain (https://www.amzonestep.com) in link explored but there are many websites that are not showing there. Questions
                1. Is this reason website's DA is not increasing 2. Is there wrong steps taken by me for this website. Well, I know there are many factors work in increasing DA but what is should do if these kind on links are not indexing in MOZ. I things these are one of the factors my DA is not Increasing. Please save my job guys. The company has sent me an ultimatum mail of 2 months that they will take back the projects from me. Please save my job. Thanks & Regards Aashirvad Kumar moz.png Screenshot-2020-02-19-at-11.48.21-am.png

                Link Explorer | | amzonestep
                0
              • NathanPetralia

                MOZ can't find internal links on my website while Google search console finds thousands of them

                When using open site explorer MOZ finds 0 internal links on https://www.glassesgallery.com while Google search console finds thousands of internal links. What could the issue be? Cr8Wk14 oGOf1gb

                Link Explorer | | NathanPetralia
                0
              • MarcoBP

                Spam Score and crawling of my site

                Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco

                Link Explorer | | MarcoBP
                0
              • wearehappymedia

                Crawling and Links - Showing Links

                Hello I have a campaign in Moz that shows a new site as having no domain authority and no links. The site does have links to it, but they don't show up on OSE. Does Moz count links in the comments section of blogs (or is this counted as social, when the commenting system is one that requires facebook login?) I know blog comments are not great for SEO, I just use them when the post/link is 100% genuinely relevant. The site does have actual links on websites not just comment links, so it isn't showing any type of link. What is the best way to see the links the site has? Thanks!

                Link Explorer | | wearehappymedia
                1
              • seoprojecter

                Does using Sucuri block Moz?

                I am using Opensite explorer on my site which uses Sucuri and it shows 0 for everything. Does Sucuri stop MOZ from reading the link? I also suspect that using Sucuri has made my SEO suffer because the first page is always saying "redirecting" . Anyone with experience to this? Thanks

                Link Explorer | | seoprojecter
                0

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.