• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. Intermediate & Advanced SEO
        4. XML sitemap generator only crawling 20% of my site

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        XML sitemap generator only crawling 20% of my site

        Intermediate & Advanced SEO
        10
        12
        2632
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • TyEl
          TyEl last edited by

          Hi guys,

          I am trying to submit the most recent XML sitemap but the sitemap generator tools are only crawling about 20% of my site.  The site carries around 150 pages and only 37 show up on tools like xml-sitemaps.com. My goal is to get all the important URLs we care about into the XML sitemap.

          How should I go about this?

          Thanks

          1 Reply Last reply Reply Quote 0
          • seotools25
            seotools25 last edited by

            I believe it's not a significant issue if the sitemap encompasses the core framework of your website. As long as the sitemap is well-organized, omitting a few internal pages is acceptable since Googlebot will crawl all pages based on the sitemap. Take a look at the <a href="https://convowear.in">example page</a> that also excludes some pages, yet it doesn't impact the site crawler's functionality.

            1 Reply Last reply Reply Quote 0
            • ChunkyChuck
              ChunkyChuck last edited by

              Yes Yoast on WordPress works fine for sitemap generation. I would also recommend that. Using on all of my blog sites.

              1 Reply Last reply Reply Quote 0
              • noahjohn7788
                noahjohn7788 last edited by

                If you are using WordPress then I would recommend to use Yoast plugin. It generates sitemap automatically regularly. I am also using it on my blog.

                1 Reply Last reply Reply Quote 0
                • ndsjiu7823
                  ndsjiu7823 Banned last edited by

                  I'm using Yoast SEO plugin for my website. It generates the Sitemap automatically.

                  1 Reply Last reply Reply Quote 0
                  • Smantha
                    Smantha last edited by

                    My new waterproof tent reviews blog facing the crawling problem. How can I fix that?

                    1 Reply Last reply Reply Quote 0
                    • asdasdfasf
                      asdasdfasf last edited by

                      use Yoast or rankmath ot fix it

                      آموزش سئو در اصفهان https://faneseo.com/seo-training-in-isfahan/

                      1 Reply Last reply Reply Quote 0
                      • Nozzle
                        Nozzle @TyEl last edited by

                        Patrick wrote a list of reasons why Screaming Frog might not be crawling certain pages here: https://a-moz.groupbuyseo.org/community/q/screamingfrog-won-t-crawl-my-site#reply_300029.

                        Hopefully that list can help you figure out your site's specific issue.

                        1 Reply Last reply Reply Quote 0
                        • TyEl
                          TyEl last edited by

                          This doesn't really answer my question of why I am not able to get all links into the XML sitemap when using xml sitemap generators.

                          1 Reply Last reply Reply Quote 0
                          • Graduationmall
                            Graduationmall last edited by

                            I think it's not a big deal if the sitemap covers the main structure of your site. If your sitemap is constructed in a really decent structure, then missing some internal pages are acceptable because Googlebot will crawl all of your pages based on your site map. You can see the following page which also doesn't cover all of its pages, but there's no influence in terms of site crawler.

                            1 Reply Last reply Reply Quote 0
                            • TyEl
                              TyEl last edited by

                              Thanks Boyd but unfortunately I am still missing a good chunk of URLs here and I am wondering why? Do those check on internal links in order to find these pages?

                              Nozzle 1 Reply Last reply Reply Quote 0
                              • Nozzle
                                Nozzle last edited by

                                Use Screaming Frog to crawl your site. It is free to download the software and you can use the free version to crawl up to 500 URLs.

                                After it crawls your site you can click on the Sitemaps tab and generate an XML sitemap file to use.

                                1 Reply Last reply Reply Quote 0
                                • 1 / 1
                                • First post
                                  Last post

                                Browse Questions

                                Explore more categories

                                • Moz Tools

                                  Chat with the community about the Moz tools.

                                • SEO Tactics

                                  Discuss the SEO process with fellow marketers

                                • Community

                                  Discuss industry events, jobs, and news!

                                • Digital Marketing

                                  Chat about tactics outside of SEO

                                • Research & Trends

                                  Dive into research and trends in the search industry.

                                • Support

                                  Connect on product support and feature requests.

                                • See all categories

                                Related Questions

                                • dmaher

                                  Spotify XML Sitemap

                                  All, Working on an SEO work up for a Spotify site.  Looks like they are using a sitemap that links to additional pages. A problem, none of the links are actually linked within the sitemap.   This feels like a strong error. https://lubricitylabs.com/sitemap.xml Thoughts?

                                  Intermediate & Advanced SEO | | dmaher
                                  0
                                • thekiller99

                                  Sitemap: unique sitemap or different sitemaps by Country

                                  Hi guys, i have a question about sitemaps. We are doing an international site, e.x. www.offers.com for landing page and www.offers.com/br for brazil, www.offers.com/it for italy, etc... i don't if we should do an unique sitemap for all countries or separate sitemaps by country, e.x.: unique sitemap: www.offers.com/sitemap.xml - including all sitemaps www.offers.com/br/sitemap.xml - sitemap for brazil market only. Thank you

                                  Intermediate & Advanced SEO | | thekiller99
                                  0
                                • McTaggart

                                  Why do people put xml sitemaps in subfolders? Why not just the root? What's the best solution?

                                  Just read this: "The location of a Sitemap file determines the set of URLs that can be included in that Sitemap. A Sitemap file located at http://example.com/catalog/sitemap.xml can include any URLs starting with http://example.com/catalog/ but can not include URLs starting with http://example.com/images/." here: http://www.sitemaps.org/protocol.html#location Yet surely it's better to put the sitemaps at the root so you have:
                                  (a) http://example.com/sitemap.xml 
                                  http://example.com/sitemap-chocolatecakes.xml
                                  http://example.com/sitemap-spongecakes.xml 
                                  and so on... OR this kind of approach - 
                                  (b) http://example/com/sitemap.xml
                                  http://example.com/sitemap/chocolatecakes.xml and 
                                  http://example.com/sitemap/spongecakes.xml I would tend towards (a) rather than (b) - which is the best option? Also, can I keep the structure the same for sitemaps that are subcategories of other sitemaps - for example - for a subcategory of http://example.com/sitemap-chocolatecakes.xml I might create http://example.com/sitemap-chocolatecakes-cherryicing.xml - or should I add a sub folder to turn it into http://example.com/sitemap-chocolatecakes/cherryicing.xml Look forward to reading your comments - Luke

                                  Intermediate & Advanced SEO | | McTaggart
                                  0
                                • ang

                                  URL structure change and xml sitemap

                                  At the end of April we changed the url structure of most of our pages and 301 redirected the old pages to the new ones. The xml sitemaps were also updated at that point to reflect the new url structure. Since then Google has not indexed the new urls from our xml sitemaps and I am unsure of why. We are at 4 weeks since the change, so I would have thought they would have indexed the pages by now. Any ideas on what I should check to make sure pages are indexed?

                                  Intermediate & Advanced SEO | | ang
                                  0
                                • recbrands

                                  Sitemap generator which only includes canonical urls

                                  Does anyone know of a 3rd party sitemap generator that will only include the canonical url's?  Creating a sitemap with geo and sorting based parameters isn't the most ideal way to generate sitemaps.  Please let me know if anyone has any ideas.  Mind you we have hundreds of thousands of indexed url's and this can't be done with a simple text editor.

                                  Intermediate & Advanced SEO | | recbrands
                                  0
                                • Lakshdeep

                                  XML Sitemap index within a XML sitemaps index

                                  We have a similar problem to http://www.seomoz.org/q/can-a-xml-sitemap-index-point-to-other-sitemaps-indexes Can a XML sitemap index point to other sitemaps indexes? According to the "Unique Doll Clothing" example on this link, it seems possible http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic Can someone share an XML Sitemap index within a XML sitemaps index example? We are looking for the format to implement the same on our website.

                                  Intermediate & Advanced SEO | | Lakshdeep
                                  0
                                • nicole.healthline

                                  How important are sitemap errors?

                                  If there aren't any crawling / indexing issues with your site, how important do thing sitemap errors are? Do you work to always fix all errors? I know here: http://www.seomoz.org/blog/bings-duane-forrester-on-webmaster-tools-metrics-and-sitemap-quality-thresholds Duane Forrester mentions that sites with many 302's 301's will be punished--does any one know Googe's take on this?

                                  Intermediate & Advanced SEO | | nicole.healthline
                                  0
                                • CareerBliss

                                  Can a XML sitemap index point to other sitemaps indexes?

                                  We have a massive site that is having some issue being fully crawled due to some of our site architecture and linking. Is it possible to have a XML sitemap index point to other sitemap indexes rather than standalone XML sitemaps? Has anyone done this successfully? Based upon the description here: http://sitemaps.org/protocol.php#index it seems like it should be possible. Thanks in advance for your help!

                                  Intermediate & Advanced SEO | | CareerBliss
                                  0

                                Get started with Moz Pro!

                                Unlock the power of advanced SEO tools and data-driven insights.

                                Start my free trial
                                Products
                                • Moz Pro
                                • Moz Local
                                • Moz API
                                • Moz Data
                                • STAT
                                • Product Updates
                                Moz Solutions
                                • SMB Solutions
                                • Agency Solutions
                                • Enterprise Solutions
                                • Digital Marketers
                                Free SEO Tools
                                • Domain Authority Checker
                                • Link Explorer
                                • Keyword Explorer
                                • Competitive Research
                                • Brand Authority Checker
                                • Local Citation Checker
                                • MozBar Extension
                                • MozCast
                                Resources
                                • Blog
                                • SEO Learning Center
                                • Help Hub
                                • Beginner's Guide to SEO
                                • How-to Guides
                                • Moz Academy
                                • API Docs
                                About Moz
                                • About
                                • Team
                                • Careers
                                • Contact
                                Why Moz
                                • Case Studies
                                • Testimonials
                                Get Involved
                                • Become an Affiliate
                                • MozCon
                                • Webinars
                                • Practical Marketer Series
                                • MozPod
                                Connect with us

                                Contact the Help team

                                Join our newsletter
                                Moz logo
                                © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                • Accessibility
                                • Terms of Use
                                • Privacy

                                Looks like your connection to Moz was lost, please wait while we try to reconnect.