XML sitemap generator only crawling 20% of my site

[TyEl]

Hi guys,

I am trying to submit the most recent XML sitemap but the sitemap generator tools are only crawling about 20% of my site.  The site carries around 150 pages and only 37 show up on tools like xml-sitemaps.com. My goal is to get all the important URLs we care about into the XML sitemap.

How should I go about this?

Thanks

[seotools25]

I believe it's not a significant issue if the sitemap encompasses the core framework of your website. As long as the sitemap is well-organized, omitting a few internal pages is acceptable since Googlebot will crawl all pages based on the sitemap. Take a look at the <a href="https://convowear.in">example page</a> that also excludes some pages, yet it doesn't impact the site crawler's functionality.

[ChunkyChuck]

Yes Yoast on WordPress works fine for sitemap generation. I would also recommend that. Using on all of my blog sites.

[noahjohn7788]

If you are using WordPress then I would recommend to use Yoast plugin. It generates sitemap automatically regularly. I am also using it on my blog.

[ndsjiu7823]

I'm using Yoast SEO plugin for my website. It generates the Sitemap automatically.

[Smantha]

My new waterproof tent reviews blog facing the crawling problem. How can I fix that?

[asdasdfasf]

use Yoast or rankmath ot fix it

آموزش سئو در اصفهان https://faneseo.com/seo-training-in-isfahan/

[Nozzle]

Patrick wrote a list of reasons why Screaming Frog might not be crawling certain pages here: https://a-moz.groupbuyseo.org/community/q/screamingfrog-won-t-crawl-my-site#reply_300029.

Hopefully that list can help you figure out your site's specific issue.

[TyEl]

This doesn't really answer my question of why I am not able to get all links into the XML sitemap when using xml sitemap generators.

[Graduationmall]

I think it's not a big deal if the sitemap covers the main structure of your site. If your sitemap is constructed in a really decent structure, then missing some internal pages are acceptable because Googlebot will crawl all of your pages based on your site map. You can see the following page which also doesn't cover all of its pages, but there's no influence in terms of site crawler.

[TyEl]

Thanks Boyd but unfortunately I am still missing a good chunk of URLs here and I am wondering why? Do those check on internal links in order to find these pages?

[Nozzle]

Use Screaming Frog to crawl your site. It is free to download the software and you can use the free version to crawl up to 500 URLs.

After it crawls your site you can click on the Sitemaps tab and generate an XML sitemap file to use.