• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. Technical SEO
        4. What tool do you use to check for URLs not indexed?

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        What tool do you use to check for URLs not indexed?

        Technical SEO
        24
        25
        26199
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • elephantseo
          elephantseo last edited by

          What is your favorite tool for getting a report of URLs that are not cached/indexed in Google & Bing for an entire site? Basically I want a list of URLs not cached in Google and a seperate list for Bing.

          Thanks,

          Mark

          Shiv24k 1 Reply Last reply Reply Quote 3
          • austinbenz8282
            austinbenz8282 last edited by

            I've had good results using Google Search Console for checking which URLs are indexed. It's pretty straightforward and gives a clear overview of any indexing issues halloweensquishmallows.

            1 Reply Last reply Reply Quote 0
            • Shiv24k
              Shiv24k @elephantseo last edited by

              @elephantseo https://docs.google.com/document/d/e/2PACX-1vQprHYh4feCfnhC72ldSnbCG0OoXBgqpKHC2bvlbs7hX3XZ8axnrY-LuIDS5-BJ_0mSsUlcT0_OyUvp/pub

              1 Reply Last reply Reply Quote 0
              • noitcudni
                noitcudni last edited by

                I can work on building this tool if there's enough interest.

                1 Reply Last reply Reply Quote 0
                • Adlanera
                  Adlanera last edited by

                  I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.

                  1 Reply Last reply Reply Quote 0
                  • CarlLarson
                    CarlLarson last edited by

                    This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
                    https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

                    Alas, after downloading and installing, it appears this feature was removed...

                    1 Reply Last reply Reply Quote 0
                    • CMC-SD
                      CMC-SD @kdaly100 last edited by

                      Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

                      1 Reply Last reply Reply Quote 0
                      • rickv101
                        rickv101 last edited by

                        Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

                        You can get it from her link which is http://bit.ly/tools-galore

                        In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

                        1 Reply Last reply Reply Quote 0
                        • Philip_Petrescu
                          Philip_Petrescu @Chris.McGiff last edited by

                          I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

                          1 Reply Last reply Reply Quote 0
                          • squareplug
                            squareplug last edited by

                            hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

                            1 Reply Last reply Reply Quote 0
                            • DanishWadhwa
                              DanishWadhwa last edited by

                              I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

                              1 Reply Last reply Reply Quote 0
                              • Chris.McGiff
                                Chris.McGiff @DavidKauzlaric last edited by

                                I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

                                Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

                                Still not definitive, but think it does give indication of where value is.

                                Philip_Petrescu 1 Reply Last reply Reply Quote 1
                                • lonniea
                                  lonniea last edited by

                                  David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

                                  re: David Kauzlaric

                                  We built an internal tool to do it for us, but basically you can do this manually.

                                  Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                  Cheers!

                                  1 Reply Last reply Reply Quote -1
                                  • ezclickmedia
                                    ezclickmedia @Syed1 last edited by

                                    I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

                                    1 Reply Last reply Reply Quote 0
                                    • ShaMenz
                                      ShaMenz @kdaly100 last edited by

                                      Nice solution Kieran!

                                      I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

                                      The intention is not really to get a complete list, more to "draught" out pages that need work.

                                      1 Reply Last reply Reply Quote 0
                                      • NakulGoyal
                                        NakulGoyal @kdaly100 last edited by

                                        I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

                                        1 Reply Last reply Reply Quote 0
                                        • kdaly100
                                          kdaly100 last edited by

                                          Saw this and tried the following which isn't automated but is one way of doing it.

                                          • First install SEO Quake plugin
                                          • Go to Google
                                          • Turn off Google Instant (http://www.google.com/preferences)
                                          • Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
                                          • Then run your site:www.example.com search query
                                          • Export this to CSV
                                          • Import to Excel
                                          • Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
                                          • This gives you a formatted list.
                                          • Then import your sitemap.xml into another TAB in Excel
                                          • Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

                                          Not exactly automated but does the job.

                                          NakulGoyal ShaMenz CMC-SD 3 Replies Last reply Reply Quote 15
                                          • Hakkasan
                                            Hakkasan last edited by

                                            Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

                                            1 Reply Last reply Reply Quote 0
                                            • Syed1
                                              Syed1 last edited by

                                              I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

                                              ezclickmedia 1 Reply Last reply Reply Quote 1
                                              • VisualSense
                                                VisualSense Subscriber last edited by

                                                Manual is a no go for large sites. If someone knows a tool like this, it woul be cool to know which/ where to find. Or..... This would make a cool SEOmoz pro tool 🙂

                                                1 Reply Last reply Reply Quote 4
                                                • Function5
                                                  Function5 @KeriMorgret last edited by

                                                  My bad - you are right that it doesn't display the actual URLs. So I guess the best thing you can do is site:examplesite.com and see what comes up.

                                                  1 Reply Last reply Reply Quote 0
                                                  • KeriMorgret
                                                    KeriMorgret @Function5 last edited by

                                                    That will tell you the number indexed, but it still doesn't tell you which of those URLs are or are not indexed. I think we all wish it would!

                                                    Function5 1 Reply Last reply Reply Quote 1
                                                    • Function5
                                                      Function5 last edited by

                                                      I would use Google Webmaster Tools as you can see how many URLs are indexed based on your sitemap. Once you have that, you can compare it to your total list. The same can be done with Bing.

                                                      KeriMorgret 1 Reply Last reply Reply Quote 1
                                                      • elephantseo
                                                        elephantseo @DavidKauzlaric last edited by

                                                        Yeah I do it manually now so was looking for something more efficient.

                                                        1 Reply Last reply Reply Quote 0
                                                        • DavidKauzlaric
                                                          DavidKauzlaric last edited by

                                                          We built an internal tool to do it for us, but basically you can do this manually.

                                                          Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                                          elephantseo Chris.McGiff 2 Replies Last reply Reply Quote 2
                                                          • 1 / 1
                                                          • First post
                                                            Last post

                                                          Browse Questions

                                                          Explore more categories

                                                          • Moz Tools

                                                            Chat with the community about the Moz tools.

                                                          • SEO Tactics

                                                            Discuss the SEO process with fellow marketers

                                                          • Community

                                                            Discuss industry events, jobs, and news!

                                                          • Digital Marketing

                                                            Chat about tactics outside of SEO

                                                          • Research & Trends

                                                            Dive into research and trends in the search industry.

                                                          • Support

                                                            Connect on product support and feature requests.

                                                          • See all categories

                                                          Related Questions

                                                          • Hermski

                                                            Japanese URL-structured sitemap (pages) not being indexed by Bing Webmaster Tools

                                                            Hello everyone, I am facing an issue with the sitemap submission feature in Bing Webmaster Tools for a Japanese language subdirectory domain project. Just to outline the key points: The website is based on a subdirectory URL ( example.com/ja/ ) The Japanese URLs (when pages are published in WordPress) are not being encoded. They are entered in pure Kanji. Google Webmaster Tools, for instance, has no issues reading and indexing the page's URLs in its sitemap submission area (all pages are being indexed). When it comes to Bing Webmaster Tools it's a different story, though. Basically, after the sitemap has been submitted ( example.com/ja/sitemap.xml ), it does report an error that it failed to download this part of the sitemap: "page-sitemap.xml" (basically the sitemap featuring all the sites pages). That means that no URLs have been submitted to Bing either. My apprehension is that Bing Webmaster Tools does not understand the Japanese URLs (or the Kanji for that matter). Therefore, I generally wonder what the correct way is to go on about this. When viewing the sitemap ( example.com/ja/page-sitemap.xml ) in a web browser, though, the Japanese URL's characters are already displayed as encoded. I am not sure if submitting the Kanji style URLs separately is a solution. In Bing Webmaster Tools this can only be done on the root domain level ( example.com ). However, surely there must be a way to make Bing's sitemap submission understand Japanese style sitemaps? Many thanks everyone for any advice!

                                                            Technical SEO | | Hermski
                                                            0
                                                          • flo_seo

                                                            Not all images indexed in Google

                                                            Hi all, Recently, got an unusual issue with images in Google index. We have more than 1,500 images in our sitemap, but according to Search Console only 273 of those are indexed. If I check Google image search directly, I find more images in index, but still not all of them. For example this post has 28 images and only 17 are indexed in Google image. This is happening to other posts as well. Checked all possible reasons (missing alt, image as background, file size, fetch and render in Search Console), but none of these are relevant in our case. So, everything looks fine, but not all images are in index. Any ideas on this issue? Your feedback is much appreciated, thanks

                                                            Technical SEO | | flo_seo
                                                            1
                                                          • MichaelGregory

                                                            Bulk URL Removal in Webmaster Tools

                                                            One of Wordpress sites was hacked (for about 10 hours), and Google picked up 4000+ urls in the index. The site is fixed, but I'm stuck with all those urls in the index. All the urls of of the form: walkerorthodontics.com/index.php?online-payday-cash-loan.htmloncewe The only bulk removal option I could find was to remove an entire folder, but I can't do that, as it would only leave the homepage and kill off everything else. For some crazy reason, the removal tools doesn't support wildcards, so that obvious solution is right out. So, how do it get rid of 4000 results? And no, waiting around for them to 404 out of the index isn't an option.

                                                            Technical SEO | | MichaelGregory
                                                            0
                                                          • Vacatia_SEO

                                                            Url folder structure

                                                            I work for a travel site and we have pages for properties in destinations and am trying to decide how best to organize the URLs basically we have our main domain, resort pages and we'll also have articles about each resort so the URL structure will actually get longer:
                                                            A. domain.com/main-keyword/state/city-region/resort-name
                                                            _    domain.com/family-condo-for-rent/orlando-florida/liki-tiki-village_ _   domain.com/main-keyword-in-state-city/resort-name-feature    _
                                                            _   domain.com/family-condo-for-rent/orlando-florida/liki-tiki-village/kid-friend-pool_ B. Another way to structure would be to remove the location and keyword folders and combine. Note that some of the resort names are long and spaces are being replaced dynamically with dashes. 
                                                            ex. domain.com/main-keyword-in-state-city/resort-name
                                                            _      domain.com/family-condo-for-rent-in-orlando-florida/liki-tiki-village_ _      domain.com/main-keyword-in-state-city/resort-name-feature_
                                                            _      domain.com/family-condo-for-rent-in-orlando-florida/liki-tiki-village-kid-friend-pool_ Question: is that too many folders or should i combine or break up? What would you do with this? Trying to avoid too many dashes.

                                                            Technical SEO | | Vacatia_SEO
                                                            0
                                                          • M_D_Golden_Peak

                                                            How to Remove /feed URLs from Google's Index

                                                            Hey everyone, I have an issue with RSS /feed URLs being indexed by Google for some of our Wordpress sites. Have a look at this Google query, and click to show omitted search results. You'll see we have 500+ /feed URLs indexed by Google, for our many category pages/etc. Here is one of the example URLs: http://www.howdesign.com/design-creativity/fonts-typography/letterforms/attachment/gilhelveticatrade/feed/. Based on this content/code of the XML page, it looks like Wordpress is generating these: <generator>http://wordpress.org/?v=3.5.2</generator> Any idea how to get them out of Google's index without 301 redirecting them? We need the Wordpress-generated RSS feeds to work for various uses. My first two thoughts are trying to work with our Development team to see if we can get a "noindex" meta robots tag on the pages, by they are dynamically-generated pages...so I'm not sure if that will be possible. Or, perhaps we can add a "feed" paramater to GWT "URL Parameters" section...but I don't want to limit Google from crawling these again...I figure I need Google to crawl them and see some code that says to get the pages out of their index...and THEN not crawl the pages anymore. I don't think the "Remove URL" feature in GWT will work, since that tool only removes URLs from the search results, not the actual Google index. FWIW, this site is using the Yoast plugin. We set every page type to "noindex" except for the homepage, Posts, Pages and Categories. We have other sites on Yoast that do not have any /feed URLs indexed by Google at all. Side note, the /robots.txt file was previously blocking crawling of the /feed URLs on this site, which is why you'll see that note in the Google SERPs when you click on the query link given in the first paragraph.

                                                            Technical SEO | | M_D_Golden_Peak
                                                            0
                                                          • EugeneF

                                                            Use webmaster tools "change of address" when doing rel=canonical

                                                            We are doing a "soft migration" of a website.  (Actually it is a merger of two websites). We are doing cross site rel=canonical tags instead of 301's for the first 60-90 days.  These have been done on a page by page basis for an entire site.  Google states that a "change of address" should be done in webmaster tools for a site migration with 301's.   Should this also be done when we are doing this soft move?

                                                            Technical SEO | | EugeneF
                                                            0
                                                          • upick-162391

                                                            Trailing Slashes In Url use Canonical Url or 301 Redirect?

                                                            I was thinking of using 301 redirects for trailing slahes to no trailing slashes for my urls. EG: www.url.com/page1/ 301 redirect to www.url.com/page1 Already got a redirect for non-www to www already. Just wondering in my case would it be best to continue using htacces for the trailing slash redirect or just go with Canonical URLs?

                                                            Technical SEO | | upick-162391
                                                            0
                                                          • DarwinChinaSEO

                                                            Why google index my IP URL

                                                            hi guys, a question please. if site:112.65.247.14 , you can see google index our website IP address, this could duplicate with our darwinmarketing.com content pages. i am not quite sure why google index my IP pages while index domain pages, i understand this could because of backlink, internal link and etc, but i don't see obvious issues there, also i have submit request to google team to remove ip address index, but seems no luck. Please do you have any other suggestion on this? i was trying to do change of address setting in Google Webmaster Tools, but didn't allow as it said "Restricted to root level domains only", any ideas? Thank you! boson

                                                            Technical SEO | | DarwinChinaSEO
                                                            0

                                                          Get started with Moz Pro!

                                                          Unlock the power of advanced SEO tools and data-driven insights.

                                                          Start my free trial
                                                          Products
                                                          • Moz Pro
                                                          • Moz Local
                                                          • Moz API
                                                          • Moz Data
                                                          • STAT
                                                          • Product Updates
                                                          Moz Solutions
                                                          • SMB Solutions
                                                          • Agency Solutions
                                                          • Enterprise Solutions
                                                          • Digital Marketers
                                                          Free SEO Tools
                                                          • Domain Authority Checker
                                                          • Link Explorer
                                                          • Keyword Explorer
                                                          • Competitive Research
                                                          • Brand Authority Checker
                                                          • Local Citation Checker
                                                          • MozBar Extension
                                                          • MozCast
                                                          Resources
                                                          • Blog
                                                          • SEO Learning Center
                                                          • Help Hub
                                                          • Beginner's Guide to SEO
                                                          • How-to Guides
                                                          • Moz Academy
                                                          • API Docs
                                                          About Moz
                                                          • About
                                                          • Team
                                                          • Careers
                                                          • Contact
                                                          Why Moz
                                                          • Case Studies
                                                          • Testimonials
                                                          Get Involved
                                                          • Become an Affiliate
                                                          • MozCon
                                                          • Webinars
                                                          • Practical Marketer Series
                                                          • MozPod
                                                          Connect with us

                                                          Contact the Help team

                                                          Join our newsletter
                                                          Moz logo
                                                          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                          • Accessibility
                                                          • Terms of Use
                                                          • Privacy

                                                          Looks like your connection to Moz was lost, please wait while we try to reconnect.