• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. Moz Tools
        3. Moz Bar
        4. What is Considered Duplicate Content by Crawlers?

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        What is Considered Duplicate Content by Crawlers?

        Moz Bar
        3
        5
        938
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • RyanRhodes
          RyanRhodes last edited by

          I am asking this because I have a couple of site audit tools that I use to crawl a site I work on every week and they are showing duplicate content issues (which I know there is a lot on this site) but some of what is flagged as duplicate content makes no sense.

          For example, the following URL's were grouped together as duplicate content:

          |

          https://www.firefold.com/contact-us

          |

          https://www.firefold.com/gabe

          | https://www.firefold.com/sale |

          |

          |

          How are these pages duplicate content? I am confused on what site audit tools are considering duplicate content.

          Just FYI, this is data from Moz crawl diagnostics but SEMrush site auditor is giving me the same type of data.

          Any help would be greatly appreciated.

          Ryan

          1 Reply Last reply Reply Quote 0
          • RyanRhodes
            RyanRhodes last edited by

            Yea I just started working on this site. I haven't used Moz Analytics much so just wanting to see how their crawler crawls pages.

            And yes I agree, there are a lot of BIG BIG BIG issues with this site.

            I got a large workload over the next few months haha.

            1 Reply Last reply Reply Quote 0
            • Lumina
              Lumina @DavidLee last edited by

              I would add that there's is no text on any of those three pages - any "text" one would see there is actually just embedded in an image - which is a huge issue for a number of reasons:

              1. Search engines see that there's no text - a big no-no.
              2. You're getting practically no SEO value from the content that would be there, even if there isn't much.
              3. It's heavier this way - which makes load times slower.

              I want to clarify that there are many, bigger issues with these pages - but as your question concerns only duplicate content, I'll leave all of that out for the time being. To summarize, Google, Yahoo, and Bing are just seeing some duplicate banners, sidebars, etc. and then some images in the body of your pages. Hence, duplicate content.

              1 Reply Last reply Reply Quote 2
              • RyanRhodes
                RyanRhodes @DavidLee last edited by

                Thanks for that information.

                It makes sense looking at the data and pages from that perspective.

                1 Reply Last reply Reply Quote 0
                • DavidLee
                  DavidLee last edited by

                  Hi Ryan!

                  Our crawler will flag pages that have at least 90% similarity in the entire source code of the site so not just the body.

                  The way you want to interpret the report is the contact-us page has 35 duplicates, so "gabe" and "sale" are not dupes of each other in this section but are only each a duplicate of "contact-us". Those URLs might appear with their own duplicates of the same pages further down in the report.

                  While on the front end the pages do not appear to be similar. The issue is likely with the amount of javascript code on those pages.

                  Our crawler cannot read javascript so we are likely only able to see the template of the page. Other search tools are probably seeing the same thing as it returns 79% similarity  using this tool: http://www.freebulkseotools.com/similar-page-checker-tool.php

                  I can't provide much insight from a dev perspective but hope this helps!

                  RyanRhodes Lumina 2 Replies Last reply Reply Quote 2
                  • 1 / 1
                  • First post
                    Last post

                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Josh_Eligon

                    Unsolved MozBar 2024 How it Works Now

                    mozbar 2024

                    I am a premium user and i am still not able to use mozbar
                    ec22b5ba-4430-4fc7-9825-7fce7b1ca300-image.png I don't know how its works.

                    Moz Bar | | Josh_Eligon
                    0
                  • ravigaurav

                    Unsolved How Should I Increase DA of my Website?

                    increase da

                    Google itself has been troubled by ranking factors. On top of that, the https://a-moz.groupbuyseo.org/ has worn a new garland of domain authority. It is even more regrettable when a website DA suddenly decreased. As recently on 22nd September, my website's DA was 16. As usual, on September 23, I posted a new informational blog related to QuickBooks antivirus exclusions on my website. The very next day my website DA was decreased. Can anyone who knows SEO help me understand why this always happens specially to me every time? I read the whole forum and all I found out is how my site's DA is decreased.
                    Can anyone let us know whether we can increase the Domain Authority of our website only by creating backlinks or there can be some other activities also?

                    Moz Bar | | ravigaurav
                    2
                  • jbcorcoran

                    How do I see amount of traffic to a particular page on a website?

                    I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?

                    Moz Bar | | jbcorcoran
                    6
                  • Spam score of 28 %

                    Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,

                    Moz Bar | | seoanalytics
                    1
                  • donnieath

                    Www and non www / duplicate content / redirects / www resolve issue

                    I am not getting docked for these specific errors, but I am getting docked for 1 page has a WWW resolve issue and 1 wrong URL in the sitemap... (SEM Rush) but when I use moz, it's not showing any issues. So I have these things set up so far: In .htaccess i have a command that removes the www. 301 redirect from www version to the non www (homepage) canonical on index.html pointing to non www version, I also set up a canonical tag for each page on the site search console with non www, www, https www, https non www all set to non www preference. Also, when I fetch the www version in google search console it says it's being 301 redirected to non www version which is basically what I want.Is there anything that i'm missing? These errors on SEM Rush are giving me anxiety lol.

                    Moz Bar | | donnieath
                    1
                  • alisacromer

                    Need to solve "Oops our crawlers were unable to access" url for new campaign

                    I'm putting the url designfirstkitchenandbath.com and getting the "oops! our crawlers were unable to access the site. Since this site is a potential client, which shows up online, I can't get access to fix the code, plus while I can write a little html I don't feel comfortable working with hard, live code on someonelse's site. Anyone have a simple solution?

                    Moz Bar | | alisacromer
                    0
                  • Essential-Pest

                    Duplicate Page and Title Issues

                    On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?

                    Moz Bar | | Essential-Pest
                    0
                  • Hardley111

                    Duplicate page issue

                    Hi Guys We were recently having trouble with an excessive amount of duplicate page titles. So I asked our web company, at a reasonable expense, to fix the issue which they did. How ever I since note that the issue has returned (please see the attached graph. Could anyone explain to me why this might have happened? I t would be great to have some insight before i go back to them. Thanks again for your help Regards Pete Capture.jpg

                    Moz Bar | | Hardley111
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.