Duplicate Content - Bulk analysis tool?

[BeckyKey]

Hi

I wondered if there's a tool to analyse duplicate content - within your own site or on external sites, but that you can upload the URL's you want to check in bulk?

I used Copyscape a while ago, but don't remember this having a bulk feature?

Thank you!

[BeckyKey]

Great thank you!

I'll give both a go!

[BeckyKey]

Great thanks

Yes I use screaming frog for this, but it was to look at actual page content. So yes to see if sites copy our content, but also to see whether we need to update our product content as some products are very similar.

I'll check the batch process on copyscape thanks!

[CleverPhD]

I have not used this tool in this way, but have used it for other crawler projects related to content clean up and it is rock solid. They have been very responsive to me on questions related to use of the software.  http://urlprofiler.com/

Duplicate content search is the project next on my list, here is how they do it.

http://urlprofiler.com/blog/duplicate-content-checker/

You let URL profiler crawl the section of your site that is most likely to be copied (say your blog) and you tell URL profiler what section of your HTML to compare against (i.e. the content section vs the header or footer).    URL profiler then uses proxies (you have to buy the proxies) to perform Google searches on sentences from your content.  It crawls those results to see if there is a site in the Google SERPs that has sentences from your content word for word (or pretty close).

I have played with Copyscape, but my markets are too niche for it to work for me.  The logic here from URL profilers is that you are searching the database that most matters, Google.

Good luck!

[rjonesx. 0]

I believe you might be able to use List Mode in ScreamingFrog to accomplish this, however it depends on ultimately what your goal is to check for duplicate content. Do you simply want to find duplicate titles or duplicate descriptions? Or do you want to find pages with sufficiently similar text as to warrant concern?

== Ooops! ==

It didn't occur to me that you were more interested in duplicate content caused by other sites copying your content rather than duplicate content among your list of URLs.

Copyscape does have a "Batch Process" tool but it is only available to paid subscribers. It does work quite nicely though.