Skip to content
Technical SEO Category blog banner BG D
Technical SEO

Table of Contents

Drunk Men has put together the results of a one-year long experiment to monitor spider activity from the major three search engines. If you're watching crawling performance on big content sites, this is must read. Note their graph (in log-based scale) of relative levels of performance over time:

pageviews in time

It may come as a surprise to many in SEO that Yahoo! is the fastest and deepest crawling bot for this content. I'd argue that in the last 2-3 months, I've been seeing MSN overtake Yahoo!'s crawler in both depth and breadth, and I've also seen Google, once induced by strong external links, become the most rampant and hungry (if not speediest) bot of all.

My favorite part of their results is actually this graphical representation of how Yahoo! crawled:

Yahoo! binary tree

Like a beautiful, black hat fractal...

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.


Back to Top

With Moz Pro, you have the tools you need to get SEO right — all in one place.

Read Next

Tom Capper What is index bloat 1
Whiteboard Friday Technical SEO

What is Index Bloat? — Whiteboard Friday

Deep dive into index bloat - a critical SEO challenge affecting medium to large websites. Learn how to identify URLs consuming your index quota without delivering traffic, understand the difference between crawl budget and index bloat, and discover practical solutions for cleanup. This Whiteboard Friday video helps you assess your site's index health and implement effective remediation steps, from content consolidation to proper URL handling.

Comments

Please keep your comments TAGFEE by following the community etiquette

Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.