Skip to content
Search engines 5511dd3

An Exemplary Experiment in Spider Behavior

Rand Fishkin

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Table of Contents

Rand Fishkin

An Exemplary Experiment in Spider Behavior

The author's views are entirely their own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Drunk Men has put together the results of a one-year long experiment to monitor spider activity from the major three search engines. If you're watching crawling performance on big content sites, this is must read. Note their graph (in log-based scale) of relative levels of performance over time:

pageviews in time

It may come as a surprise to many in SEO that Yahoo! is the fastest and deepest crawling bot for this content. I'd argue that in the last 2-3 months, I've been seeing MSN overtake Yahoo!'s crawler in both depth and breadth, and I've also seen Google, once induced by strong external links, become the most rampant and hungry (if not speediest) bot of all.

My favorite part of their results is actually this graphical representation of how Yahoo! crawled:

Yahoo! binary tree

Like a beautiful, black hat fractal...

Back to Top

With Moz Pro, you have the tools you need to get SEO right — all in one place.

Read Next

How to Communicate Site Migration to Clients — Whiteboard Friday

How to Communicate Site Migration to Clients — Whiteboard Friday

Jun 06, 2025
18 Real-Life Analogies to Explain SEO to Anyone

18 Real-Life Analogies to Explain SEO to Anyone

Jun 05, 2025
3 Unexpected Examples of Technical SEO Issues (And How to Resolve Them) — Whiteboard Friday

3 Unexpected Examples of Technical SEO Issues (And How to Resolve Them) — Whiteboard Friday

Mar 14, 2025

Comments

Please keep your comments TAGFEE by following the community etiquette

Comments are closed. Got a burning question? Head to our Q&A section to start a new conversation.