Investigating a huge spike in indexed pages

[farbeseo]

I've noticed an enormous spike in pages indexed through WMT in the last week. Now I know WMT can be a bit (OK, a lot) off base in its reporting but this was pretty hard to explain. See, we're in the middle of a huge campaign against dupe content and we've put a number of measures in place to fight it. For example:

1. Implemented a strong canonicalization effort

2. NOINDEX'd content we know to be duplicate programatically

3. Are currently fixing true duplicate content issues through rewriting titles, desc etc.

So I was pretty surprised to see the blow-up. Any ideas as to what else might cause such a counter intuitive trend? Has anyone else see Google do something that suddenly gloms onto a bunch of phantom pages?

[farbeseo]

I haven't contacted the forum yet but that's my next step.

Pages indexed: 91k

Blocked by robots.txt: 8.4million

I don't even know how you could create 8.4 million indexable pages from our content.

[FedeEinhorn]

Have you contacted the Google Webmaster Help forums? As that seems to be a glitch in Google.

How many pages are scraped by Mozbot? If the amount that mozbot shows is different, then you should either sit and wait until Google removes those indexed pages or create a conversation on the forums so someone at google can give you a hint of what is going on.

[farbeseo]

Any help out there? Since the original question was posted, I've seen some improvement but even with aggressive canonicalization and noindexing, I'm still seeing a boatload of indexed pages. I am still seeing pages indexed that I've asked explicitly to be omitted by robots.txt (/search.aspx and */filter). I'm guessing it's just going to take a while to deindex what's there. Still, 91k pages indexed is quite a lot when you consider we only have about 3-4k pages and some articles.

Is anyone aware of any significant releases by Google?

[farbeseo]

Quite recent. We were actually seeing a nice downward trend in the huge number of pages indexed and then the number tripled. Crazy is an understatement. I would have thought the number of pages would fall given the number of pages that now use canonicals.

[FedeEinhorn]

How long have you waited since you applied all the rules to avoid duplicate content, as if it was just recently, then Google should be "rebuilding" the index of your site and stats may be a little crazy while that is happening.

If it was over 2 month ago and you are seeing the increase now, then I'd suggest you revise the rules you created to see if your own Website isn't creating all those new pages.

Hope that helps.