Crawl Stats Decline After Site Launch (Pages Crawled Per Day, KB Downloaded Per Day)

[BandG]

Hi all,

I have been looking into this for about a month and haven't been able to figure out what is going on with this situation. We recently did a website re-design and moved from a separate mobile site to responsive. After the launch, I immediately noticed a decline in pages crawled per day and KB downloaded per day in the crawl stats. I expected the opposite to happen as I figured Google would be crawling more pages for a while to figure out the new site. There was also an increase in time spent downloading a page. This has went back down but the pages crawled has never went back up. Some notes about the re-design:

• URLs did not change
• Mobile URLs were redirected
• Images were moved from a subdomain (images.sitename.com) to Amazon S3
• Had an immediate decline in both organic and paid traffic (roughly 20-30% for each channel)

I have not been able to find any glaring issues in search console as indexation looks good, no spike in 404s, or mobile usability issues. Just wondering if anyone has an idea or insight into what caused the drop in pages crawled? Here is the robots.txt and attaching a photo of the crawl stats.

User-agent: ShopWiki
Disallow: /

User-agent: deepcrawl
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: SLI_Systems_Indexer
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: BrightEdge Crawler/1.0 ([email protected])
Disallow: /

User-agent: *
Crawl-delay: 5
Disallow: /cart/
Disallow: /compare/
```[fSAOL0](https://ibb.co/fSAOL0)

[R0bin_L0rd]

Yea that's definitely tricky. I'm assuming you haven't taken out any load balancing that was previously in place between desktop and m. meaning your server is struggling a lot more? The Page Speed Insights tool can be good info but if possible I'd have a look at that user experience index to get an idea of how other users are experiencing the site.

A next port of call could be your server logs? Do you have any other subdomains which are performing differently in search console?

In terms of getting Google to crawl more, unfortunately at this point my instinct would be to keep trying to optimise the site to make it as crawl-friendly as possible and wait for Google to start crawling more. It does look like the original spike in time spent downloading has subsided a bit but it's still higher than it was. Without doing the maths, given that pages crawled and kilobytes downloaded have dropped, the level of slowdown may have persisted and the drop in that graph could have been caused by Google easing back. I'd keep working on making the site as efficient and consistent as possible and try to get that line tracking lower as an immediate tactic.

[BandG]

Hi Robin,

Thanks a lot for the reply. A lot of good information there.

• The crawl delay has been on the site as long as I have known so it was left in place just to minimize changes
• Have not changed any of the settings in Search Console. It has remained at "Let Google optimize for my site"
• Have not received the notification for mobile first indexing
• The redirects were one to one for the mobile site. I do not believe there are any redirect chains from those.
• The desktop pages remained roughly the same size but on a mobile device, pages are slightly heavier compared to the sepatate m dot site. The separate m dot site had a lot of content stripped out and was pretty bare to be fast. We introduced more image compression than we have ever done and also deferred image loading to make the user experience as fast as possible. The site scores in the 90s on Google's page speed insights tool.
• Yes, resizing based on viewport. Content is basically the same between devices. We have some information in accordions on product detail pages on and show fewer products on the grids on mobile.
• They are not the same images files but they are actually smaller than they were previously as we were not compressing them and using different sizes in different locations to minimize page weight.

I definitely lean towards it being performance related as in the crawl stats there seems to be a correlation between time spent downloading a page and the other two stats. I just wonder how you get Google to start crawling more once the performance is fixed or if they will figure it out.

[R0bin_L0rd]

Hi there, thanks for posting!

Sounds like an interesting one, some questions that come to mind which I'd just like to run through to make sure we're not missing anything;

• Why do you have Crawl-delay set for all user agents? Officially it's not something Google supports but the reason for that could be the cause of this
• Have you changed any settings in search console? There is a slider for how often you want Google to crawl a site
• Have you had the Search Console notification that you're now on the mobile-first index?
• When you redirected the mobile site, was it all one-to-one redirects? Is there any possibility you've introduced redirect chains?
• After the redesign - are the pages now significantly bigger (in terms of amount of data needed to fully load the page)? Are there any very large assets that are now on every page?
• When you say responsive, is it resizing based on viewport? How much duplication has been added to the page? Is there a bunch of content that is there for mobile but not loaded unless viewed from mobile (and vice versa)?
• When you moved the images, were they the same exact image files or might they now be the full-size image files?

This is just first blush so I could be off the mark but those graphs suggest to me that Google is having to work harder to crawl your pages and, as a result, is throttling the amount of time spent on your site. If the redesign or switch to responsive involved making the pages significantly "heavier" where that could be additional JavaScript, bigger images, more content etc. that could cause that effect. If you've got any sitespeed benchmarking in place you could have a look at that to see whether things have changed. Google also uses pagespeed as a ranking factor so that could explain the traffic drop.

The other thing to bear in mind is that combining the mobile and desktop sites was essentially a migration, particularly if you were on the mobile-first index. It may be that the traffic dip is less related to the crawl rate, but I understand why we'd make the connection there.