Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to Diagnose "Crawled - Currently Not Indexed" in Google Search Console
-
The new Google Search Console gives a ton of information about which pages were excluded and why, but one that I'm struggling with is "crawled - currently not indexed". I have some clients that have fallen into this pit and I've identified one reason why it's occurring on some of them - they have multiple websites covering the same information (local businesses) - but others I'm completely flummoxed.
Does anyone have any experience figuring this one out?
-
@intellect did you find a solution to that?
-
-
@dalerio-consulting what should can we do with excluded section then. let say this page of my website is under duplicate canonical tag in excluded section. then should i leave it if its not very serious or should i request indexing ? Are these excluded pages issues very serious to take?
-
Hey Brett!
Basically what we believe this status means is Google saying "I can crawl and access the URL but I don't believe this page belongs in the index". They key here is to figure out why Google might not believe the page should be considered for indexation. We analyzed a good number of Index Coverage reports across all of our different clients.
Here are the most commons reasons URLs get reported as "Crawled - Currently Not Indexed":
- False positives
- RSS Feed URLs
- Paginated URLs
- Expired products
- 301 redirects
- Thin content
- Duplicate content
- Private-facing content
You can find a breakdown of each reason on the post we wrote here: https://a-moz.groupbuyseo.org/blog/crawled-currently-not-indexed-coverage-status
However, there's likely many more reasons why Google does't think the page is eligible for indexation.
-
Crawled - Currently not indexed is the most common way for pages or posts on your site not to be indexed. It is also the most difficult one to pinpoint because it happens for a multitude of reasons.
Google needs computing power to analyze each website. How it works is that Google assigns a certain crawl budget to each site, and that crawl budget determines how many pages of your site will be indexed. Google will always index your top pages, therefore, the excluded pages are of less quality rank-wise.
Every website has pages that are not indexed, and the healthy ratio of non-indexed pages will depend on the niche of the website.
There are however 2 ways for you to get your pages out of the "Crawled - Currently not indexed" pit:
- Decrease the number of pages/posts. It's a matter of quality v quantity, so make sure that put more attention into linking every new post so that they get indexed in no time. Don't forget to utilize robots.txt to block pages that aren't useful to the site from indexing so that the crawl budget can be assigned to the other posts.
- Increase the crawl budget. You can do that by raising the quality of the pages/posts. Make more internal and external backlinks for your posts and homepage, make sure that the articles are unique and keyword-optimized, and work hard to aim so that each article will rank on that first page.
SEO is a tough business, but if managed carefully, over time it will pay off.
Daniel Rika - Dalerio Consulting
https://dalerioconsulting.com
[email protected] -
Crawled - currently not indexed list includes sitemap and robots.txt
We have searched and try to understand this issue. But we did not get final result regarding this issue
If any one fixed this issues, please share your suggestions as soon as possible
-
Hi There,
Google has been struggling to eliminate spam pages, content and structurally ordering them; this is an inherent problem especially with badly structured e-commerce websites.
You might be aware that "Crawled - Currently Not Indexed" means that your page(s) has been found by Google but it is not currently indexed, this might not be an error, just that your pages are in a queue. That might be due to the following reasons:
- There are a lot of pages to index, so it's going to take Google some time to get through them and mark them as either indexed or not.
- There might be duplicate pages / canonical issues for the website of the pages. Google might be seeing a lot of duplicate pages without canonical tags on your site, to improve the number of pages indexed you need to either improve pages so they are no longer duplicated or add canonical tags to help Google attribute it to the correct page
You need to justify each and every page for their merits, and then let google decide whether it think it should be available in their search and also against what keywords at what rank. To summarise, just help 'Google search' by structuring your data right, it might reward you by ranking your pages at right places for the right keywords.Thanks and Regards,Vijay
-
Search Console > Status > Index Coverage > Crawled - currently not indexed
Yes, I had the same Issues last month, in my case the crawler took it 6 weeks to update the Index Coverage. And apparently, there are not too many things that you can do it about it.
Regards
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
Hello, I'm hoping one of you search geniuses can help me. We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account. The way I see it, there are a couple of possibilities.
Reporting & Analytics | | EricFish
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm. 2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site. The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site. If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you. Thanks for your help. Eric0 -
Referral Traffic from Google
Hello, I have a question about my company's new website. I've worked in SEO and studied Google Analytics results for a few years now but have never really come across something like this. I started in this position in January of this year and when I started breaking down the traffic sources in Google Analytics, I noticed most of the traffic was coming from Google.com as a referral source. I had never seen Google.com as a referral source before so I looked into options for what it could be. It was not a paid ad and our organic traffic was coming through in Analytics, Before I could get any further, our new website was launched (we switched CRM's to WordPress) and the referral traffic from google went from 2,966 in January of 2015 to 22 in February 2015. for more comparison, in February of 2014, the referral traffic from Google was 2,496. I expected a drop when we switched CRM's but we correctly re-directed all pages and created a new sitemap and our organic traffic is up since the switch (not enough to cover drop in referral). I thought at first this had to do with our Google sellers account being de-activated when we made the switch, but I quickly fixed this over a month ago and no change. I'm wondering if anyone has ever seen Google.com come through as a referral source in Google Analytics and if they we're able to figure out what it actually was. This would be a great help! Thank you, Alex
Reporting & Analytics | | RASEO1 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
Google Analytics - Keywords (not set) or ( not provided) WHY???
In Analytics, most of my visitors are landing on my home page, but when trying to see wich keywords they use, most of them are: (not set) or ( not provided) See screencast: http://screencast.com/t/AKwPW76qLVsN Can you tell why? What is going on? Is there a way to solve this? Thank you, BigBlaze
Reporting & Analytics | | BigBlaze2050 -
Google Analytics and DNS change
Our new alumni application is going be tested at domain uva.imodules.com . We are going to collect traffic data with a Google analytics account number UA-884652-XX. So going to uva.imodules.com/myPage.html would send its data to Google Analytics with that account number. Then when it is ready for production we are going to just change the domain name of the application and switch the DNS over to dardencommunity.darden.virginia.edu . So going to dardencommunity.darden.virginia.edu /myPage.html would send its data to Google Analtics with that SAME account number. Aside from having the testing domain data in the same profile are there any other issues/problems we may run into?
Reporting & Analytics | | Darden0 -
Localhost:4444 Showing Up in Google Analytics
Hello All, Lately in my Google Analytics account I have noticed a referral source labelled: localhost:4444 The number of visits is really high from this source, but I have no idea (no clue!) what it actually means. Can anyone shed some light on what this is about? Should I be creating some sort of filter to screen out this as a referral source (assuming it is not legitimate)? Many thanks in advance. Cheers!
Reporting & Analytics | | Robert-B0 -
No Social Sources in Google Analytics - what am I doing wrong?
Hello Everyone, I'm having a strange issue: I DO NOT have in my Google Analytics the "Social" tab under the Traffic Sources category. Look at the first image of this post: http://marketingland.com/google-analytics-social-reports-8138 How do you "get" that to show? Hope somebody has this issue and can help, Thanks a lot, Alex
Reporting & Analytics | | pwpaneuro0