Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best meta description for Category Pages, Tag Pages and Main Article?
Hi, I want to index all my categories and tags. But I fear about duplicating the meta description. for example: I have a tag name "Learn Stock Market", a category name "Learning", and a main article "What is Stock Market". What is your suggestion for meta description of these three pages that looks great for seo google?
On-Page Optimization | | mbmozmb0 -
Moz bar not working on https://www.fitness-china.com/gym-equipment-names-pictures-prices
Moz bar not working on our website about gym equipment names https://www.fitness-china.com/gym-equipment-names-pictures-prices How long fix it?
On-Page Optimization | | ahislop5740 -
Impact of keyword/keyphrases density on header/footer
Hi, It might be a stupid question but I prefer to clear things out if it's not a problem: Today I've seen a website where visitors are prompted no less than 5 times per page to "call [their] consultants".
On-Page Optimization | | GhillC
This appears twice on the header, once on the side bar (mouse over pop up), once in the body of most of the pages and once in the footer. So obviously, besides the body of the pages, it appears at least 4 times on every single pages as it's part of the website template. In the past, I never really wondered re the menu, the footer etc as it's usually not hammering the same stuff repeatedly everywhere. Anyway, I then had a look at their blog and, given the average length of their articles, the keyword density around these prompts is about 0.5% to 0.8% for each page. This is huge! So basically my question is as follow: is Google's algorithm smart enough to understand what this is and make abstraction of this "content" to focus on the body of the pages (probably simply focusing on the tags)? Or does it send wrong signals and confuse search engine more than anything else? Reading stuff such as this, I wonder how does it work when this is not navigational or links elements. Thanks,
G Note: I’m purposely not speaking about the UX which is obviously impacted by such a hammering process.0 -
Will it upset Google if I aggregate product page reviews up into a product category page?
We have reviews on our product pages and we are considering averaging those reviews out and putting them on specific category pages in order for the average product ratings to be displayed in search results. Each averaged category review would be only for the products within it's category, and all reviews are from users of the site, no 3rd party reviews. For example, averaging the reviews from all of our boxes products pages, and listing that average review on the boxes category page. My question is, will this be doing anything wrong in the eyes of Google, and if so how so? -Derick
On-Page Optimization | | Deluxe0 -
H2s & H3s for Category Navigation
Hi all. I am wondering how best to format a category navigation menu. Currently I don't think we're using H2s correctly on our website. Am I right to think that the top level category e.g. Games should be formatted as an H2 and the sub-categories underneath this should be formatted as H3s (to show a hierarchy)? Is there a limit on how many H2s and H3s you should use? Obviously only one H1 per page. Thanks in advance Paul
On-Page Optimization | | kevinliao0 -
Ecommerce On-Site SEO: Keywords in Category Descriptions
Hello, I'm doing on-site SEO for a client's ecommerce site. Are 160 words enough for a category description? I'm using the keywords once at the top of the description, and once at the bottom of the description, with the ones at the bottom reworded so that they are the keywords with a different word order. I used to put the keywords in 3 times but it just feels like stuffing. Is twice, worded differently the second time, enough for a category description? Thanks.
On-Page Optimization | | BobGW0 -
Does Google index dynamically generated content/headers, etc.?
To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>, often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?
On-Page Optimization | | editabletext0 -
How much copy should there be on an e-commerce category page?
I'm not looking for a precise number, obviously. I'm more interested in a general range. More text means more long-tail and synonym opportunities, but of course you don't want too much copy above the fold, pushing your products down. Maybe you can get away with a short paragraph or two at the top of the page. You can always put more copy below the products, but in a recent SEOmoz e-commerce webinar, the presenter seemed to think that was silly and unnecessary. He even suggested that the algo might intentionally ignore text below products, since it's clearly not intended to be read. What do you think?
On-Page Optimization | | CMC-SD0