Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I replace categories with a static page
Hello there. I want to replace all of WordPress categories with static pages so that users see a well designed and constructed presentation of all the articles within each topic instead of just a long list of excerpts. I've already done this with 2 categories and although it is hard work I can't help feeling it is a much better thing for my users. However, I'm concerned that I am embarking on this project without being totally sure that it makes sense from an Seo point of view, or whether there are any downsides I haven't thought of? My idea is that the WordPress categories are set to noindex and nofollow. Search engines should find all of my static category pages and all of the content within each category will be spidered from there instead. Just to be sure you know what I mean here is a link to a normal category - https://www.whitegoodshelp.co.uk/category/consumer/ and here is my static page replacement for it - https://www.whitegoodshelp.co.uk/consumer-rights-appliances/ Both pages contain links to all articles within the category except the one generated by WordPress is just a long paginated list, and my replacement is a proper category page, which is hopefully far more useful . Can someone please confirm that there are no downsides to this strategy? 🙂
On-Page Optimization | | Snowdune1 -
Home page and category page target same keyword
Hi there, Several of our websites have a common problem - our main target keyword for the homepage is also the name of a product category we have within the website. There are seemingly two solutions to this problem, both of which not ideal: Do not target the keyword with the homepage. However, the homepage has the most authority and is our best shot at getting ranked for the main keyword. Reword and "de-optimise" the category page, so it doesn't target the keyword. This doesn't work well from UX point of view as the category needs to describe what it is and enable visitors to navigate to it. Anybody else gone through a similar conundrum? How did you end up going about it? Thanks Julian
On-Page Optimization | | tprg0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
WordPress and category/subcategory landing pages
Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?
On-Page Optimization | | mihaiaperghis0 -
Does show/hide element with javascript impact SEO
Hi I am developing an ecommerce site and want to place text on all category and home page. The challenge is that 300 words of text for the pages does not fit into the design appropriately especially on the home page. If I were to use a show/hide element with javascript would this be seen as spam or a trick to the search engines. I do not think it is spam as it will be actual content for the site and the visitor can view it if they click on the show button. Would love to hear your thoughts?
On-Page Optimization | | VivaArturo0 -
How to avoid keyword stuffing on e-Commerce Category pages
Hi, I'm optimizing a large, consumer electronic e-commerce superstore. Based on client's choice of keywords, I'm using product category pages as my target urls. Because of the proprietary CMS structure, product names and titles, featured on my landing pages (product category pages) create a keyword overkill, affecting various ranking factors. For example, one of the target urls / landing pages, dedicated to a specific product category, mentions the keyword over 190 times because of so many product titles in the "body" section. Would inline "rel="canonical" help? If yes, what part of the website should it "canonize"? If rel="canonical" is not the answer, what strategies would you suggest? Thanks!
On-Page Optimization | | dimanyc0 -
Impact of removing category sidebar with keywords?
Our site (a niche financial publication: insideARM.com) requires some more room in the sidebar. We're considering removing the categories (we call them topics) sidebar block, or cutting down the number of items displayed within it. My concern is that we'd be removing a direct link to landing pages for important keyword terms from our most powerful page (the index). Sure, we have the terms listed in the footer, but I am worried that the position change will lower the value of the links. Our users don't really use these links for navigational purposes, which is why it comes up as a potential removed item. Am I wrong to worry about this? Would we be crippling our category pages by doing this?
On-Page Optimization | | insideARM0 -
Category Pages with Sub-Categories
The image will explain it all... Each category page starts on the subject of the first sub-category page. This happens twice (well actually 3 times since this section of the site is called showroom and it starts on the tab mowers). Is this a terrible approach? If so, how could a site like this be better navigation-ally organized. cat-subcat.png
On-Page Optimization | | drewschmaltz0