• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. On-Page Optimization
        4. Correct robots.txt for WordPress

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Correct robots.txt for WordPress

        On-Page Optimization
        3
        5
        800
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • AslanBarselinov
          AslanBarselinov last edited by

          Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?

          I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.

          Thank you so much.

          FzSQkqB.jpg

          1 Reply Last reply Reply Quote 1
          • jasongmcmahon
            jasongmcmahon last edited by

            Me too, can you upload or screenshot the actual file that you are using

            1 Reply Last reply Reply Quote 0
            • AslanBarselinov
              AslanBarselinov last edited by

              I have edited it down to

              User-Agent: *
              
              Allow: /wp-content/uploads/
              Disallow: /wp-content/plugins/
              Disallow: /wp-admin/
              Disallow: /contact/
              Disallow: /refer/
              
              It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
              
              1 Reply Last reply Reply Quote 1
              • jasongmcmahon
                jasongmcmahon last edited by

                Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:

                User-Agent: *
                Allow: /wp-content/uploads/
                Disallow: /wp-content/plugins/
                Disallow: /wp-admin/
                Disallow: /readme.html
                Disallow: /refer/

                Sitemap: http://www.example.com/post-sitemap.xml
                Sitemap: http://www.example.com/page-sitemap.xml

                1 Reply Last reply Reply Quote 0
                • effectdigital
                  effectdigital last edited by

                  Just seems overly complex and like there's way more in there than there needs to be

                  I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else

                  User-Agent: *

                  Disallow: /wp-content/plugins/

                  Disallow: /comments

                  Disallow: /*?s=

                  Disallow: /*&s=

                  Disallow: /search


                  See if that helps

                  1 Reply Last reply Reply Quote 1
                  • 1 / 1
                  • First post
                    Last post

                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Gavo

                    Meta title not showing up correctly on SERP

                    We have an issue with some clients on a Wordpress CMS where title tags implemented on a Yoast SEO plugin for the Homepage are not coming up as we'd implemented them. However, the source code shows we'd implemented them correctly according to what we'd wanted.**For example, this is the title tag we implemented in the CMS:Towing Services Alberta | Jack's TowingSource code shows:Towing Services Alberta | Jack's TowingHowever, SERP results shows:**Jack's Towing | Towing Services Alberta This is not an issue with the rest of the other pages and there isn't a global template for our client's sites. It's perplexing that it's only happening on the Homepage and this is across for 3 of our clients' sites.Even more perplexing, recently we've noticed that SERP is only showing the client's business name as the title tag and this is across for 2 out of the 3 clients we'd mentioned above. Nothing has changed in the back-end.Would appreciate some insight on this issue!

                    On-Page Optimization | | Gavo
                    4
                  • aj613

                    Random /feed 404 error from a wordpress site

                    My Moz Analytics report shows a 404 error on a page which I think should not exist at all.  The URL is http://henryplumbingco.com/portfolio-item/butler-elementary/feed/.  When I checked webmaster tools, it looks like there are a number of random /feed urls throwing 404 errors. I am using WordPress and the Enfold theme. Anyone know how to get rid of these errors? Thanks,

                    On-Page Optimization | | aj613
                    0
                  • Exa

                    Meta Robots index & noindex Both Implemented on Website

                    I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!

                    On-Page Optimization | | Exa
                    0
                  • Philip-DiPatrizio

                    Best way to separate blogs, media coverage, and press releases on WordPress?

                    I'm curious what some of your thoughts are on the best way to handle the separation of blog posts, from press releases stories, from media coverage.  With 1 WordPress installation, we're obviously utilizing the Posts for these types of content. It seems obvious to put press releases into a "press release" category and media coverage into a "media coverage" category.... but then what about blog posts?  We could put blog posts into a "blog" category, but I hate that.  And what about actual blog categories?  I tried making sub-categories for the blog category which seemed like it was going to work, until the breadcrumbs looked all crazy. Example: Homepage > Blog > Blog > Sub-Category Homepage = http://www.example.com First 'Blog' = http://www.example.com/blog Second 'Blog' = http://www.example.com/category/blog Sub-Category = http://www.example.com/category/blog/sub-category This just doesn't seem very clean and I feel like there has to be a better solution to this.  What about post types?  I've never really worked with them.  Is that the solution to my woes? All suggestions are welcome! EDIT: I should add that we would like the URL to contain /blog/ for blog posts /media-coverage/ for media coverage, and /press-releases/ for press releases.  For blog posts, we don't want the sub-category to be in the URL.

                    On-Page Optimization | | Philip-DiPatrizio
                    0
                  • mihaiaperghis

                    WordPress and category/subcategory landing pages

                    Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?

                    On-Page Optimization | | mihaiaperghis
                    0
                  • Jon-C

                    Wordpress & trailing slash on domain name

                    Hi recently changed my site so it is based on Wordpress, got my preferred domain set in Google webmaster as www.domian.co.uk but since moving to wordpress my domain is now having a slash put on the end of it like www.domain.co.uk/ Most of the links going to my homepage do not have the slash on the end so am I right in thinking I should get rid of the slash from my site so I have one consistent url? If so any ideas how to banish the slash? Cannot seem to do it through the Wordpress general settings (despite preferred domain being set as www.domain.co.uk !) Thanks T

                    On-Page Optimization | | Jon-C
                    0
                  • alpen

                    Moving a site from .cfm to Wordpress - How to keep the authority?

                    Hi guys, My client has a site built with Cold Fusion (web pages end in .cfm) and we're moving them over to Wordpress (for many reasons), keeping the same menu structure and navigation. Their previous SEO company was pretty awful, however, they did manage to establish some decent authority/backlinks for the website and its 20 or so pages. My questions: I assume I'll want to do 301 redirects for each page, possibly by editing the .htaccess file?  Any advice on this? Anything else I need to consider in this move? Thanks!

                    On-Page Optimization | | alpen
                    0
                  • KMack

                    Wordpress: Should I NO INDEX Categories & Archives Pages?

                    I am new to SEOmoz & trying to work my way through the ca-trillion errors that have been found on my site, but for each one I want to ensure that I am helping rather than harming my site. The tool has (as a "notice") said that my category pages & Archives are NO-INDEX, is this how these pages should be dealt with? In addition, the crawler has also (as a "warning error) discovered that my categories, and Archives do not have a meta description..is this of great importance for non indexed pages of this type? Thanks so much to the SEOmoz forum members, you have so far been of invaluable help to me.

                    On-Page Optimization | | KMack
                    2

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.