• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. Technical SEO
        4. Removing robots.txt on WordPress site problem

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Removing robots.txt on WordPress site problem

        Technical SEO
        7
        21
        41241
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • Wallander
          Wallander last edited by

          Hi..am a little confused since I ticked the box in WordPress to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap.

          Checked source code and the robots instruction has gone so a little lost. Any ideas please?

          1 Reply Last reply Reply Quote 0
          • nextlevelweb
            nextlevelweb last edited by

            Hi,

            I edited the robots.txt file for my website http://debtfreefrombankruptcy.com yesterday to allow search engines to crawl my site. However, Google isn't recognizing the new file and is still saying that my sitemap is blocked from search. Here is a link to the file itself:

            http://www.debtfreefrombankruptcy.com/robots.txt

            The Blocked URLs tester said that the file allows Google to crawl the site, but in actuality it still isn't recognizing the new file. Any advice would be appreciated. Thanks!

            1 Reply Last reply Reply Quote 0
            • CMcCrone
              CMcCrone last edited by

              I can help you out as this issue DROVE ME NUTS.

              1. I didnt have a Robots.txt (yet)

              2. I had Yoast installed

              3. Im pretty sure it created a Robots.txt even though it doesnt exist in my root (.com/here)

              4. My Google webmaster tools shows this

              User-agent: Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category//* Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /? Disallow: /?Allow: /wp-content/uploadsAllow: /assets Create a Robots.txt

              1. login to wordpress 2. Click SEO in your side toolbar (Yoast WordPress Plugin settings) 3. Go to edit files under SEO (in the side toolbar)

              And now you have the option to edit your Robots.txt file.

              1 Reply Last reply Reply Quote 0
              • evolvingSEO
                evolvingSEO @Wallander last edited by

                Hi Sophia

                I just checked and see your homepage indexed in google.co.uk with a cache date of April 26th. You should be all set!

                -Dan

                1 Reply Last reply Reply Quote 0
                • Wallander
                  Wallander last edited by

                  Quick update - by amending the robots text file and switching sitemap plugin over to Yoast I finally got the sitemap to index without robots.txt warnings although the Home page of site was not indexed, 'oh dear'.  5 out of the 7 pages in the sitemap were indexed by Google so It's a start but some more investigating to be done on my side.

                  evolvingSEO 1 Reply Last reply Reply Quote 0
                  • Wallander
                    Wallander @Wallander last edited by

                    Dan,

                    Cant thank you enough! The sitemap request is still pending in Google - maybe I sent too many requests But it's time to sit back and wait for the good news hopefully. Thanks again.

                    1 Reply Last reply Reply Quote 0
                    • evolvingSEO
                      evolvingSEO @Wallander last edited by

                      Hi Sofia

                      I just ran the same validator on your sitemap and it went through fine - see screenshot

                      I intended to mean that you should just be sure Google Webmaster Tools accepts the sitemap as valid - if so, there's no need to run through a 3rd party validator. Apologies if I didn't state it clearly!

                      Let me know, but from what I can see it looks good!

                      -Dan

                      EDIT - Looking more closely, it looks like your ran the homepage through the validator - you would actually enter the sitemap address its self in the validator - http://containerforsale.co.uk/sitemap.xml

                      1 Reply Last reply Reply Quote 0
                      • Wallander
                        Wallander @Wallander last edited by

                        Hi Dan,

                        I followed the above advice and switched to the Yoast generated sitemap but after testing on http://www.xml-sitemaps.com/validate-xml-sitemap.html I got the following result - no idea what it means but it looks nasty...

                        Schema validating with XSV 3.1-1 of 2007/12/11 16:20:05Schema validator crashed

                        The maintainers of XSV will be notified, you don't need to
                        send mail about this unless you have extra information to provide.
                        If there are Schema errors reported below, try correcting
                        them and re-running the validation.Target: http://containerforsale.co.uk
                           (Real name: http://containerforsale.co.uk
                            Server: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4)The target was not assessedLow-level XML well-formedness and/or validity processing output
                        Warning: Undefined entity raquo
                         in unnamed entity at line 16 char 83 of http://containerforsale.co.uk
                        Warning: Undefined entity nbsp
                         in unnamed entity at line 160 char 10 of http://containerforsale.co.uk
                        Error: Expected ; after entity name, but got =
                         in unnamed entity at line 274 char 631 of http://containerforsale.co.u

                        1 Reply Last reply Reply Quote 0
                        • evolvingSEO
                          evolvingSEO @Wallander last edited by

                          Sofia

                          You are using Yoast SEO plugin for WordPress, so use the XML sitemap within Yoast. You don't need a separate plugin for the XML sitemap. And yes, within Yoast turn the sitemap on.

                          Hope that helps!

                          -Dan

                          1 Reply Last reply Reply Quote 0
                          • Wallander
                            Wallander @evolvingSEO last edited by

                            Indeed, thanks everyone - it's really appreciated!

                            I have updated the robots.txt as indicated and re submitted site map but looks like Google still has problems with my site since the error warning for robots is there after the processing is done.

                            Quick question - I am using a plugin called Google XML Sitemaps which has the following tick box option.

                            'Add sitemap URL to the virtual robots.txt file'.
                            The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!'

                            Should this box be ticked or un-ticked please? Fyi I currently don't have the box ticked.

                            evolvingSEO Wallander 4 Replies Last reply Reply Quote 0
                            • evolvingSEO
                              evolvingSEO last edited by

                              Thanks guys for all the responses and helping!

                              Three Things to try

                              1.Fix Robots.txt

                              Sofia - I just checked your robots.txt now and it reads;

                              User-agent: *
                              
                              Disallow: Sitemap: http://containerforsale.co.uk/sitemap.xml.gz
                              
                              • with the sitemap on the same line as disallow - I'd check on that and make sure its on a separate line.
                              • ALSO, you don't need the .gz on the sitemap file just sitemap.xml

                              2. Re-submit Sitemap

                              • RESUBMIT your sitemap to webmaster tools and make sure its valid.

                              3. Submit URL to Webmaster Tools (only last resort)

                              this is only last case scenario, shouldn't have to do this on the homepage if everything is correct.

                              • go to fetch as googlebot ->run the fetch ->then submit URL
                              • do this for the homepage
                              • see article on google blog for reference

                              Let us know if you're all set, thanks!

                              -Dan

                              Wallander 1 Reply Last reply Reply Quote 0
                              • Wallander
                                Wallander @Copstead last edited by

                                Ok thanks Brent, I changed to

                                User-agent: *

                                Disallow:

                                Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                Guess I will just have to wait for Google to refresh now...

                                1 Reply Last reply Reply Quote 0
                                • Copstead
                                  Copstead @Copstead last edited by

                                  yes, the urls being blocked are includes from your Wordpress program.

                                  1 Reply Last reply Reply Quote 0
                                  • Wallander
                                    Wallander @Copstead last edited by

                                    Thanks for the heads up.

                                    The warning just says 7 Url''s blocked by robots.txt. - have seen this issue posted on the WordPress boards by others but no real insight into solutions.

                                    Perhaps I should try your idea of

                                    Change the robots.txt file to this:

                                    User-agent *

                                    Disallow:

                                    Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                    1 Reply Last reply Reply Quote 0
                                    • Copstead
                                      Copstead @Copstead last edited by

                                      Well there is a robots.txt file.  You can view it here: http://containerforsale.co.uk/robots.txt

                                      What warnings are you getting in your sitemap submission area?  It appears to look alright: http://containerforsale.co.uk/sitemap.xml  But I tried to validate it and got a 504 Gateway Time-out error. http://www.xml-sitemaps.com/index.php?op=validate-xml-sitemap&go=1&sitemapurl=http%3A%2F%2Fcontainerforsale.co.uk%2Fsitemap.xml&submit=Validate

                                      1 Reply Last reply Reply Quote 0
                                      • Wallander
                                        Wallander @Copstead last edited by

                                        Its weird, the front page warning on Google webmaster for robots has disappeared now, but still got the warnings in the sitemap submission area. My host suggests I just wait a bit longer for Google to update because he said same as you - that there doesn't seem to be any robot.txt file.

                                        1 Reply Last reply Reply Quote 0
                                        • Copstead
                                          Copstead @Wallander last edited by

                                          Doesn't appear to be blocked, so maybe it has something to do with your /wp-includes/ directory.

                                          Change the robots.txt file to this:

                                          User-agent *

                                          Disallow:

                                          Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                          Wallander Copstead 5 Replies Last reply Reply Quote 0
                                          • Wallander
                                            Wallander last edited by

                                            Hey Guys,

                                            Thanks for your replies...the domain is http://containerforsale.co.uk ,My host told me to look in the Public HTML file folder for the robots.txt file and just delete it but can't see it in there?

                                            My host said he found a tester site and it doesn't report any issues:

                                            http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php

                                            This is the display I get from http://containerforsale.co.uk/robots.txt

                                            User-agent *

                                            Disallow: /wp-admin/
                                            Disallow: /wp-includes/

                                            Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                            Copstead 1 Reply Last reply Reply Quote 0
                                            • Anthony_NorthSEO
                                              Anthony_NorthSEO last edited by

                                              Hi Sofia,

                                              Two things you need to consider when troubleshooting this:

                                              The actual robots.txt file (located in the root directory of your site) and the meta-robots tags in the section of your HTML. When you say you checked the source code and the robots instructions were missing, I think you were talking about the meta-robots tags in the actual HTML of your site.

                                              Webmaster Tools is probably referring to the actual robots.txt file in your domain's root path, which would differ entirely and not be visible by checking the HTML on your site. Like Nakul and Brent said, if you'll let us know your site's URL and paste the content of your robots.txt file here, I'm sure one of us can help you resolve the problem fairly quickly.

                                              Thanks!

                                              Anthony

                                              1 Reply Last reply Reply Quote 0
                                              • Copstead
                                                Copstead last edited by

                                                copy whatever you have in your robots.txt file here and we will tell you the issue.

                                                SEOmoz has a great article about Robots.txt files here: http://www.seomoz.org/learn-seo/robotstxt

                                                1 Reply Last reply Reply Quote 1
                                                • NakulGoyal
                                                  NakulGoyal last edited by

                                                  The robots.txt would probably not be a part of the Wordpress Configuration. Allow indexing is controlled via Meta Data by the Wordpress Architecture.

                                                  I would look for something like this in yourdomain.com/robots.txt

                                                  disallow /

                                                  or something like that. If that does not help, PM me your site URL and I would be glad to look it up for you.

                                                  1 Reply Last reply Reply Quote 0
                                                  • 1 / 1
                                                  • First post
                                                    Last post

                                                  Browse Questions

                                                  Explore more categories

                                                  • Moz Tools

                                                    Chat with the community about the Moz tools.

                                                  • SEO Tactics

                                                    Discuss the SEO process with fellow marketers

                                                  • Community

                                                    Discuss industry events, jobs, and news!

                                                  • Digital Marketing

                                                    Chat about tactics outside of SEO

                                                  • Research & Trends

                                                    Dive into research and trends in the search industry.

                                                  • Support

                                                    Connect on product support and feature requests.

                                                  • See all categories

                                                  Related Questions

                                                  • RoxBrock

                                                    Robots.txt & meta noindex--site still shows up on Google Search

                                                    I have set up my robots.txt like this: User-agent: *
                                                    Disallow: / and I have this meta tag in my on a Wordpress site, set up with SEO Yoast name="robots" content="noindex,follow"/> I did "Fetch as Google" on my Google Search Console My website is still showing up in the search results and it says this: "A description for this result is not available because of this site's robots.txt" This site has not shown up for years and now it is ranking above my site that I want to rank for this keyword. How do I get Google to ignore this site? This seems really weird and I'm confused how a site with little content, that has not been updated for years can rank higher than a site that is constantly updated and improved.

                                                    Technical SEO | | RoxBrock
                                                    1
                                                  • Webicultors

                                                    Is sitemap required on my robots.txt?

                                                    Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad

                                                    Technical SEO | | Webicultors
                                                    0
                                                  • Silviu

                                                    Removing a large number of unnecessary pages from a site

                                                    Hi all, I got a big problem with my website. I have a lot of page, duplicate page made from various combinations of selects, and for all this duplicate content we've be hit by a panda update 2 years ago. I don't want to bring new content an all of these pages, about 3.000.000, because most of them are unnecessary. Google indexed all of them (3.000.000), and I want to redirect the pages that I don't need anymore to the most important ones. My question, is there any problem in how google will see this change, because after this it will remain only 5000-6000 relevant pages?

                                                    Technical SEO | | Silviu
                                                    0
                                                  • allstatetransmission

                                                    Robots.txt and Multiple Sitemaps

                                                    Hello, I have a hopefully simple question but I wanted to ask to get a "second opinion" on what to do in this situation. I am working on a clients robots.txt and we have multiple sitemaps. Using yoast I have my sitemap_index.xml and I also have a sitemap-image.xml I do put them in google and bing by hand but wanted to have it added into the robots.txt for insurance. So my question is, when having multiple sitemaps called out on a robots.txt file does it matter if one is before the other? From my reading it looks like you can have multiple sitemaps called out, but I wasn't sure the best practice when writing it up in the file. Example: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://sitename.com/sitemap_index.xml Sitemap: http://sitename.com/sitemap-image.xml Thanks a ton for the feedback, I really appreciate it! :) J

                                                    Technical SEO | | allstatetransmission
                                                    0
                                                  • allstatetransmission

                                                    Removing images from site and Image Sitemap SEO advice

                                                    Hello again, I have received an update request where they want me to remove images from this site (as of now its a bunch of thumbnails) current page design: http://1stimpressions.com/portfolio/car-wraps/ and turn it into a new design which utilized a slider (such as this): http://1stimpressions.com/portfolio/ They don't want the thumbnails on the page anymore. My question is since my site has a image sitemap that has been indexed will removing all the images hurt my SEO greatly? What would the recommended steps to take to reduce any SEO damage be, if so? Thank you again for your help, always great and very helpful feedback! 🙂 cheers!

                                                    Technical SEO | | allstatetransmission
                                                    0
                                                  • JohnHuynh

                                                    How to remove all sandbox test site link indexed by google?

                                                    When develop site, I have a test domain is sandbox.abc.com, this site contents are same as abc.com. But, now I search site:sandbox.abc.com and aware of content duplicate with main site abc.com My question is how to remove all this link from goolge. p/s: I have just add robots.txt to sandbox and disallow all pages. Thanks,

                                                    Technical SEO | | JohnHuynh
                                                    0
                                                  • TalkInThePark

                                                    Googlebot does not obey robots.txt disallow

                                                    Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin

                                                    Technical SEO | | TalkInThePark
                                                    0
                                                  • kchandler

                                                    Robots.txt File Redirects to Home Page

                                                    I've been doing some site analysis for a new SEO client and it has been brought to my attention that their robots.txt file redirects to their homepage. I was wondering: Is there a benfit to setup your robots.txt file to do this? Will this effect how their site will get indexed? Thanks for your response! Kyle Site URL: http://www.radisphere.net/

                                                    Technical SEO | | kchandler
                                                    0

                                                  Get started with Moz Pro!

                                                  Unlock the power of advanced SEO tools and data-driven insights.

                                                  Start my free trial
                                                  Products
                                                  • Moz Pro
                                                  • Moz Local
                                                  • Moz API
                                                  • Moz Data
                                                  • STAT
                                                  • Product Updates
                                                  Moz Solutions
                                                  • SMB Solutions
                                                  • Agency Solutions
                                                  • Enterprise Solutions
                                                  • Digital Marketers
                                                  Free SEO Tools
                                                  • Domain Authority Checker
                                                  • Link Explorer
                                                  • Keyword Explorer
                                                  • Competitive Research
                                                  • Brand Authority Checker
                                                  • Local Citation Checker
                                                  • MozBar Extension
                                                  • MozCast
                                                  Resources
                                                  • Blog
                                                  • SEO Learning Center
                                                  • Help Hub
                                                  • Beginner's Guide to SEO
                                                  • How-to Guides
                                                  • Moz Academy
                                                  • API Docs
                                                  About Moz
                                                  • About
                                                  • Team
                                                  • Careers
                                                  • Contact
                                                  Why Moz
                                                  • Case Studies
                                                  • Testimonials
                                                  Get Involved
                                                  • Become an Affiliate
                                                  • MozCon
                                                  • Webinars
                                                  • Practical Marketer Series
                                                  • MozPod
                                                  Connect with us

                                                  Contact the Help team

                                                  Join our newsletter
                                                  Moz logo
                                                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                  • Accessibility
                                                  • Terms of Use
                                                  • Privacy

                                                  Looks like your connection to Moz was lost, please wait while we try to reconnect.