• majorAlexa

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. Technical SEO
        4. Crawl solutions for landing pages that don't contain a robots.txt file?

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Crawl solutions for landing pages that don't contain a robots.txt file?

        Technical SEO
        3
        10
        1662
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • Nomader
          Nomader last edited by

          My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?

          1 Reply Last reply Reply Quote 1
          • Nomader
            Nomader @BlueprintMarketing last edited by

            No problem Tom. Thanks for the additional info — that is helpful to know.

            1 Reply Last reply Reply Quote 1
            • BlueprintMarketing
              BlueprintMarketing @Nomader last edited by

              Bryan,

              I’m glad that you found what you where looking for.

              I must have missed the part about it being 100% Instapage when you said CMS I thought  meant something on else with instapage I think of it as landing pages not a CMS

              I want to help so you asked about Google search console how often you need to request  google index  your site.

              First make sure

              You should have 5 urls in Google search console

              your domain, http://www. , http:// , https://www. & https://

              • nomader.com
              • https://www.nomader.com
              • https://nomader.com
              • http;//www.nomader.com
              • http://nomader.com

              you should not have to  requests google index once you’re pages are in googles index. There is no time line to make you need to requests  google index.

              Use search consoles  index  system to see if you need to make a request  and look for notifications

              Times you should request google crawl when adding new unlinked pages , when making big changes to your site , whatever adding pages  with out a xml sitemap or fixing problems  / testing.

              I want to help so as you said you’re going to be using Shopify.

              Just before you go live  running on Shopify in the future you should make a xml sitemap of the Instapage site

              You can do it for free using https://www.screamingfrog.co.uk/seo-spider/

              you’re running now name it  /sitemap_ip.xml or /sitemap2.xml  upload it to Shopify

              & make sure it’s not the same name so it will work with your Shopify xml sitemap  /sitemap.xml

              submit the /sitemap._ip.xml to search console then add the Shopify /sitemap.xml

              You can run multiple xml sitemaps as long as they are not  overlapping

              just remember never add  non-200 page, 404s, 300sno flow , no index or redirects  to a xml sitemap  ScreamingFrog  will ask if you want to when you’re making the sitemap.

              Shopify will make its own xml sitemaps and and having the current site as a second xml sitemap will help to make sure your change to the site will not hurt the intipage par of  the Shopify site

              https://support.google.com/webmasters/answer/34592?hl=en

              know  adding a XML  Sitemap is a smart move

              I hope that was of help I’m so about miss what you meant.

              respectfully,

              Tom

              https://builtwith.com/relationships/nomader.com

              https://builtwith.com/redirects/nomader.com

              Nomader 1 Reply Last reply Reply Quote 1
              • Nomader
                Nomader @seoelevated last edited by

                Thanks so much for your thoughtful, detailed response. That answers my question.

                1 Reply Last reply Reply Quote 0
                • seoelevated
                  seoelevated Subscriber last edited by

                  Bryan,

                  If I understand your intent, you want your pages indexed. I see that your site has 5 pages indexed (/, /help, /influencers, /wholesale, /co-brand). And that you have some other pages (e.g. /donations), which are not indexed, but these have "noindex" tags explicitly in their HEAD sections.

                  Not having a robots.txt file is equal to having a robots.txt file with a directive to allow crawling of all pages. This is per http://www.robotstxt.org/orig.html, where they say "The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome."

                  So, if you have no robots.txt file, the search engine will feel free to crawl everything it discovers, and then whether or not it indexes those pages will be guided by presence or absence of NOINDEX tags in your HEAD sections. From a quick browse of your site and its indexed pages, this seems to be working properly.

                  Note that I'm referencing a distinction between "crawling" and "indexing".  The robots.txt file provides directives for crawling (i.e. access discovered pages, and discovering pages linked to those). Whereas the meta robots tags in the head provide directives for indexing (i.e. including the discovered pages in search index and displaying those as results to searchers). And in this context, absence of a robots.txt file simply allows the search engine to crawl all of your content, discover all linked pages, and then rely on meta robots directives in those pages for any guidance on whether or not to index those pages it finds.

                  As for a sitemap, while they are helpful for monitoring indexation, and also provide help to search engines to discover all desired pages, in your case it doesn't look especially necessary. Again, I only took a quick look, but it seems you have your key pages all linked from your home page, and you have meta directives in pages you wish to keep out of the index. And you have a very small number of pages. So, it looks like you are meeting your crawl and indexation desires.

                  Nomader 1 Reply Last reply Reply Quote 1
                  • Nomader
                    Nomader @BlueprintMarketing last edited by

                    Hi Tom,

                    Unfortunately, Instapage is a proprietary CMS that does not currently support robots.txt or site maps. Instapage is primarily built for landing pages, and not actual websites so that's their reasoning for not adding SEO support for basics like robots.txt and site maps.

                    Thanks anyway for your help.

                    Best,

                    -Bryan

                    BlueprintMarketing 1 Reply Last reply Reply Quote 0
                    • BlueprintMarketing
                      BlueprintMarketing @Nomader last edited by

                      hi

                      so I see the problem now

                      https://www.nomader.com/robots.txt

                      Does not have a robots.txt file upload it to the root of your server or  specific place where Developer and/or CMS / Hosting company recommends  I could not figure out what to type of CMS you’re useing  if you’re using one

                      make a robots.txt file using

                      http://tools.seobook.com/robots-txt/generator/

                      https://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/exportrobots.php

                      https://a-moz.groupbuyseo.org/learn/seo/robotstxt

                      It will look like this below.

                      User-Agent: *
                      Disallow:

                      Sitemap: https://www.nomader.com/sitemap.xml

                      it looks like you’re using Java  for your website?

                      https://builtwith.com/detailed/nomader.com

                      I am guessing  you’re not using a subdomain to host the Landing Pages?

                      If you are using a subdomain you would have to create a robots.txt file for that but from everything I can see you’re using your regular domain. So you would simply create these files ( i’m in a car on a cell phone so I did quick to see  check if you have a  XML site map file but I do think you do

                      https://www.nomader.com/sitemap.xml

                      You can purchase a tool called Screaming Frog SEO spider  if your site is over 500 pages  you will need to pay for it it’s approximately $200 however you will be able to create a wonderful site map you can also create a XML site map by googling   xml sitemap  generators. However I would recommend Screaming Prod because you can separate the images and it’s a very good tool to have.

                      Because you will need to generate a new site map whenever you update your site or add Landing Pages it will be done using screaming frog and uploaded to the same place in the server. Unless you can create a dynamic sitemap  using whatever website of the  infrastructure structure using.

                      Here are the directions to add your site  Google Search Console /  Google Webmaster Tools

                      https://support.google.com/webmasters/answer/34592?hl=en

                      If you need any help with any of this please do not hesitate to ask I am more than happy to help  you can also generate a site map in the old version of Google Webmaster Tools / Google Search Console.

                      Hope this helps,

                      Tom

                      Nomader 1 Reply Last reply Reply Quote 1
                      • Nomader
                        Nomader @BlueprintMarketing last edited by

                        Thanks for the reply Thomas. Where do you see that my site has the robots.txt file? As far as I can tell, it is missing. Instapage does not offer robots.txt as I mentioned in my post. Here's a community help page of theirs where this question was asked and answered: https://help.instapage.com/hc/en-us/community/posts/213622968-Sitemap-and-Robotx-txt

                        So in the absence of having a robots.txt file, I guess the only way to counter this is to manually request a fetch/index from Google console? How often do you recommend I do this?

                        BlueprintMarketing 1 Reply Last reply Reply Quote 0
                        • BlueprintMarketing
                          BlueprintMarketing @BlueprintMarketing last edited by

                          You don’t need to worry about instapage & robot.txt your site has the robots.txt & instapage is not set to no index.

                          so yes use google search console to fetch / index the pages it’s very easy if you read the help information I posted below

                          https://help.instapage.com/hc/en-us#

                          hope that helps,

                          Tom

                          Nomader 1 Reply Last reply Reply Quote 1
                          • BlueprintMarketing
                            BlueprintMarketing last edited by

                            If you cannot turn off  “Meta Noindex“ you cannot fix it with robots.txt I  suggest you contact the developer of the  Instapage  landing pages app. If it’s locked to no index as you said that is the only of for countering a pre coded  by the company Meta Noindex issue?

                            I will look into this for you I bet that you can change it but not via robots.txt. I

                            will update it in the morning for you.

                            All the best,

                            Tom

                            BlueprintMarketing 1 Reply Last reply Reply Quote 1
                            • 1 / 1
                            • First post
                              Last post

                            Browse Questions

                            Explore more categories

                            • Moz Tools

                              Chat with the community about the Moz tools.

                            • SEO Tactics

                              Discuss the SEO process with fellow marketers

                            • Community

                              Discuss industry events, jobs, and news!

                            • Digital Marketing

                              Chat about tactics outside of SEO

                            • Research & Trends

                              Dive into research and trends in the search industry.

                            • Support

                              Connect on product support and feature requests.

                            • See all categories

                            Related Questions

                            • Pete4

                              Why is Google Webmaster Tools showing 404 Page Not Found Errors for web pages that don't have anything to do with my site?

                              I am currently working on a small site with approx 50 web pages.  In the crawl error section in WMT Google has highlighted over 10,000 page not found errors for pages that have nothing to do with my site.  Anyone come across this before?

                              Technical SEO | | Pete4
                              0
                            • jmueller

                              Two META Robots tags on a page - which will win?

                              Hi, Does anybody know which meta-robots tag will "win" if there is more than one on a page? The situation:
                              our CMS is not very flexible and so we have segments of META-Tags on the page that originate from templates.
                              Now any author can add any meta-tag from within his article-editor.
                              The logic delivering the pages does not care if there might be more than one meta-robots tag present (one from template, one from within the article). Now we could end up with something like this: Which one will be regarded by google & co?
                              First?
                              Last?
                              None? Thanks a lot,
                              Jan

                              Technical SEO | | jmueller
                              0
                            • reidsteven75

                              How Does Google's "index" find the location of pages in the "page directory" to return?

                              This is my understanding of how Google's search works, and I am unsure about one thing in specific: Google continuously crawls websites and stores each page it finds (let's call it "page directory") Google's "page directory" is a cache so it isn't the "live" version of the page Google has separate storage called "the index" which contains all the keywords searched.  These keywords in "the index" point to the pages in the "page directory" that contain the same keywords. When someone searches a keyword, that keyword is accessed in the "index" and returns all relevant pages in the "page directory" These returned pages are given ranks based on the algorithm The one part I'm unsure of is how Google's "index" knows the location of relevant pages in the "page directory".  The keyword entries in the "index" point to the "page directory" somehow. I'm thinking each page has a url in the "page directory", and the entries in the "index" contain these urls.   Since Google's "page directory" is a cache, would the urls be the same as the live website (and would the keywords in the "index" point to these urls)? For example if webpage is found at wwww.website.com/page1, would the "page directory" store this page under that url in Google's cache? The reason I want to discuss this is to know the effects of changing a pages url by understanding how the search process works better.

                              Technical SEO | | reidsteven75
                              0
                            • TruvoDirectories

                              Can too many pages hurt crawling and ranking?

                              Hi, I work for local yellow pages in Belgium, over the last months we introduced a succesfull technique to boost SEO traffic: we have created over 150k of new pages, all targeting specific keywords and all containing unique content, a site architecture to enable google to find these pages through crawling, xml sitemaps, .... All signs (traffic, indexation of xml sitemaps, rankings, ...) are positive. So far so good. We are able to quickly build more unique pages, and I wonder how google will react to this type of "large scale operation": can it hurt crawling and ranking if google notices big volumes of content (unique content)? Please advice

                              Technical SEO | | TruvoDirectories
                              0
                            • Webmaster123

                              I accidentally blocked Google with Robots.txt. What next?

                              Last week I uploaded my site and forgot to remove the robots.txt file with this text: User-agent: * Disallow: / I dropped from page 11 on my main keywords to past page 50. I caught it 2-3 days later and have now fixed it. I re-imported my site map with Webmaster Tools and I also did a Fetch as Google through Webmaster Tools. I tweeted out my URL to hopefully get Google to crawl it faster too. Webmaster Tools no longer says that the site is experiencing outages, but when I look at my blocked URLs it still says 249 are blocked. That's actually gone up since I made the fix. In the Google search results, it still no longer has my page title and the description still says "A description for this result is not available because of this site's robots.txt – learn more." How will this affect me long-term? When will I recover my rankings? Is there anything else I can do? Thanks for your input! www.decalsforthewall.com

                              Technical SEO | | Webmaster123
                              0
                            • HMK-NL

                              No indexing url including query string with Robots txt

                              Dear all, how can I block url/pages with query strings like page.html?dir=asc&order=name with robots txt? Thanks!

                              Technical SEO | | HMK-NL
                              0
                            • tylerfraser

                              Can I Disallow Faceted Nav URLs - Robots.txt

                              I have been disallowing /*? So I know that works without affecting crawling. I am wondering if I can disallow the faceted nav urls. So disallow: /category.html/? /category2.html/? /category3.html/*? To prevent the price faceted url from being cached: /category.html?price=1%2C1000
                              and
                              /category.html?price=1%2C1000&product_material=88 Thanks!

                              Technical SEO | | tylerfraser
                              0
                            • nicole.healthline

                              Is blocking RSS Feeds with robots.txt necessary?

                              Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!

                              Technical SEO | | nicole.healthline
                              0

                            Get started with Moz Pro!

                            Unlock the power of advanced SEO tools and data-driven insights.

                            Start my free trial
                            Products
                            • Moz Pro
                            • Moz Local
                            • Moz API
                            • Moz Data
                            • STAT
                            • Product Updates
                            Moz Solutions
                            • SMB Solutions
                            • Agency Solutions
                            • Enterprise Solutions
                            • Digital Marketers
                            Free SEO Tools
                            • Domain Authority Checker
                            • Link Explorer
                            • Keyword Explorer
                            • Competitive Research
                            • Brand Authority Checker
                            • Local Citation Checker
                            • MozBar Extension
                            • MozCast
                            Resources
                            • Blog
                            • SEO Learning Center
                            • Help Hub
                            • Beginner's Guide to SEO
                            • How-to Guides
                            • Moz Academy
                            • API Docs
                            About Moz
                            • About
                            • Team
                            • Careers
                            • Contact
                            Why Moz
                            • Case Studies
                            • Testimonials
                            Get Involved
                            • Become an Affiliate
                            • MozCon
                            • Webinars
                            • Practical Marketer Series
                            • MozPod
                            Connect with us

                            Contact the Help team

                            Join our newsletter
                            Moz logo
                            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                            • Accessibility
                            • Terms of Use
                            • Privacy

                            Looks like your connection to Moz was lost, please wait while we try to reconnect.