Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Robots.txt and Multiple Sitemaps
- 
					
					
					
					
 Hello, I have a hopefully simple question but I wanted to ask to get a "second opinion" on what to do in this situation. I am working on a clients robots.txt and we have multiple sitemaps. Using yoast I have my sitemap_index.xml and I also have a sitemap-image.xml I do put them in google and bing by hand but wanted to have it added into the robots.txt for insurance. So my question is, when having multiple sitemaps called out on a robots.txt file does it matter if one is before the other? From my reading it looks like you can have multiple sitemaps called out, but I wasn't sure the best practice when writing it up in the file. Example: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://sitename.com/sitemap_index.xml Sitemap: http://sitename.com/sitemap-image.xml Thanks a ton for the feedback, I really appreciate it! :) J
- 
					
					
					
					
 Awesome! yea I submitted them to bing and google by hand, I just figured it couldn't hurt to have it in my robots too. Appreciate the feedback  
- 
					
					
					
					
 Yes, what you have is the proper format. The best way to submit sitemaps, of course, is to submit them via Google & Bing Webmaster Tools. Sitemaps won't have much impact on your site unless you have a really large site, so I wouldn't focus on them too much. The best way to get content crawled & indexed by Google is good internal link structure and authoritative external links. 
Browse Questions
Explore more categories
- 
		
		Moz ToolsChat with the community about the Moz tools. 
- 
		
		SEO TacticsDiscuss the SEO process with fellow marketers 
- 
		
		CommunityDiscuss industry events, jobs, and news! 
- 
		
		Digital MarketingChat about tactics outside of SEO 
- 
		
		Research & TrendsDive into research and trends in the search industry. 
- 
		
		SupportConnect on product support and feature requests. 
Related Questions
- 
		
		
		
		
		
		Disallow wildcard match in Robots.txt
 This is in my robots.txt file, does anyone know what this is supposed to accomplish, it doesn't appear to be blocking URLs with question marks Disallow: /?crawler=1 Technical SEO | | AmandaBridge
 Disallow: /?mobile=1 Thank you0
- 
		
		
		
		
		
		Desktop & Mobile XML Sitemap Submitted But Only Desktop Sitemap Indexed On Google Search Console
 Hi! The Problem We have submitted to GSC a sitemap index. Within that index there are 4 XML Sitemaps. Including one for the desktop site and one for the mobile site. The desktop sitemap has 3300 URLs, of which Google has indexed (according to GSC) 3,000 (approx). The mobile sitemap has 1,000 URLs of which Google has indexed 74 of them. The pages are crawlable, the site structure is logical. And performing a Landing Page URL search (showing only Google/Organic source/medium) on Google Analytics I can see that hundreds of those mobile URLs are being landed on. A search on mobile for a longtail keyword from a (randomly selected) page shows a result in the SERPs for the mobile page that judging by GSC has not been indexed. Could this be because we have recently added rel=alternate tags on our desktop pages (and of course corresponding canonical ones on mobile). Would Google then 'not index' rel=alternate page versions? Thanks for any input on this one. PmHmG Technical SEO | | AlisonMills0
- 
		
		
		
		
		
		Multiple robots.txt files on server
 Hi! I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step. One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names: robots.txt (original dupplicate) Technical SEO | | mjukhud
 robots.txt-Original (original)
 robots.txt-NEW (other content)
 robots.txt-Working (other content dupplicate) Would really appreciate help and expertise suggestions. Thanks!0
- 
		
		
		
		
		
		Include or exclude noindex urls in sitemap?
 We just added tags to our pages with thin content. Should we include or exclude those urls from our sitemap.xml file? I've read conflicting recommendations. Technical SEO | | vcj0
- 
		
		
		
		
		
		One robots.txt file for multiple sites?
 I have 2 sites hosted with Blue Host and was told to put the robots.txt in the root folder and just use the one robots.txt for both sites. Is this right? It seems wrong. I want to block certain things on one site. Thanks for the help, Rena Technical SEO | | renalynd270
- 
		
		
		
		
		
		Should I include tags in sitemap?
 Hello All, I was wondering if you should include tags and categories in your sitemap. In the past on previous blogs I have always left tags and categories out. The reason for this is a good friend of mine who has been doing SEO for a long time and inhouse always told me that this would result in duplicate content. I thought that it would be a great idea to get some input from the SEOmoz community as this obviously has a big affect on your blog and the number of pages indexed. Any help would be great. Thanks, Luke Hutchinson. Technical SEO | | LukeHutchinson1
- 
		
		
		
		
		
		How to generate a visual sitemap using sitemap.xml
 Are there any tools (online preferably) which will take a sitemap.xml file and generate a visual site map? Seems like an obvious thing to do, but can't find any simple tools for this? Technical SEO | | k3nn3dy30
 
			
		 
			
		 
					
				 
					
				 
					
				 
					
				 
					
				 
					
				