• seohunters9

        See all notifications

        Skip to content
        Moz logo Menu open Menu close
        • Products
          • Moz Pro
          • Moz Pro Home
          • Moz Local
          • Moz Local Home
          • STAT
          • Moz API
          • Moz API Home
          • Compare SEO Products
          • Moz Data
        • Free SEO Tools
          • Domain Analysis
          • Keyword Explorer
          • Link Explorer
          • Competitive Research
          • MozBar
          • More Free SEO Tools
        • Learn SEO
          • Beginner's Guide to SEO
          • SEO Learning Center
          • Moz Academy
          • MozCon
          • Webinars, Whitepapers, & Guides
        • Blog
        • Why Moz
          • Digital Marketers
          • Agency Solutions
          • Enterprise Solutions
          • Small Business Solutions
          • The Moz Story
          • New Releases
        • Log in
        • Log out
        • Products
          • Moz Pro

            Your all-in-one suite of SEO essentials.

          • Moz Local

            Raise your local SEO visibility with complete local SEO management.

          • STAT

            SERP tracking and analytics for enterprise SEO experts.

          • Moz API

            Power your SEO with our index of over 44 trillion links.

          • Compare SEO Products

            See which Moz SEO solution best meets your business needs.

          • Moz Data

            Power your SEO strategy & AI models with custom data solutions.

          Let your business shine with Listings AI
          Moz Local

          Let your business shine with Listings AI

          Learn more
        • Free SEO Tools
          • Domain Analysis

            Get top competitive SEO metrics like DA, top pages and more.

          • Keyword Explorer

            Find traffic-driving keywords with our 1.25 billion+ keyword index.

          • Link Explorer

            Explore over 40 trillion links for powerful backlink data.

          • Competitive Research

            Uncover valuable insights on your organic search competitors.

          • MozBar

            See top SEO metrics for free as you browse the web.

          • More Free SEO Tools

            Explore all the free SEO tools Moz has to offer.

          NEW Keyword Suggestions by Topic
          Moz Pro

          NEW Keyword Suggestions by Topic

          Learn more
        • Learn SEO
          • Beginner's Guide to SEO

            The #1 most popular introduction to SEO, trusted by millions.

          • SEO Learning Center

            Broaden your knowledge with SEO resources for all skill levels.

          • On-Demand Webinars

            Learn modern SEO best practices from industry experts.

          • How-To Guides

            Step-by-step guides to search success from the authority on SEO.

          • Moz Academy

            Upskill and get certified with on-demand courses & certifications.

          • MozCon

            Save on Early Bird tickets and join us in London or New York City

          Unlock flexible pricing & new endpoints
          Moz API

          Unlock flexible pricing & new endpoints

          Find your plan
        • Blog
        • Why Moz
          • Digital Marketers

            Simplify SEO tasks to save time and grow your traffic.

          • Small Business Solutions

            Uncover insights to make smarter marketing decisions in less time.

          • Agency Solutions

            Earn & keep valuable clients with unparalleled data & insights.

          • Enterprise Solutions

            Gain a competitive edge in the ever-changing world of search.

          • The Moz Story

            Moz was the first & remains the most trusted SEO company.

          • New Releases

            Get the scoop on the latest and greatest from Moz.

          Surface actionable competitive intel
          New Feature

          Surface actionable competitive intel

          Learn More
        • Log in
          • Moz Pro
          • Moz Local
          • Moz Local Dashboard
          • Moz API
          • Moz API Dashboard
          • Moz Academy
        • Avatar
          • Moz Home
          • Notifications
          • Account & Billing
          • Manage Users
          • Community Profile
          • My Q&A
          • My Videos
          • Log Out

        The Moz Q&A Forum

        • Forum
        • Questions
        • My Q&A
        • Users
        • Ask the Community

        Welcome to the Q&A Forum

        Browse the forum for helpful insights and fresh discussions about all things SEO.

        1. Home
        2. SEO Tactics
        3. Intermediate & Advanced SEO
        4. Setting A Custom User Agent in Screaming Frog

        Moz Q&A is closed.

        After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

        Setting A Custom User Agent in Screaming Frog

        Intermediate & Advanced SEO
        3
        15
        5709
        Loading More Posts
        • Watching

          Notify me of new replies.
          Show question in unread.

        • Not Watching

          Do not notify me of new replies.
          Show question in unread if category is not ignored.

        • Ignoring

          Do not notify me of new replies.
          Do not show question in unread.

        • Oldest to Newest
        • Newest to Oldest
        • Most Votes
        Reply
        • Reply as question
        Locked
        This topic has been deleted. Only users with question management privileges can see it.
        • Malika1
          Malika1 last edited by

          Hi all,

          Probably a dumb question, but I wanted to make sure I get this right.

          How do we set a custom user agent in Screaming Frog? I know its in the configuration settings, but what do I have to do to create a custom user agent specifically for a website?

          Thanks much!

          • Malika
          1 Reply Last reply Reply Quote 0
          • BlueprintMarketing
            BlueprintMarketing last edited by

            Setting a custom user agent determines things like HTTP/2 so there can be a big difference if you change it to something that might not take advantage of something like HTTP/2

            Apparently, it is coming  to Pingdom very soon just like it is to Googlebot

            http://royal.pingdom.com/2015/06/11/http2-new-protocol/

            This Is an excellent example of a user agent's ability to modify the way your site is crawled as well as how efficient it is.

            https://www.keycdn.com/blog/https-performance-overhead/

            It is important to note that we didn’t use Pingdom in any of our tests because they use Chrome 39, which doesn’t support the new HTTP/2 protocol. HTTP/2 in Chrome isn’t supported until Chrome 43. You can tell this by looking at the User-Agent in the request headers of your test results.

            pingdom user-agent

            Pingdom user-agent

            Note: WebPageTest uses Chrome 47 which does support HTTP/2.

            Hope that clears things up,

            Tom

            ujNvaad.png t49XhGi.png

            1 Reply Last reply Reply Quote 0
            • BlueprintMarketing
              BlueprintMarketing @Malika1 last edited by

              Hi Malika,

              Think about screaming frog and what it has to detect in order to do that correctly it needs the correct  user agent syntax for it will not be able to make a crawl that would satisfy people.

              Using a proper syntax for a user agent is essential and I have tried to be non-technical in this explanation I hope it works.

              the reason screaming frog needs the user agent because the user-agent was added to HTTP to help web application developers deliver a better user experience.  By respecting the syntax and semantics of the header, we make it easier and faster for header parsers to extract useful information from the headers that we can then act on.

              Browser vendors are motivated to make web sites work no matter what specification violations are made.  When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that.  It is only by us application developers developing a healthy respect

              When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that.  It is only by us application developers developing a healthy respect

              It is only by us application developers developing a healthy respect for the standards of the web, that the browser vendors will be able to start tightening up their codebase knowing that they don’t need to account for non-conformances.

              For client libraries that do not enforce the syntax rules, you run the risk of using invalid characters that many server side frameworks will not detect.  It is possible that only certain users, in particular, environments would identify the syntax violation.  This can lead to difficult to track down bugs.

              I hope this is a good explanation I've tried to keep it very to the point.

              Respectfully,

              Thomas

              1 Reply Last reply Reply Quote 1
              • Malika1
                Malika1 @BlueprintMarketing last edited by

                Hi Thomas,

                would you have a simpler tutorial for me to understand? I am struggling a bit.

                Thanks heaps in advance 🙂

                1 Reply Last reply Reply Quote 0
                • Malika1
                  Malika1 @Malika1 last edited by

                  I think I want something that is dumbed down to my level for me to understand. The above tutorials are great but not being a full time coder, I get lost while reading those.

                  BlueprintMarketing 1 Reply Last reply Reply Quote 0
                  • Malika1
                    Malika1 @MattRoney last edited by

                    Hi Matt,

                    I havent had a luck with this one yet. 😞

                    Malika1 1 Reply Last reply Reply Quote 0
                    • MattRoney
                      MattRoney last edited by

                      Hi Malika! How'd it go? Did everything work out? 🙂

                      Malika1 1 Reply Last reply Reply Quote 1
                      • BlueprintMarketing
                        BlueprintMarketing @Malika1 last edited by

                        happy I could be of help let me know if there's any issue and I will try to be of help with it. All the best

                        Malika1 1 Reply Last reply Reply Quote 0
                        • Malika1
                          Malika1 @BlueprintMarketing last edited by

                          Hi Thomas,

                          That's a lot of useful information there. I will have a go on it and let you know how it went. 🙂

                          Thanks heaps!

                          BlueprintMarketing 1 Reply Last reply Reply Quote 1
                          • BlueprintMarketing
                            BlueprintMarketing last edited by

                            please let me know if I did not answer the question or you have any other questions

                            Malika1 1 Reply Last reply Reply Quote 0
                            • BlueprintMarketing
                              BlueprintMarketing @Malika1 last edited by

                              this gives you a very clear breakdown of  user agents and their  set of syntax rules.  The following is valid example of user-agent that is full of special characters,

                              read this please http://www.bizcoder.com/the-much-maligned-user-agent-header

                               

                              user-agent: foo&bar-product!/1.0a$*+ (a;comment,full=of/delimiters
                              

                              references but you want to pay attention to the first URL

                              http://www.howtogeek.com/113439/how-to-change-your-browsers-user-agent-without-installing-any-extensions/

                              https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference

                              | Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 |

                              http://stackoverflow.com/questions/15069533/http-request-header-useragent-variable

                              1 Reply Last reply Reply Quote 0
                              • BlueprintMarketing
                                BlueprintMarketing @Malika1 last edited by

                                if you formatted it correctly see below

                                User-Agent = product *( RWS ( product / comment ) )

                                and it was received by your headers yes you could fill in the blanks and test it.

                                https://mobiforge.com/research-analysis/webviews-and-user-agent-strings

                                http://mobiforge.com/news-comment/standards-and-browser-compatibility

                                1 Reply Last reply Reply Quote 0
                                • BlueprintMarketing
                                  BlueprintMarketing @Malika1 last edited by

                                  No, you Cannot just put anything in there. The site has to recognize it and ask why you are doing this?

                                  I have listed how to build and already built in addition to what your browser will create by using useragentstring.com

                                  Must be formatted correctly and have it work with a header it is not as easy as it sometimes seems but not that hard either.

                                  You can make & use this to make your own from your Mac or PC

                                  http://www.useragentstring.com/

                                  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2747.0 Safari/537.36

                                  how to build a user agent

                                  • https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference
                                  • https://developer.mozilla.org/en-US/docs/Setting_HTTP_request_headers
                                  • https://msdn.microsoft.com/en-us/library/ms537503(VS.85).aspx

                                  Lists of user agents

                                  https://support.google.com/webmasters/answer/1061943?hl=en

                                  http://www.user-agents.org/

                                  https://msdn.microsoft.com/en-us/library/ms537503(v=vs.85).aspx

                                  1 Reply Last reply Reply Quote 0
                                  • Malika1
                                    Malika1 @BlueprintMarketing last edited by

                                    Hi Thomas,

                                    Thanks for responding, much appreciated!

                                    Does that mean, if I type in something like -

                                    HTTP request user agent -

                                    Crawler access V2

                                    &

                                    Robots user agent

                                    Crawler access V2

                                    This will work too?

                                    BlueprintMarketing 3 Replies Last reply Reply Quote 0
                                    • BlueprintMarketing
                                      BlueprintMarketing last edited by

                                      To crawl using a different user agent, select ‘User Agent’ in the ‘Configuration’ menu, then select a search bot from the drop-down or type in your desired user agent strings.

                                      http://i.imgur.com/qPbmxnk.png

                                      &

                                      Video http://cl.ly/gH7p/Screen Recording 2016-05-25 at 08.27 PM.mov

                                      Or

                                      1. http://i.imgur.com/FAPVdSe.gifv
                                      2. http://i.imgur.com/p6obu0W.gifv

                                      Also see

                                      http://www.seerinteractive.com/blog/screaming-frog-guide/

                                      https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#user-agent

                                      https://www.screamingfrog.co.uk/seo-spider/user-guide/

                                      https://www.screamingfrog.co.uk/seo-spider/faq/

                                      https://www.screamingfrog.co.uk/seo-spider/

                                      dvlhxvF.gif p6obu0W.gif qPbmxnk.png

                                      Malika1 1 Reply Last reply Reply Quote 2
                                      • 1 / 1
                                      • First post
                                        Last post

                                      Browse Questions

                                      Explore more categories

                                      • Moz Tools

                                        Chat with the community about the Moz tools.

                                      • SEO Tactics

                                        Discuss the SEO process with fellow marketers

                                      • Community

                                        Discuss industry events, jobs, and news!

                                      • Digital Marketing

                                        Chat about tactics outside of SEO

                                      • Research & Trends

                                        Dive into research and trends in the search industry.

                                      • Support

                                        Connect on product support and feature requests.

                                      • See all categories

                                      Related Questions

                                      • bherman

                                        Can Schema handle two sets of business hours?

                                        I have a client who, due to covid, will have two sets of business hours. Morning hours for business customers, and afternoon hours for general customers. Is it possible to designate this distinction in schema?

                                        Intermediate & Advanced SEO | | bherman
                                        0
                                      • Jazee

                                        Set Placeholder Page ASAP or Wait For Full Website?

                                        It can take some time for a new business website to get picked up by all the search engines and indexed. Let's assume it's going to take a month to build your new full-fledged business website.  Would it be advantageous in the mean time to immediately launch the domain with an introductory website using a template site so you might have just two pages, a home page with logo, title, brief description of pages, a couple images, etc and a contact page. Would this help give the site a "jump start" on being indexed? Or could that do more harm than good by putting up something "quick & dirty" versus the complete website with much more content, that has been SEO optimized?

                                        Intermediate & Advanced SEO | | Jazee
                                        0
                                      • mrodriguez1440

                                        Screaming Frog returning both HTTP and HTTPS results...

                                        Hi, About 10 months I switched from HTTP to HTTPS. I then switched back (long story). I noticed that Screaming Frog is picking up the HTTP and HTTPS version of the site. Maybe this doesn't matter, but I'd like to know why SF is doing that. The URL is: www.aerlawgroup.com Any feedback, including how to remove the HTTPS version, is greatly appreciated. Thanks.

                                        Intermediate & Advanced SEO | | mrodriguez1440
                                        0
                                      • kevinliao

                                        What happens to a domain in SERPs when it's set to redirect to another?

                                        We have just acquired a competing website and are wondering whether to leave it running as is for now, or set the domain to redirect to our own site. If we set up this redirect, what would happen to the old site in Google SERPs? Would the site drop off from results? If so, would we capture this new search traffic or is it a free for all and all sites compete for the search traffic as normal? Thanks in advance. Paul

                                        Intermediate & Advanced SEO | | kevinliao
                                        0
                                      • Kyle_M

                                        Layered navigation and hiding nav from user agent

                                        I am trying to deal with the duplicate content issues presented by Magento's layered navigation feature (aka faceted navigation). I installed Amasty's Improved Navigation extension (https://amasty.com/improved-layered-navigation.html) and it offers the option to hide the layered navigation from specific user agents (ie googlebot, bingbot, etc). This seems like cloaking to me and I hesitate to try it, unless hiding faceted navigation from specific user agents is known to be acceptable to Google (white hat practice). Does anyone know if this the case?

                                        Intermediate & Advanced SEO | | Kyle_M
                                        0
                                      • McTaggart

                                        Strange 404s in Screaming Frog

                                        I just ran a website (Drupal) through screaming frog and the only 404s I found related to web pages which were the same as URLs already used on the website plus the company phone number so... www.company.com/[their phone number] - www.company.com/services[their phone number] - any ideas what might be causing this problem?

                                        Intermediate & Advanced SEO | | McTaggart
                                        0
                                      • Colin.Accela

                                        Setting up 301 Redirects after acquisition?

                                        Hello! The company that I work for has recently acquired two other companies. I was wondering what the best strategy would be as it relates to redirects / authority. Please help! Thanks

                                        Intermediate & Advanced SEO | | Colin.Accela
                                        0
                                      • PeterRota

                                        Set up a rel canonical

                                        I have a question. I was wondering, if it was possible to set up a rel canonical. When I can't access the non canonical pages? For example, my site as at www.site.com , but the non cannocail is at site.com is their any way to set thet up without actually edting it at site.com ? Thanks for your help

                                        Intermediate & Advanced SEO | | PeterRota
                                        0

                                      Get started with Moz Pro!

                                      Unlock the power of advanced SEO tools and data-driven insights.

                                      Start my free trial
                                      Products
                                      • Moz Pro
                                      • Moz Local
                                      • Moz API
                                      • Moz Data
                                      • STAT
                                      • Product Updates
                                      Moz Solutions
                                      • SMB Solutions
                                      • Agency Solutions
                                      • Enterprise Solutions
                                      • Digital Marketers
                                      Free SEO Tools
                                      • Domain Authority Checker
                                      • Link Explorer
                                      • Keyword Explorer
                                      • Competitive Research
                                      • Brand Authority Checker
                                      • Local Citation Checker
                                      • MozBar Extension
                                      • MozCast
                                      Resources
                                      • Blog
                                      • SEO Learning Center
                                      • Help Hub
                                      • Beginner's Guide to SEO
                                      • How-to Guides
                                      • Moz Academy
                                      • API Docs
                                      About Moz
                                      • About
                                      • Team
                                      • Careers
                                      • Contact
                                      Why Moz
                                      • Case Studies
                                      • Testimonials
                                      Get Involved
                                      • Become an Affiliate
                                      • MozCon
                                      • Webinars
                                      • Practical Marketer Series
                                      • MozPod
                                      Connect with us

                                      Contact the Help team

                                      Join our newsletter
                                      Moz logo
                                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                      • Accessibility
                                      • Terms of Use
                                      • Privacy

                                      Looks like your connection to Moz was lost, please wait while we try to reconnect.