We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 51020
    • 670 Posts
    Hi There,

    I have been using StercSiteMap for a while, but have just noticed that pages I have marked as 'no follow, no index' are appearing in Search results on google, so i think I must have configured it wrongly.

    I have this in the head:

    	<meta name="robots" content="[[+seoTab.robotsTag]]">
    


    And if I set a page to be 'no follow and no index' it outputs:

    	<meta name="robots" content="noopd,noydir,noindex,nofollow">
    


    All good so far - but the said pages are featuring in my sitemap.xml, which is generated using the supplied snippet:

    [[!StercSeoSiteMap? &type=`index`]]
    


    But I would expect those resources marked as 'no follow' to be hidden from the sitemap.xml?

    Also, I have a robots.txt in my root directory, which I need to prevent the manager, core, connectors etc from being crawled, but if this is here, does that cancel out the above meta tags?

    Thanks as always
    Andy

    This question has been answered by andytough. See the first response.

      • 44922
      • 131 Posts
      Hi Andy,

      A robots.txt shouldn't be needed to prevent access to MODX manager folder etc.

      With your main question, if the pages were already cached before you implemented the 'nofollow', then you'll still see them. Click on the 'cached' link in the SERPs to see when your pages were indexed last.

      Remember, Google still will show 'nofollow' pages in the index, it just won't crawl or pull description and page data through. If you look online, there's anecdotal evidence to suggest that the sitemap.xml does override the on page 'nofollow', so maybe PM the creators of your Extra and ask them why they include no follow pages in the sitemap.xml.

      I'm never a fan of SEO plugins unless you're dealing with a big site with many pages, I have to say....


        • 51020
        • 670 Posts
        Quote from: lancipoos at Nov 20, 2017, 03:58 PM
        Hi Andy,

        A robots.txt shouldn't be needed to prevent access to MODX manager folder etc.

        With your main question, if the pages were already cached before you implemented the 'nofollow', then you'll still see them. Click on the 'cached' link in the SERPs to see when your pages were indexed last.

        Remember, Google still will show 'nofollow' pages in the index, it just won't crawl or pull description and page data through. If you look online, there's anecdotal evidence to suggest that the sitemap.xml does override the on page 'nofollow', so maybe PM the creators of your Extra and ask them why they include no follow pages in the sitemap.xml.

        I'm never a fan of SEO plugins unless you're dealing with a big site with many pages, I have to say....



        Thanks. I thought Google preferred to see a physical robots.txt in the root directory of the site. I will see if the developer can explain why the pages are in the sitemap.xml.
        Thanks
        Andy
        • discuss.answer
          • 38783
          • 571 Posts
          In addition to the Findability section which contains index/noindex, follow/nofollow, StercSeoPro includes a separate tab Google Sitemap in which you specify whether or not a page appears in the sitemap.xml files.
            If I help you out on these forums I would be very grateful if you would consider rating me on Trustpilot: https://uk.trustpilot.com/review/andytough.com

            email: [email protected] | website: https://andytough.com
            • 51020
            • 670 Posts
            Quote from: andytough at Nov 20, 2017, 05:29 PM
            In addition to the Findability section which contains index/noindex, follow/nofollow, StercSeoPro includes a separate tab Google Sitemap in which you specify whether or not a page appears in the sitemap.xml files.
            I have seoPro installed but I cannot see a Google Sitemap tab!
            Where is it?

            Thanks
            Andy
              • 51020
              • 670 Posts
              Quote from: andytough at Nov 20, 2017, 05:29 PM
              In addition to the Findability section which contains index/noindex, follow/nofollow, StercSeoPro includes a separate tab Google Sitemap in which you specify whether or not a page appears in the sitemap.xml files.

              Ah I found it - how have I never seen that before?!
              Sorted - thanks for your help!

              Andy