Hiding pages from Google Custom Search?

We implemented Google Custom Search on our site, however it will often return pages that have "Searchable" set to "No", and "Show in menu: No" -- so why does Google still see the pages?

For example, I’d like to prevent CSE from seeing XML files that MODX creates.

Does anyone know how to do this, without manually typing in every page into a robots.txt file?

IIRC, those checkboxes in the Manager are for MODx scripts, like AjaxSearch. I don’t think it affects stuff like Google’s search.

Quote from: worchyld at Dec 29, 2008, 06:46 AM

We implemented Google Custom Search on our site, however it will often return pages that have "Searchable" set to "No", and "Show in menu: No" -- so why does Google still see the pages?

For example, I’d like to prevent CSE from seeing XML files that MODX creates.

Does anyone know how to do this, without manually typing in every page into a robots.txt file?

If you have links to these pages and they do not explicitly have a noindex/nofollow attribute, robots.txt is your only course of action. Using something like the Google Analytics plugin might help in this respect, but there are a number of ways to address these issues...

When you mark a page as "NOT searchable" - a Robots meta tag
with "noindex, nofollow" is inserted to keep web search engines
from indexing that document.

More ---> http://modxcms.com/MetaTagsExtra-930.html

It turned out some of the pages had "Show in menu: Yes", I can’t switch them off because they are used in ditto calls.

I’m going to use the meta tags and robots.txt

The other alternative I’m using is to filter out any reference to *xml in the CSE and filter out any filetype:xml in the CSE.

Thanks

Sorry, quick question. Wouldn’t putting Meta tags like a robots meta tag inside an XML file prevent it from being a validate XML file?

Yes you are right MetaTagsExtra snippet will not help you.

Hmm... you may want to use the rel="nofollow" attribute in your page’s "Link Attributes" field. I haven’t tried this, but in theory, the "nofollow" attribute means that a search engine should ignore that page... MODx’s "Link Attributes" field is just an easy place to put that value. It wouldn’t hurt to try, unless you want the "regular" Google to search and index those pages.

The problem with using ’nofollow’ ( I think that should actually be ’noindex’) meta tags or restrictions in the robot.txt, is they also exclude the pages from the Google index. Which may or may not be what you want. For example, you may only want your product pages to appear in the search, but you still want Google to index your other pages.

According to the docs for Custom Search:

You can restrict the search to sites (webpages or websites) that you select, or you can create a junior Google that emphasizes certain sites. You list the specific websites and webpages in the Sites tab.

http://code.google.com/apis/customsearch/docs/start.html#managing