robot.txt code meaning

Hi,
i was just testing my robot.txt with google webmaster tools.
but i need to know what it the meaning of these default values that are already
in the text file.

# Default modx exclusions
User-agent: *
Disallow: /assets/cache/
Disallow: /assets/docs/
Disallow: /assets/export/
Disallow: /assets/import/
Disallow: /assets/modules/
Disallow: /assets/plugins/
Disallow: /assets/snippets/
Disallow: /install/
Disallow: /manager/

is it modx values by default? should i worry about it or just leave it as it is?

thanks for any help.

Quote from: nadz83 at Feb 03, 2010, 03:15 AM

Hi,
i was just testing my robot.txt with google webmaster tools.
but i need to know what it the meaning of these default values that are already
in the text file.

# Default modx exclusions
User-agent: *
Disallow: /assets/cache/
Disallow: /assets/docs/
Disallow: /assets/export/
Disallow: /assets/import/
Disallow: /assets/modules/
Disallow: /assets/plugins/
Disallow: /assets/snippets/
Disallow: /install/
Disallow: /manager/

is it modx values by default? should i worry about it or just leave it as it is?

thanks for any help.

the Disallow: /xxx/ stops crawlers from search engines from indexing these pages in the search results.. no1 should know about these URLS especially the /install/ so u shouldnt erase them, but you can add to them it if theirs a page you dont want indexed