Answered robots.txt on modxcloud?????

Great!!! Thank you all!!!! Thank you very much!!!!

Have a nice time!!!! Enjoy modx and modx Community!!!! Its the greatest place i know...

I'm late to the party here, but wanted to inform you all that we did implement a new feature last week, where new Developer Clouds automatically have a robots.txt (virtual) file that directs robots not to index any of the site. If you want to override this default behavior, all you have to do is put an actual file in place.

Nice to see different ideas for solving the problem of excluding just parts of a site.

Great, if we inject the development cloud into a production cloud is the robots file removed?

Quote from: eighthday at Mar 28, 2013, 09:00 AM

Great, if we inject the development cloud into a production cloud is the robots file removed?

Yes it is. Not that it matters, but we're doing this via server configuration and not by putting an actual file in place. Gives us a lot more flexibility, and I wouldn't be surprised to see other cool little features in the future based on this idea.

Quote from: netProphET at Mar 28, 2013, 02:35 AM

I'm late to the party here, but wanted to inform you all that we did implement a new feature last week, where new Developer Clouds automatically have a robots.txt (virtual) file that directs robots not to index any of the site. If you want to override this default behavior, all you have to do is put an actual file in place.

Nice to see different ideas for solving the problem of excluding just parts of a site.

I'm even later to the party - but thought it worthwhile to clarify something in case other people stumble across this thread as I did.

So to clarify, the robots.txt that's returned for your Cloud URL - e.g. cxxxx.paas1.tx.modxcloud.com is always the default "deny all". And that's not changed even you enable the "Allow Search Engines to Index this site" option on the cloud. However that setting does change the robots.txt file returned for the "custom" URL assigned to that cloud .. e.g. dev.yourcompany.modxcloud.com.

This caught me out because I had implemented a robots.txt and was testing it using the cloud address, and it was refusing to serve my custom robots.txt file. Hope this helps someone else, and thanks to Mike Schell at Modx Cloud for clarifying this for me!

Cheers

Mark

Thanks for the follow up @endacemark. Here's a synopsis of how robots.txt works in MODX Cloud:

• Internal URLs will always serve deny: all to robots. These are the ones that start with a cXXXX number.
• You can enable robots.txt for your cloud urls. These will look like site-name.user-account.modxcloud.com.
• Robots.txt should be enabled by default for any custom domains/URLs you add to a project in MODX Cloud.