Quote from: BobRay at Jan 31, 2011, 01:07 PM
For problems like this, the core developers often aren’t in a position to investigate them unless the symptoms point to a particular piece of code because they can’t reproduce the problem on their platforms.
YEEEEEEEEESSSSSSSSSSSSS!!! I have found a pattern! And: A quick work around which is SOOOOOO simple!!!
First, let’s have a look at the webservers logfiles (I deleted some columns):
IP DATE / TIME REQUEST ERROR SIZE DOMAIN
95.108.151.244 [18/Nov/2010:03:53:14 GET /robots.txt HTTP/1.1 404 5700 my-modx-domain.de
95.108.151.244 [18/Nov/2010:03:53:14 GET /robots.txt HTTP/1.1 404 5704 www.my-modx-domain.de
95.108.151.244 [18/Nov/2010:03:53:16 GET / HTTP/1.1 200 435 my-modx-domain.de
95.108.151.244 [18/Nov/2010:03:53:16 GET / HTTP/1.1 200 435 www.my-modx-domain.de
or here
IP DATE / TIME REQUEST ERROR SIZE DOMAIN
95.108.150.235 [08/Dec/2010:06:42:15 GET /robots.txt HTTP/1.1 404 5703 www.my-modx-domain.de
95.108.150.235 [08/Dec/2010:06:42:15 GET /robots.txt HTTP/1.1 404 5699 my-modx-domain.de
95.108.150.235 [08/Dec/2010:06:42:19 GET / HTTP/1.1 200 435 my-modx-domain.de
95.108.150.235 [08/Dec/2010:06:42:20 GET / HTTP/1.1 200 435 www.my-modx-domain.de
or here
IP DATE / TIME REQUEST ERROR SIZE DOMAIN
208.115.111.250 [29/Jan/2011:19:11:31 GET /robots.txt HTTP/1.1 404 9782 my-modx-domain2.com
208.115.111.250 [29/Jan/2011:19:11:31 GET /robots.txt HTTP/1.1 404 9786 www.my-modx-domain2.com
64.34.218.178 [29/Jan/2011:19:21:04 GET / HTTP/1.1 200 399 my-modx-domain2.com
64.34.218.178 [29/Jan/2011:19:21:05 GET / HTTP/1.1 200 399 www.my-modx-domain2.com
any many more times. Allways with the same pattern:
An external website (a crawler) tries to get the "/robots.txt" which is not part of the standard MODx installation and gets an error 404. This happens at the SAME TIME for the domain WITH and WITHOUT the WWW. Since the error page is by default the id 1 the crawler is redirected to the start page where something in the cache goes wrong. The size of the resulting data drops from about 5600 bytes (or about 9700) to about 400 bytes: The request delivers only the header without any data. From now on every visitor of the startpage gets the "blank white screen of death".
A
first step to avoid that problem would be the
creation of the "/robots.txt". Crawler finds file, crawler is happy, everybody is happy! (Perhaps "/robots.txt" could be included in the future MODx releases?!)
Second, using a
different error page than the start page should also work but I dont know if then the error page gets blanked if the "/robots.txt" is requested at the same time at two places (with and without the "www"). Still the default id 1 for all those "special" pages can cause trouble in some cases (as seen above).
Finaly: Perhaps the MODx caching algorithm can be modified to be prepared for these rare (crawler) situations. That would make MODx even more robust and the user can decide if he wants an "/robots.txt" or if his error page is identical with the start page (Which many customers like A LOT since "they make error free products and there are no erros on their website". I asked and they told me.).
Hope this helps
Oliver
☆ A M B ☆
