Index.html results in 404 but old site has many links to index.html

Hi,

I have converted a site from a previous index.html effort. Using google webmasters I discovered there are lots of links to www.domain.com/index.html. This must be a common occurrence. Looking at a few MODx sites they all return a 404 for index.html. I’m concerned that Search Engines will find lots of links to the site (with index.html) but when the bots get there they will see a 404 and assume the link is dead. Potentially disastrous for SEO.

I recently discovered - http://www.searchmasters.co.nz/redirect-check/
Which looks at your site and gives handy tips on htaccess ReWrites etc.

I tried it for http://www.verdantconsulting.co.uk/ and got this code (snippet).



Google Webmasters Labs Fetch as Googlebot when directed to http://www.verdantconsulting.co.uk/index.html returns a 404 but with the genuine page tagging along. Therefore, the page looks correct but the BOT may ignore the contents due to the 404. This is a handy feature in cases where a completely wrong URL is used - at least the punter sees the home page.

So, a couple of queries.
1) The code I added (to the end of the default htaccess) doesn’t run the redirect. Perhaps it is wrong. Or maybe something intercepts this kind of approach. So /index.html doesn’t become /
2) Is it healthy for all MODx sites to treat index.html (the old faithful) as a 404? Will this treatment kill SEO for many inbound links with the full path?

A bit complicated. Hope someone can follow this.

Parthian

First, what version of MODx?

Second, if this is MODx 2.0 (aka Revolution) and the page with the alias index is your site_start then it should redirect to /. This is default behavior for site_start, and it should be fine as it will be a 301 redirect. If you are getting a 404, make sure you did not include .html in the alias, as this is added by the Content Types in MODx Revolution.

Blast - thought I’d added the version. MODx Evo 1.03.

Quote from: parthian at Aug 02, 2010, 12:46 PM

Blast - thought I’d added the version. MODx Evo 1.03.

In that case, did you give your start_page an alias of index ?

OG,

The home page had a URL Alias of home - which rarely appeared. Set it to ’index’ but this resulted in www.domain.com/index returning a 200 as expected but www.domain.com/index.html continues to return a 404. Perhaps the alias should be set to index.html?

But I don’t actually want index.html to be used. It’s just a coping strategy due to lots of old (but valuable) links to the site. I’m thinking that the htaccess should handle this before modx even gets started. Not sure about that though.

Parthian

Is what I usually use on a LAMP server.
Rewrites http:/example.com/ to http://www.example.com/ and then index.html to the root

Quote from: parthian at Aug 02, 2010, 04:24 PM

The home page had a URL Alias of home - which rarely appeared. Set it to ’index’ but this resulted in www.domain.com/index returning a 200 as expected but www.domain.com/index.html continues to return a 404. Perhaps the alias should be set to index.html?

If your furl_suffix is not .html, then yes, it should be index.html

No need to make mod_rewrite work harder.

Thanks Chaps,

Here’s where I am. Goal is to improve SEO and keep nice neat technology independent FURLs (so no extensions).
I reckon (guess) that Bots will not like being sent to a link which returns a 404. As things stand, with the way the htaccess works, anything that isn’t an actual file or directory gets remapped to index.php such that the actual site appears (so punter is happy) but a 404 is also returned which I think will be bad for SEO.

This is the standard MODx htaccess code

This uses L for Last but no R=301 so it is quicker (no handshake with client to tell if of the Permanent Redirect) but for me has the downside that the L(ast) rewrites locally index.html to index.php (thus displaying the page) but also means server sends a 404.

I tried moving this code
to before MODxFURLs code and bingo: the client sees a redirect from www.d.com/index.html to what I want www.d.com/
This also prevents 404s and because the client sees the correct URL that I want spread around the web any links they create (in articles/blogs etc) will also be correct. Thus stopping the propogation of the old www.d.com/index.html

OpenGeek, a timely reminder about the extensions - forgot about them, and a good point about my approach causing more work but I think this is right for me. A few tests in FF with Firebug using a www.d.com/index.html bookmark results in index.html 301’d to / and this takes between 60 and 100ms (UK to UK though so could be slower elsewhere). A small price I think for good SEO and the FURLs I want. Perhaps I’ve missed something.


Assuming above is correct I’ve one problem. What URL Alias to give the home page. I was using ’home’ but this only appears in the URL when using the breadcrumbs link to home. I don’t want people bookmarking/linking to www.d.com/home as this will be back to square one. A blank URL Alias results in www.d.com/1 which is worse. Any ideas? Maybe a change to breadcrumbs?

Parthian

Should have said make sure the rules are before the FURL rules
You could just give your start page an alias of index.html, which then should get re-written back to the root.

bunk58

Yes that would have been handy...
I’ve tried quite a few sites from the Showcase about half return a 404 for index.html urls. Not a problem if they’re new sites or never been linked/indexed etc as index.html but potentially seriously bad for SEO if they have. The link juice may die... :’(

I wonder if the developers would think it important enough to add a comment to the default htaccess discussing the issue? With a commented out example as you supplied. Just a thought. I’ll need to do more BOT research to find out if it really is a serious problem.

And using URL alias of index.html with the index.html to root 301, before the FURL stuff, works a treat.

Cheers