[IDEA] Changes to handling of unpublished documents

In a strive to make MODx a more SEO CMF, here’s another one of my proposals.

I’ve been reading up on HTTP Status Codes and noticed the 410-Gone code and I got an idea. Why not make an option for users to send this status code along if there’s a request for an unpublished document. The idea fits the description on Wikipedia.

Reason being that the way I use the publish-unpublish system is to for news articles, and there are some articles I wish to be taken down after a certain time. Well if someone’s bookmarked the article and comes back to it later, after it’s been un-published, it would be far more informational to supply them with a "Gone" message rather than a "Not Found" message. It also helps search engines keep their indexes clean, that is if they know what the 410 status code is!

Am I right or missing something important here about the 410 code?

I’d be happy to implement it into the 095dev branch as an optional feature, but I want to see if anyone else will use it

That actually could be useful for membership sites with temporary access to free articles that then go membership-only. Then you could have a 404, a 410 and other pages as well. Duplicate the article to the private pages, and unpublish the free version.

I don’t think all unpublished content should return a "Gone" code. For example, many people write their documents in the unpublished state before they go live. Sending a "Gone" code for a page that has not existed publically yet does not make sense. Also, unpublished container folders are another good example. Maybe this can be done with a checkbox or prompt so when you mark an article as unpublished you get the choice.

Quote from: Mark at Apr 20, 2007, 09:22 PM

I don’t think all unpublished content should return a "Gone" code. For example, many people write their documents in the unpublished state before they go live. Sending a "Gone" code for a page that has not existed publically yet does not make sense. Also, unpublished container folders are another good example. Maybe this can be done with a checkbox or prompt so when you mark an article as unpublished you get the choice.

I agree with Mark. This can be hit-and-miss; I have many unpublished articles that I want 404 (as of now), but I can see benefit in having the ability to 410. Never used it in the past, but could easily see myself doing so in the future.

I can see benefits to returning statuses other than just 404, and there is definitely room for improvement in the current 404 system.

How I’d like Error Codes to work:

Upon changes to the liveness of a page URL - either the URL is changed or the page is unpublished, the page URL is added to a Unreachable Pages table along with an appropriate action.

When MODx is asked for a page, it looks at the available published documents, then looks at this Unreachable Pages table, and then if no match is found in either does the normal 404 thing.

For each page in the Unreachable Pages table, it should be possible to set the HTTP status code and the page ID to redirect to or display if appropriate.

It should be possible to set blanket rules to match personal preferences. You could set what deleted or hidden pages returned - perhaps you’d prefer them to redirect to the subscription page.

Renames would automatically be saved as redirects. Thus, if you post Brittanny-Spears-Hare-Shaved.html, and a bunch of people link to it before you correct it to Britney-Spears-Hair-Shaved.html, this system would automatically keep their links working.

It should also be possible to manually add URLs that never existed in the MODx system but which are common mistakes or have been inadvertently published by mistake. (As an enhancement, perhaps the most common incorrect URLs could be imported from some stats logs).

This system could also be useful for people importing sites from other systems with different URL structures.

If people wanted to tweak the status code of every removed page, they could. Or they could just leave it to do the most logical thing. This sounds very much the MODx way.

This idea is aimed at people getting to the right page as logically as possible, but this should have positive SEO implications, particularly as it enables people to optimise existing page names for search engines without screwing up existing links.

(And there’s probably a better name for it than Unreachable Pages).

Wow, sounds like a great add-on (or set of add-ons, maybe a module?). However, I don’t see any real benefit to providing this as part of the MODx core, other than enabling it at the framework level, which really already exists somewhat (i.e. sendRedirect() and sendForward() methods already allow you to set the response codes however you want).

Agreed, this is potentially bloating the db with a history of every page rename and makes more sense as an optional add-on.

It would need the following hooks:

for basic operation:
* onNotFoundMatchingPublishedDocument (with requested url passed through)

to automatically add previous urls:
* onDocumentNameChanged (with old page name and current ID passed through)
* onDocumentPublishStatusChanged (with old and new statuses and current ID passed through)

to automatically add previous url formats:
* onPrefixChanged (with old and new prefixes passed through)
* onSuffixChanged (with old and new suffixes passed through)
and a bunch of other things to monitor changes to the friendly URL structure, but that might be overdoing things a bit.

My hesitation with that approach is that our event model is already huge, and the more events we add to it, the slower it’s going to get. In a sense, reliance on events will be reduced in 0.9.7 by way of the object-oriented ability to extend the core classes. For example, I could create an extension of modResource that simply adds this functionality by overriding the save() function, addressing the issue at the absolute core level of processing, with a minimal of overhead compared with using the event structure:

Just thoughts and I’m not saying the events model is not the proper place for some of this, but we have to start thinking about where they are most appropriate to use.

I’d be leaning towards core extension rather than creating events since this kind of functionality is something affects the core, as such the core should be overridden/extended.

Thanks for the info Jason! I’m anxiously awaiting a usable version 0.9.7 for testing!