Suggestions on how you'd handle 20,000 content documents?

Hello All,
I’m developing a site with MODx and I have a content archive from my client with 20,000+ html content articles inside. This content doesn’t ever need to be updated/edited, just displayed on the front end of the site, and so I don’t need these articles to be actual modx documents as I understand that MODx has a 5000 document limit. The MODx manager will be used as the administrative tool for all the new articles the client creates from now on. I have the site working exactly as I want it right now, but currently only with MODx documents that have been created in the backend...

I’m looking for suggestions as to how people would go about integrating the old content files into the MODx site, where hopefully, the front end is seamless in behavior when switching between content drawn form MODx documents and then from the old html documents.

I imagine that my solution will have to be to dump all of the HTML into a separate table and then modify my search & ditto snippets to draw the content from there as well as the modx tables. I’d then have a page inside modx which will display content from the alternative table based on some url parameters. If this is the case, does anyone have any good suggestions as to how I can continue to use alias path fURLs with the old content?

eg:
www.mysite.com/features/new-content-article
- and -
www.mysite.com/features/old-content-article

rather than:

www.mysite.com/features/oldcontent?&id=29

Thanks in advance for any help!

PS - If you’ve ever built a site with 5000+ docs, could you share your experiences? Thanks!

I would definitely use a separate table and write some sql for sorting and displaying the articles (forget about using Ditto). Your best bet would be to write a module to provide an integrated management facility in the manager, and a snippet to output the articles to the site. Add a rewrite rule to the .htaccess file to take care of the url scheme (the simplest form would be something like /features/article/29/).

Susan has written an article on writing a module using Ajax...
http://www.sottwell.com/database-module.html

I would be inclined to create at least one, maybe four database tables to load the documents into, either by category or by date groupings. Then create a snippet to gather documents from those tables in much the same way Ditto and Reflect do for MODx documents. The tables could actually just contain metadata about the documents, such as document title, date and author (I presume you would be able to get that data from somewhere, even if you had to write a script to parse the documents themselves to pick it out). On piece of metdata would be the path to the actual document.

This only needs to be done once, so I don’t see much need for a whole module to do it.

If you put the files in an organized directory structure where a snippet can retrieve them, you could even use a "directory" snippet to sniff the directory structure and build an "archive" directory page based on that. It won’t change, so it would be worth the bit of time it would take to organize these properly in the first place. With a database with metadata, though, you could offer directories by date, by category, by author, or a mixture.

So you would have an "archives" page with a snippet that lists the groups of archived files by category and/or date or whatever (a perfect application for AJAX), then builds the links to the document display page with the filename requested in the query string. The display page has a snippet to read that GET value and include the file.

A lot depends on the form of these old documents. If they’re HTML documents it would be easy to run a script to break out the <body>...</body> from the rest of it, then you could make up a template with your current header, masthead, and a live menu, and just include the body of the original with the retrieval snippet in the place one normally puts the content tags.

Hey all,
Thanks for these suggestions - I will post back here with my final solution when I get round to it

Thanks again!