Search Development

I was thinking last night about the best way to make our site search better (i.e. including tvs) and thought of a new way to do it. What if we created a search index table that contained the docid and the generated output of the document as a field in the table. This would be generated everytime a page is updated in the manager (maybe an option to toggle this) or by clicking on a refresh search index button in the manager. This would allow for tv searching since all of the content would be loaded into the table. I think the seach would be faster too since you wouldn’t have to pull information from multiple tables.

Any ideas suggestions about this?

I don’t think something like this would be to hard to implement but could provide much better searching than we currently have.

Quote from: kylej at Aug 09, 2006, 02:52 PM

I was thinking last night about the best way to make our site search better (i.e. including tvs) and thought of a new way to do it. What if we created a search index table that contained the docid and the generated output of the document as a field in the table. This would be generated everytime a page is updated in the manager (maybe an option to toggle this) or by clicking on a refresh search index button in the manager. This would allow for tv searching since all of the content would be loaded into the table. I think the seach would be faster too since you wouldn’t have to pull information from multiple tables.

Any ideas suggestions about this?

I don’t think something like this would be to hard to implement but could provide much better searching than we currently have.

Kyle, I think your suggestion is an ideal way to offload a more integrated, full-text search index of your site’s actual generated output. Could easily add boolean full-text search capabilities on MySQL platforms that support them (4+). And even uncached documents could cache content in the search index for a period of time at which point it might be automatically refreshed during the postProcess() call. Quick edit would also need to handle updating this index.

I was thinking in addition to a site_content foreign key ref, the text field for the content, we include an expiration timestamp that can optionally be used to update the search index with dynamic page content based on this timestamp during postProcess(). Or alternatively, system configuration settings could used to schedule updates during postProcessing.

Jason, thanks for the input. One question for you, is there a way to check which version of mysql a user is running? Or would I just need a parameter in the search snippet to determine which type of searching to do, incase someone was on mysql < 4?

Here are the plans I have now for implementing this.

Create a new table named search_index with the fields content_id, doc_output, expire_date
Only items marked searchable & published will be included in the index
Add an option to the site settings page in the manager for:
-If page is uncached regenerate search index after? (hours)
Add a button to the manager for refresh search_index, this will go through and update all pages in the index
Add function to save_content.processor.php to insert/update page content in search_index
Add function to postprocess() to update the search_index if the expiration date has passed for uncached pages
Update the AjaxSearch snippet to search the index instead of the content table

Anything I am missing? I will have to look into the quickedit functionality as I have never looked at its code.