Revolution for Extremely Large Site: 800k pages ?

I have been tasked with re-developing a very large us travel site, currently 800,000 pages indexed in Google. I believe I can knock this number down but with some easy math can see it sitting at 600k and growing.

The very base question is can Revolution support this amount of content?

I'm an avid fan of Modx and have developed many Evo/Revo sites to date so I need very little convincing but none have approached this level of content.

Just need that little push (valid push) to get me over my hesitation. Thanks for any replies!

Is it 800k pages, or 20 pages with 800k records of some custom data?

Can you give more information about the structure and deployment of the site? Also, how much traffic?

you are the person i was most hoping to see on this post. Thanks for listening in and perfect questions.

the core of the site can run on 7 pages (templates), but I would anticipate that growing to 15 with the addition of some user-based templates such as profile pages, blog(s) and maybe a few other unforeseen pages. The site targets travel so within those pages are basically events, attractions and businesses with placeholders for news, social media and other fillers. I would guess that 10 TV's per, with some overlap (i.e. address)

being a (currently) U.S. based site the structure (architecture) is state->region->county->city with the content in the cities populating the larger geographic containers.

The site currently runs on the Rackspace Cloud without a hitch with current php/mysql.

During the peak season, the site pulls a little over 300k unique vistors / 700k page views. This does grow year over year and with the improvements that modx would provide, I would expect a decent increase.

Hope this helps, thank you for your time and feel free to ask anything else

It doesn't seem that all your content is straight content, but something that could be put into properly structured custom database tables, rather than being forced into (generally inefficient for this sort of thing) Resources and TVs. You could then either create a Custom Manager Page to manage the data, or a custom resource class to extend the Resource and provide your own view (similar to Articles).

You could put these all in Resources; but that doesn't seem very efficient, as you're describing basically a few views with a bunch of custom data that is interpreted based on the view. I would recommend looking into creating a custom DB model for the data and rendering that way.

Doing via straight resources would put 800k+ ID->alias indexes into the context cache array; from what I know of PHP, it slows down a bit with arrays that big. Resource data is cached per-resource, but still - it has to cache the ID to alias map in one file for easy lookup. And with 800k+ Resources, it would get heavy. Using a single view Resource for similar data would drastically reduce the number of Resources you're using.

Plus, doing the filtering and navigation you're proposing, I could see that getting quite nasty having to circumnavigate that around TVs and such. I'd recommend strongly the custom DB tables route.

Also, you can use memcached and the mysql slave/master replication to help offset the load. Jason can chime in a bit later on this.

I have a question related to this. On the CRC doc it says that CRCs are not recommended for use when you just want to add a few fields to a resource -- that's better done with TVs. But what about when you have a moderately large number of the same type of resource (on the order of a thousand or two), and you want to improve performance? Would a CRC with a few extra fields offer better performance than a template with a few TVs?

i can handle the custom db tables and will check out the CRC doc. Will I be able to take advantage of the modx packages in the same manner as a 'normal' site. One of the benefits for using ModX on this project vs. handcoding was the time savings of not having to develop 'packages'.

Again, thanks for the time!

freejung,

If you're doing some sort of product catalog or something that might use a custom DB table, a CRC can be used to be the "view" or container of that data. You load it normally as a Resource, then (similarly to how Articles does it) override the process method to grab your custom DB table data and present your view that way. Think of it like adding your own sub-application.

You could also, however, do this simply through Snippets on a Resource that call data from the custom DB table for each "view" in your site. That's a simpler solution, unless you really want it prepackaged and to have your own UI for the view Resource.

OK, thanks -- I get the idea. In comparison to just doing it with every item as a resource, the CRC would take the place of the parent resource, and the children (individual items) would be stored in a custom DB table. You use the CRC instead of a CMP to manage add/delete/edit items. Then, instead of using the getResources snippet, you use the processing method of the CRC to display a list of items.

Presumably you could do the same thing with a CMP, a custom DB table, and a snippet.

Is there a performance advantage to either method?

I don't know... the way Articles works is pretty slick. I think it might be worth a bit of extra effort just to learn how to do something like that. [ed. note: esnyder last edited this post 14 years, 9 months ago.]