We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 24168
    • 45 Posts
    Hi all,

    I have been developing a blog type of site with the following:


    • MODX2.2
    • Articles 1.4.7
    • Caching on
    • DB caching on

    The site has approx 5600 articles using the Articles EXTRA. This seems to be working fine when I have 1500 articles but when I import all 5600 articles the site runs out of memory and generally runs extremely slowly.

    I have set the php.ini memory to 64M and it is still running out of memory or exceeding the 120 second script timeout. I am not doing anything complicated but it seems to be affected by the number of articles and as if it is trying to process ALL of the articles even tho I am just trying to show 10-15 at a time using getResources:

    [[getResources? &parents=`14` &tvFilters=`Featured==` &includeTVs=`1` &includeTVList=`Image,Featured,articlestags` &sortby=`{"publishedon":"DESC"}` &limit=`[[*HomepageArticleCount]]` &tpl=`ArticleRow` &showHidden=`1` ]]


    Strangely, turning off small things like the header and footer and it seems to run better. Increasing the RAM limit and timeout doesn't really help as it runs for minutes and even doesn't render the page. If I could get it to run so that cache kicks in it seems a lot faster (2-4 seconds). It just doesn't seem to build the page to set it in the cache.

    Are they any things I can do to fix this as I am in real trouble if I can't fix this! [ed. note: Qodo last edited this post 14 years, 8 months ago.]
      Stewart
      Freelance Web Designer & Developer:
      http://www.qodo.co.uk

      My extras:
      TwitterX http://modx.com/extras/package/twitterx
      TableOfContentsX http://modx.com/extras/package/tableofcontentsx
      • 1343 ☆ A M B ☆
      • 2,213 Posts
      Hello,

      Can you tell us more about the server, is this a VPS/Dedicated server? When you increased the memory/timeout did you verify they took affect, or did you change them server wide?
        Patrick | Server Wrangler
        About Me: Website | Tweets |  MODX Hosting
        • 24168
        • 45 Posts
        Thanks for the reply.

        It's on my local desktop machine quad core 4gb ram. I have also tried it on my VPS which is 4 cores 2gb ram but similar result.

        The changes to ram and timeout definitely took effect but didn't help (ie they ran for 120 seconds and/or used up all of the 256m ram etc).

        It seems that if I turn of certain elements of the page template it works. Things like Wayfinder etc gradually make the pages load within 10 seconds.

        Most pages have a few calls to getResources similar to above, Wayfinder call and tagLister. If I take some of them off it starts to work better - it almost seems like it is hitting some sort of limitation.

        Any help appreciated! [ed. note: Qodo last edited this post 14 years, 8 months ago.]
          Stewart
          Freelance Web Designer & Developer:
          http://www.qodo.co.uk

          My extras:
          TwitterX http://modx.com/extras/package/twitterx
          TableOfContentsX http://modx.com/extras/package/tableofcontentsx
          • 24168
          • 45 Posts
          Just imported 3000 articles instead of the 5000 and it still hangs... so strange!!
            Stewart
            Freelance Web Designer & Developer:
            http://www.qodo.co.uk

            My extras:
            TwitterX http://modx.com/extras/package/twitterx
            TableOfContentsX http://modx.com/extras/package/tableofcontentsx
            • 1343 ☆ A M B ☆
            • 2,213 Posts
            Does everything work without articles in play? What if any calls are uncached? Did you try adding &depth=`2` to the getresource call(s) to see if that helps any?

            Also, I suspect we know where the problem is (getresource call for articles), but can you try using http://www.jasoncoward.com/technology/2010/10/the-executioner-snippet.html on it to see if that sheds any more light on the problem?

            Do the other pages on the site work, or is everything affected? Last up, could you share one of the affected templates?
              Patrick | Server Wrangler
              About Me: Website | Tweets |  MODX Hosting
              • 24168
              • 45 Posts
              Thanks for the reply again.

              Everything is very fast with < 1000 articles. Almost all of the calls are cached and once the page renders the cached version seems very fast too.

              Even the Articles homepage (/articles) loads VERY slowly when it has 6000 articles. It almost seems like MODX is loading ALL of the articles from the database even though I only want to show 10 articles at a time?

              The non getResources and Articles pages (ie a contact page) loads within a second or two so it only seems to be the heavy pages that are failing.

              Homepage template (wayfinder removed to help speed)

              [[$Header? &class=`homepage` ]]
              <div id="articles_feature_container">
              <ul id="articles_feature">
              [[getResources? &parents=`14` &tvFilters=`Featured==yes` &includeTVs=`1` &includeTVList=`Image` &sortby=`{"publishedon":"DESC"}` &limit=`[[*HomepageFeaturedCount]]` &tpl=`FeaturedArticle` &showHidden=`1` ]]
              </ul>
              </div>
              <div id="left">
                <div id="main">
                  [[*content]]
                  [[getResources? &parents=`14` &tvFilters=`Featured==` &includeTVs=`1` &includeTVList=`Image,Featured,articlestags` &sortby=`{"publishedon":"DESC"}` &limit=`[[*HomepageArticleCount]]` &tpl=`ArticleRow` &showHidden=`1` ]]
                  <p id="viewall"><a class="button" href="[[~14]]">View all articles</a></p>
                </div>
              </div>
              <div id="right">
                [[$ad_right]]
                [[$RightColTabs]]
                [[$EmailSubscribe]]
                [[$AddressBox]]
              </div>
              [[$Footer]]


              The [[$RightColTabs]] Chunk above has 3 snippet calls.

              [[getResources? &parents=`14` &includeTVs=`1` &includeTVList=`Image` &sortby=`{"publishedon":"DESC"}` &limit=`5` &tpl=`ArticleRowMini` &showHidden=`1` ]
              [[!getFeed? &url=`removedforprivacy` &tpl=`twitterFeedTpl` &limit=`5` ]]
              [[QuipLatestComments? &limit=`5` &tpl=`ArticleRowComment` ]]


              I should also mention the ArticleRow chunk has a phpthumbof call:

              <div class="post article[[+idx]]">
                  <h2 class="title"><a href="[[~[[+id]]]]">[[+pagetitle]]</a></h2>
                  <p class="post-info">
                    <span class="calendar">[[+publishedon:strtotime:date=`%b %d, %Y`]]</span>
                    <span class="tags">[[tolinks? &items=`[[+tv.articlestags]]` &target=`[[*id]]` &useTagsFurl=`1` ]]</span>
                    <a href="[[~[[+id]]]]#comments" class="comments">Comments ([[!QuipCount? &thread=`article-b14-[[+id]]`]])</a>
                 </p>
                  <img class="entry_img" src="[[+tv.Image:isnot=``:then=`[[phpthumbof? &input=`/assets/files/[[+tv.Image]]` &options=`&w=110&h=110&zc=1`]]`:else=`/images/image110x110.png` ]]" alt="[[+pagetitle]]" />
                  <div class="entry">
                    <p>[[+introtext:default=`[[+content:ellipsis=`400`]]`]]</p>
                  </div>
                  <p class="postmeta">
                    <a href="[[~[[+id]]]]" class="readmore button">Read more</a>
                  </p>
              </div>


              This page loads reasonably with 2000 articles (11 seconds)
              It takes 40+ seconds with 6000 articles

              The other pages like articles/tags/SampleTag fails almost everytime with 6000 articles.

              I hope this helps and I will read that link to see if I can figure anything out. [ed. note: Qodo last edited this post 14 years, 8 months ago.]
                Stewart
                Freelance Web Designer & Developer:
                http://www.qodo.co.uk

                My extras:
                TwitterX http://modx.com/extras/package/twitterx
                TableOfContentsX http://modx.com/extras/package/tableofcontentsx
                • 24168
                • 45 Posts
                Does Articles use getResources when it loads a page like /articles/tags/SampleTag this would explain why it is slow with similar calls.
                  Stewart
                  Freelance Web Designer & Developer:
                  http://www.qodo.co.uk

                  My extras:
                  TwitterX http://modx.com/extras/package/twitterx
                  TableOfContentsX http://modx.com/extras/package/tableofcontentsx
                  • 1343 ☆ A M B ☆
                  • 2,213 Posts
                  I suspect you are hitting resource wall, granted at the moment I don't know what to recommend as a solution. One of our clients will be undergoing this same transition soon, granted they only have 3000 resources at the moment.

                  You might try simplifying your call, for example use:

                  [[getResources? &parents=`14` &tvFilters=`Featured==` &sortby=`{"publishedon":"DESC"}` &limit=`10` &tpl=`ArticleRow` &showHidden=`1` &depth=`1` ]]


                  Sure, the TVs aren't going to be factored in, but it will give us a better idea if the problem is due to the resource count or attempting to process all of the information. At least that's the hope.

                  Articles is mainly a revised frontend for managing blogs and the likes, so your resources are still in the main content tree, and the normal extras like taglister are used to power everything.

                  There is still more we can try, but I suspect you will need to move the articles to their own table and use custom snippets to pull them into the main site, rowboat for example could replace getresources. Let's see if the above call works any better or someone else has an idea first.
                    Patrick | Server Wrangler
                    About Me: Website | Tweets |  MODX Hosting
                    • 24168
                    • 45 Posts
                    Thanks for the reply again.

                    With the TVs
                    MySQL: 0.0000 s , 0 request(s), PHP: 2.6398 s , total: 2.6398 s , document retrieved from database

                    Without TVs
                    MySQL: 0.0000 s , 0 request(s), PHP: 14.5698 s , total: 14.5698 s , document retrieved from database

                    I suspect this is DB caching that I have been playing around with.... but so strange.

                    Using a custom table - wouldn't that mean the articles are outside of MODX and I wouldn't be able to edit them? I really wanted to use MODX as I've been using it for ages and love it...
                      Stewart
                      Freelance Web Designer & Developer:
                      http://www.qodo.co.uk

                      My extras:
                      TwitterX http://modx.com/extras/package/twitterx
                      TableOfContentsX http://modx.com/extras/package/tableofcontentsx
                      • 18373 ☆ A M B ☆
                      • 3,141 Posts
                      I wonder if it could have anything to do with the archives and tags..

                      It has to go through all resources before it can display that, and it would make sense for that to occur on the first page load. Even if you don't have the tags or archives on the page, the Articles addon still processes it. What's worst is that it most likely fetches all that on every resources' first page load.. ie every individual article.

                      Same for the tagging: tagLister would be retrieving all tags on all resources' first load.

                      It would be better if it would store the output (or at least the retrieved data) to the cache on the first load to allow speeding up the rest.. even better would be if it'd retrieve all that data when saving a resource, and not when the first site visitor hits.

                      I think that, thanks to your large scale blog, we've uncovered some performance based flaws in those snippets.

                      If you're not using archives or tagging (you do seem to use tagging from your last post), you could try ruling those out by temporarily renaming their snippets or uninstalling their packages completely. I don't think Articles has an option to not use archiving/tagging yet.
                        Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

                        Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.