Apache Benchmarking Issue?

135 Posts

Eleventeen Reply #1, 15 years, 9 months ago

Hey all! I’m getting ready to launch my new MODx-ified website which normally receives a moderate amount of traffic. I decided to benchmark the main page with AB, just to see how it compares to the current incarnation of my site. I was expecting a reasonable decrease in performance because of increased database usage, additional PHP execution, and simply the overhead of using a CMS to manage my content. But in my case when running AB and comparing the MODx-ified site to my current site (some PHP and DB content), performance is decreased by 75-80%!

AB was run with a concurrency of 5, and a total of 3000 requests.

MODx Site:

Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Finished 3000 requests


Server Software:        Apache
Server Hostname:        <domain.tld>
Server Port:            80

Document Path:          /
Document Length:        21459 bytes

Concurrency Level:      5
Time taken for tests:   45.908146 seconds
Complete requests:      3000
Failed requests:        47
   (Connect: 0, Length: 47, Exceptions: 0)
Write errors:           0
Total transferred:      65062087 bytes
HTML transferred:       63861667 bytes
Requests per second:    65.35 [#/sec] (mean)
Time per request:       76.514 [ms] (mean)
Time per request:       15.303 [ms] (mean, across all concurrent requests)
Transfer rate:          1384.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    37   75  34.7     73     770
Waiting:        0   70  35.3     68     765
Total:         37   75  34.7     73     770

Percentage of the requests served within a certain time (ms)
  50%     73
  66%     74
  75%     75
  80%     75
  90%     78
  95%     81
  98%     85
  99%    148
 100%    770 (longest request)

Current (non-MODx) Site:

Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Finished 3000 requests


Server Software:        Apache
Server Hostname:        <domain.tld>
Server Port:            80

Document Path:          /
Document Length:        90667 bytes

Concurrency Level:      5
Time taken for tests:   8.835535 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      272451000 bytes
HTML transferred:       272001000 bytes
Requests per second:    339.54 [#/sec] (mean)
Time per request:       14.726 [ms] (mean)
Time per request:       2.945 [ms] (mean, across all concurrent requests)
Transfer rate:          30113.06 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:     7   14   2.9     14      62
Waiting:        1    4   1.7      4      20
Total:          7   14   2.9     14      62

Percentage of the requests served within a certain time (ms)
  50%     14
  66%     15
  75%     16
  80%     16
  90%     17
  95%     19
  98%     21
  99%     22
 100%     62 (longest request)

Now, my MODx site has around 1,300 total Resources, so I tested a completely vanilla install. No addons, no new resources - just the base traditional 2.0.6-pl release.

Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Finished 3000 requests


Server Software:        Apache
Server Hostname:        <domain.tld>
Server Port:            80

Document Path:          /
Document Length:        131 bytes

Concurrency Level:      5
Time taken for tests:   25.606648 seconds
Complete requests:      3000
Failed requests:        12
   (Connect: 0, Length: 12, Exceptions: 0)
Write errors:           0
Total transferred:      1664316 bytes
HTML transferred:       391428 bytes
Requests per second:    117.16 [#/sec] (mean)
Time per request:       42.678 [ms] (mean)
Time per request:       8.536 [ms] (mean, across all concurrent requests)
Transfer rate:          63.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    32   42  10.6     41     236
Waiting:        0   41   9.7     41     236
Total:         32   42  10.6     41     236

Percentage of the requests served within a certain time (ms)
  50%     41
  66%     42
  75%     42
  80%     43
  90%     44
  95%     46
  98%     49
  99%     53
 100%    236 (longest request)

As you can see, the vanilla MODx install fared better than my site loaded with content, but it’s the performance is still 33% of my current mostly-static site.

I’m concerned that with the amount of traffic my site gets, and the results of these benchmarks, that the performance of my site is going to be decreased considerably. I’ve cached all resources and snippet calls (well, most - except for things that absolutely must be uncached, which are very few - only one on my main page).

Server stats:

PHP 5.3.4
MySQL 5.0.91
Apache 2.2.17
APC 3.1.6
4GB RAM
2x Xeon Harpertown CPUs @ 2.66GHz

One other thing - when running the benchmark on either MODx site, I’m seeing failed requests by AB. Those requests are accompanied by the following error in the apache error_log:

[notice] child pid 22736 exit signal Bus error (7)

And a number of these messages show up in the MODx error log:

No class specified for loadClass

I wonder if MODx could be crashing under high loads?

1,336 Posts

shamblett Reply #2, 15 years, 9 months ago

Interesting, could you test again with APC turned off(apc.enabled = off I think) and see if you still get the crashes?

Use MODx, or the cat gets it!

MODX Staff
10,725 Posts

opengeek Reply #3, 15 years, 9 months ago

Sounds like you have an environment problem, possibly related to a bug in your site definition. And FWIW, no framework is going to deliver your content as fast as plain PHP files, period. With proper attention to caching and efficient code design though, you can definitely create a scalable solution with MODx.

I would definitely not turn APC off or performance will completely degrade compared to what you are seeing already. You need to find out what is causing the Apache crash; that is usually indicative of a problem in the PHP configuration (i.e. compiled against wrong libraries for database or other extensions) AFAIK.

Jason Coward
Chief Architect @ MODX
http://www.jasoncoward.com | http://twitter.com/drumshaman | https://github.com/opengeek

1,336 Posts

shamblett Reply #4, 15 years, 9 months ago

I would definitely not turn APC off or performance will completely degrade ..

Yes, sorry I meant only to run the test again to see if it crashes, not to leave it off.

Use MODx, or the cat gets it!

135 Posts

Eleventeen Reply #5, 15 years, 9 months ago

Quote from: shamblett at Dec 23, 2010, 02:41 AM

Interesting, could you test again with APC turned off(apc.enabled = off I think) and see if you still get the crashes?

That was actually one of the first things I attempted. I know that xCache and EA both have issues with PHP 5.3, and I figured something similar might be going on with APC so I disabled it and re-ran the benchmarks with similar results (same errors/crashes, further degraded performance). I should have included those results last night (just as a comparison with caching vs non-caching), but it was late. tongue

Sounds like you have an environment problem, possibly related to a bug in your site definition. And FWIW, no framework is going to deliver your content as fast as plain PHP files, period. With proper attention to caching and efficient code design though, you can definitely create a scalable solution with MODx.

I was expecting a reasonable speed decrease over my current static-ish site, but an 80% decrease seems crazy. I remember benchmarking Evo back in the 095 days and ended up with results almost on-par with a static site. That was a slower server though, completely different setup and probably close to 4 years ago.

Any suggestions on where to look first if recompiling Apache doesn’t help?

Thanks guys!

MODX Staff
10,725 Posts

opengeek Reply #6, 15 years, 9 months ago

Quote from: Ricjustsaid at Dec 23, 2010, 04:16 PM

Sounds like you have an environment problem, possibly related to a bug in your site definition. And FWIW, no framework is going to deliver your content as fast as plain PHP files, period. With proper attention to caching and efficient code design though, you can definitely create a scalable solution with MODx.
I was expecting a reasonable speed decrease over my current static-ish site, but an 80% decrease seems crazy. I remember benchmarking Evo back in the 095 days and ended up with results almost on-par with a static site. That was a slower server though, completely different setup and probably close to 4 years ago.

Any suggestions on where to look first if recompiling Apache doesn’t help?

It definitely should not be 80%; this is obviously a side effect of whatever is causing the crash.

I suggest reviewing what Snippets and Plugins you have running in the site (maybe disable some until you can’t reproduce the crash) and making sure the PHP compilation has all the proper dependencies. Finding out why Apache is crashing is going to be the only way to solve this. Can you reproduce the crash by just visiting a specific Resource in the site or not?

Jason Coward
Chief Architect @ MODX
http://www.jasoncoward.com | http://twitter.com/drumshaman | https://github.com/opengeek

135 Posts

Eleventeen Reply #7, 15 years, 9 months ago

Quote from: OpenGeek at Dec 23, 2010, 04:26 PM

It definitely should not be 80%; this is obviously a side effect of whatever is causing the crash.

I suggest reviewing what Snippets and Plugins you have running in the site (maybe disable some until you can’t reproduce the crash) and making sure the PHP compilation has all the proper dependencies. Finding out why Apache is crashing is going to be the only way to solve this. Can you reproduce the crash by just visiting a specific Resource in the site or not?

The odd thing is that it even happens on a fresh install of MODx, on the blank homepage included in the base install. There aren’t any snippets or plugins installed or present on the site, so that takes any 3rd-party code out of the equation.

The crash only seems to occur when MODx is under high load. I’ve never seen the error when it was just me browsing the site. Also, when running the benchmark on my current site, everything seems fine - no apache errors, crashes, etc.

I’ll try recompiling Apache on off-hours tonight and re-run the benchmarks to see if that has anything to do with it (fingers crossed!).

135 Posts

Eleventeen Reply #8, 15 years, 9 months ago

Okay, I recompiled Apache and re-ran the benchmark on the clean Revo site, and ended up with a similar result as before:

Benchmarking <domain.tld> (be patient)
Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Finished 3000 requests


Server Software:        Apache
Server Hostname:        <domain.tld>
Server Port:            80

Document Path:          /
Document Length:        131 bytes

Concurrency Level:      5
Time taken for tests:   27.200434 seconds
Complete requests:      3000
Failed requests:        9
   (Connect: 0, Length: 9, Exceptions: 0)
Write errors:           0
Total transferred:      1665987 bytes
HTML transferred:       391821 bytes
Requests per second:    110.29 [#/sec] (mean)
Time per request:       45.334 [ms] (mean)
Time per request:       9.067 [ms] (mean, across all concurrent requests)
Transfer rate:          59.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:    32   44  32.6     41     615
Waiting:        0   44  32.6     40     614
Total:         32   44  32.6     41     615

Percentage of the requests served within a certain time (ms)
  50%     41
  66%     41
  75%     42
  80%     42
  90%     44
  95%     46
  98%     70
  99%    161
 100%    615 (longest request)

Just out of curiosity, I installed a fresh copy of Evo 1.0.4 (no addons or anything, only the main home resource), ran the benchmark and got the following results:

Benchmarking <domain.tld> (be patient)
Completed 300 requests
Completed 600 requests
Completed 900 requests
Completed 1200 requests
Completed 1500 requests
Completed 1800 requests
Completed 2100 requests
Completed 2400 requests
Completed 2700 requests
Finished 3000 requests


Server Software:        Apache
Server Hostname:        <domain.tld>
Server Port:            80

Document Path:          /
Document Length:        408 bytes

Concurrency Level:      5
Time taken for tests:   16.47080 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      2457000 bytes
HTML transferred:       1224000 bytes
Requests per second:    186.95 [#/sec] (mean)
Time per request:       26.745 [ms] (mean)
Time per request:       5.349 [ms] (mean, across all concurrent requests)
Transfer rate:          149.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    25   26   1.7     26      50
Waiting:       22   25   1.6     25      37
Total:         25   26   1.7     26      50

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     26
  75%     27
  80%     27
  90%     28
  95%     29
  98%     30
  99%     31
 100%     50 (longest request)

Better performance than Revo, and no errors reported in the Apache error_log. It looks like Revo *is* causing a crash under heavy load? I wish I had another server to test this on...

1,336 Posts

shamblett Reply #9, 15 years, 9 months ago

I’ll do this on one of my servers over the hols, could you please post your ab command line so we are using the same thing, this server has PHP 5.3.3, Apache 2.2.17, no APC, 8 cores/6g ram and is a 64 bit machine so its fairly close seeing as we’ve ruled out APC from the tests above.

I can turn on PHP profiling on this if needed and run a debug version of Apache(this is good gen for child crashes http://httpd.apache.org/dev/debugging.html#crashes) to see if it sheds any light on what’s causing this.

Use MODx, or the cat gets it!

135 Posts

Eleventeen Reply #10, 15 years, 9 months ago

That’s a nice machine! grin

The command I used was:

ab -c 5 -n 3000 [site URL]

Merry Christmas all!