Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Need some help advice [Stress testing & Performance] Apa |
Page Previous 1, 2 |
| Author |
|
dke
Joined: 13 Jul 2007
Posts: 61
Location: sweden
|
 Posted: Fri 24 Aug '07 15:44 Post subject: Re: Coppermine performance test |
 |
|
| tdonovan wrote: | Your posts made me curious, so I downloaded Coppermine Gallery version 1.4.12 and gave it a try with Apache 2.2.4 (from Apache Lounge), PHP 5.2.2 via mod_fcgid, and MySQL 5.0.46 on the same machine.
My Windows 2000 machine (3.4ghz 2gb pentium) is roughly similar to yours.
Using XDebug shows that most of the time is spent executing PHP code, so an optimizer like eAcelerator is definitely warranted. I used XCache which is similar to eAccelerator, and I got a similar performance increase. It increased my throughput from 10 to 30 pages/sec using Apache Bench (ab.exe) against /coppermine/index.php.
It appears that ab is not a very fair measurement tool for an application like Coppermine, where users select images from thumbnails and can vote on various images. There are two reasons:1. Using ab causes every request to start a new Coppermine user session because ab doesn't retain cookies.
2. No real users would make a thumbnail selection or choose a picture rating in milliseconds. Instead, I recorded a script using the old, free (but still very useful) Web Application Stress Tool. I recorded a script (incluing browser pauses) as I wandered through the app as fast as I could - selecting thumbnails in about 1/2 second and voting immediatialy on pictures. Faster than any real user would go - but still more realistic than hammering the app with ab. I also configured WAST to add a random 5-50ms pause to each request, so the test would simulate users going at slightly different speeds.
I found that I could support 100 such fast-acting users comfortably, with all response times below 1 sec. CPU use would occasionally get near 100%, but the whole machine (and Coppermine) stayed very responsive throughout the test.
FYI - here are two other tuning things I tried. They helped a little - but not really enough to bother with unless you want to squeeze every last bit of performance out of Coppermine:* Changed the MySQL storage engine for the session table (cpg1411_sessions) to an in-memory table and added an index to speed up the search for expired sessions: | Quote: | mysql> DELETE FROM cpg1411_sessions;
mysql> ALTER TABLE cpg1411_sessions ENGINE MEMORY;
mysql> ALTER TABLE cpg1411_sessions ADD INDEX (time); |
The disadvantage to this is that Coppermine user sessions won't be retained over a MySQL shutdown and restart.
* Altered coppermine\include\functions.inc.php line 181 to call mysql_pconnect instead of mysql_connect. This makes Coppermine re-use persistent connections instead of creating a MySQL connection for each request. Probably helpful if your MySQL is on another machine, but it didn't do a lot for me. It did cut way down on the Connections shown in mysql_info.php.
Hope this stuff is useful (or if not useful - at least interesting) to you.
It's still a guess how many "real life" users I could support - but I would guess 200-300 unless all my users had the curious ability to make photographic decisions in fractions of a second 
-tom- |
Hi Tom!
You're post is incredible interesting indeed! I've yet to test your optimization tricks, ill post feedback here once i've tested them!
I'm currently running the 2.2.5 RC build that steffen got in alot of trouble for (damnit i hope you don't shotdown) and i've noticed a great increase in the ping-times delivered by apache on high load. Before (with 2.2.4/2.0.59) my highest ping-time at 100% of requests used was around 4000-25000+, but using 2.2.5 i get a max of 1200-1500.
I've also experimented alot with KeepAlive and i've come to the conclusion that it is needed. I know that i mentioned in a previous thread (should be moved to this one sorry for that) that i felt a performance increase of turning keepalives off, but that was a hoax, it was because alot of the requests that were incoming never got served. KeepAlive is NEEDED!
Im humbled by the time you've taken to help me Tom, clearly ive gained alot of knowledge and performance talking with you!
Thanks. |
|
| Back to top |
|
dke
Joined: 13 Jul 2007
Posts: 61
Location: sweden
|
| Posted: Fri 24 Aug '07 19:01 Post subject: |
|
|
I've tested your changes, and using pconnect seems to have it's disadvantages.. I haven't gotten that much sql experience but i'm getting this error message after using the AB command, my guess is that there is to many connections started to sql?
| Code: | Coppermine critical error:
Unable to connect to database !
MySQL said: Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug |
Browsing around trying to figure out this message, i saw there was a bug with mysql and windows not being able to handle more then 197 connections, but when i do "mysql status;" in that database ive only around 60 connections.
Also i found this:
| Quote: | In our ASP.Net code, we open a database connection and then close it after the use. Even
after closing the connection, that thread was killed on the server automatically and was
in the state "sleep". But for the next database connection within few seconds, the new
thread was being created instead of using the sleeping thread.
that was adding up lot of open (rather sleeping) threads on the server and one fine moment
(after 1390 sleeping threads), the database used to sopt functioning.
This happens in spite of specifying "Pooling=false" so that is the bug in the MySQL ODBC
driver. another problem was MySQL by default keeps all open threads in "sleep" mode for
8-hours.. thats surprisingly too high.
I had to manually reduce the time for which threads remain open from 8-hours to 20 seconds
and my application started functioning smoothly without any errors. |
However it never became clear how to lower this "Sleep thread mode" from 8 hours to 20s.
Feedback using your database optimization, it was clearly worth it, the index handles 2 requests more per/second using ab now. As you say, ab might not be the fairest tool for coppermine, but still i see a improvement so thats a win situation, thanks for that.
| Quote: | | The disadvantage to this is that Coppermine user sessions won't be retained over a MySQL shutdown and restart. |
I guess this only affects users which are actually logged in? I don't allow registrations and have guest on free surf so that shouldn't be a problem for me right?
| Quote: | | I found that I could support 100 such fast-acting users comfortably, with all response times below 1 sec. CPU use would occasionally get near 100%, but the whole machine (and Coppermine) stayed very responsive throughout the test. |
Weird thing is when i emulate a mass-spam with AB i generate 100 cpu instantly, but when i have real users browsing the page, everything goes extreamly slowly, takes up to 2 seconds for a thumbnail to load, and i get 0 cpu usage on both httpd.exe and mysql.exe very strange indeed. |
|
| Back to top |
|
dke
Joined: 13 Jul 2007
Posts: 61
Location: sweden
|
| Posted: Tue 28 Aug '07 11:39 Post subject: |
|
|
I've manage to optimize coppermine really good now, im up to 45 requests/sec when benchmarking localhost/ but this does not generate the issue i have with real users browsing my page.
It's somthing with how many connections (sends) apache can handle simontaneously. I've been looking at the documentation for apache, and i can't find any setting that controls how many sends apache can handle more then "threadsperchild" but i've never reached above 15 actual sends before all sends start getting REALLY slow.
What i can see when using AB binary to benchmark the webpage is that the ping-times at highest load is getting REALLY high, from 99% load (300ms) to 100% load (17 000ms) which is actually 17 real seconds for a respons?
When i USE f_cgi i can get really low ping-times on 100% load, however this is not because fcgi is better, this is because i used the command : DefaultMaxClassProcessCount 2 which means fcgi only handles 2 sends (MAX) at all times, which means i can push it to send with those 2 threads 100% and still get below 200 ms on 100% load. But this does not work in the end, cause if 5 people are browsing and i only have 2 senders, it can get really slow after awhile.
However if i make this "DefaultMaxClassProcessCount" value higher, ill get the exact same thing as i do when using php5apache2.dll.
Another thing to mention is that when the server gets to it's knees throwing out the requests really slow, the server hardware is showing no signs of even small load on cpu/ram. The only load i can see is when there's few people browsing the page (and the page remains responsive) but i never get cpu above 30%.
It's pretty clear that there's something wrong somewhere that i cannot find. I cannot push my hardware to the max, i cannot even push it to 10% of what it can handle with my current configuration. The only way i can get some load on my hardware is to use FCGI with "DefaultMaxClassProcessCount 2" but i've already explained the issue with this and why it helps.
I've been trying to solve this problem for about 3 months now, i cannot get past it, i've tried (on a fast note) win 2k3 server just to see if i see any clear difference instantly, and i saw no difference.
One thing that interest me alot is to modify the TCP/IP Stack for windows, i think it can be some hardcoded thing into all windows kernels how many sends TCP/IP can handle simontaneously?
I'm on the edge here, i know that a webserver with my performance should be able to host what i am hosting, but im starting to think that unix / fbsd is the only way to go, i really would like to skip the pain of learning unix just to be able to host a small site, but if thats what it takes, i might try it, but please if you have any advice i greatly welcome them all.
EDIT UPDATE:
Futher investigations show that my page can become slow by just 1 single user, i dont know how this is, but right now i have 1 ip browsing and the webpage is running extreamly slow. Im starting to think it might be certain browsers that totally wrecks my page ignoring the rulesets of keepalive and such.
I've read about "mod_setenvif.so" and it seems to be the right tool for this job, any recommendations for the setting to use? I'd prefer to ignore any weird browsers only allowing IE 6.0> firefox 1.4> opera (newer versions).
If only there were a serverside module that could limit EACH ip adress to the maximum of 2 requests simontaneously.
Last edited by dke on Thu 30 Aug '07 19:17; edited 1 time in total |
|
| Back to top |
|
tdonovan
Moderator
Joined: 17 Dec 2005
Posts: 611
Location: Milford, MA, USA
|
| Posted: Tue 28 Aug '07 16:30 Post subject: |
|
|
re: "i can't find any setting that controls how many sends apache can handle more then "threadsperchild" "
That's not likely to be your bottleneck. Apache on Windows has no other limit on concurrency, and is capable (with the default settings) of having 250 network responses underway simultaneously.
I suspect your problem is now something outside of Apache.
With WinXP, if you have the "Home" or "Pro" edition you may be limited to 5 or 10 external connections (see Inbound connections limit in Windows XP.
This should definitely not be a problem with Win2003 Server however. AFAIK the "Server" editions allow unlimited net connections.
If your network connection is DSL, that is notoriously asymmetrical. It can have very good download speeds but very poor upload (i.e. file-serving) speeds.
Just a few thoughts...
-tom- |
|
| Back to top |
|
dke
Joined: 13 Jul 2007
Posts: 61
Location: sweden
|
| Posted: Tue 28 Aug '07 20:15 Post subject: |
|
|
| Code: | I suspect your problem is now something outside of Apache.
With WinXP, if you have the "Home" or "Pro" edition you may be limited to 5 or 10 external connections (see Inbound connections limit in Windows XP.
This should definitely not be a problem with Win2003 Server however. AFAIK the "Server" editions allow unlimited net connections. |
Very possible, however i noticed this issue early on my troubleshot and fixed it. I was getting tons of reports "No more connections can be made to this remote computer at this time because there are already as many connections as the computer can accept." in the event log, but i fixed it by modifying the tcpip.dll with some program, it never changed the issue though (i was trying to solve the same issue back then as now).
The really weird thing i would like to add is that if i RDP on my server hosting the page, and try to surf it through http://localhost/ i get the exact same result as if i was surfing from the outside, everything runs slow, can it still be a limitation in windows xp? |
|
| Back to top |
|
dke
Joined: 13 Jul 2007
Posts: 61
Location: sweden
|
| Posted: Wed 29 Aug '07 0:02 Post subject: |
|
|
#### PROBLEM SOLVED ####
WindowsXP clearly was the issue, installed 2003 server and everything is running VERY smooth now!
Thanks for all the optimization tricks! |
|
| Back to top |
|
|
|
|
|
|
