logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in  RSS Apache Lounge  


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.


Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache 2.4 Server ran out of threads to serve requests
Author
Sunburst



Joined: 09 Mar 2012
Posts: 20

PostPosted: Sat 31 Mar '12 10:18    Post subject: Apache 2.4 Server ran out of threads to serve requests Reply with quote

Hi all

I got this error each day for a week:
Quote:
Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting


So I kept increasing ThreadsPerChild until I came to 1024 and this morning my apache server went offline and httpd.exe had taken 99% of the CPU making the whole server run extremely slow (of course) and no sites where working. I restarted the Apache service and the sites were running again.

I have been working some years with the Apache server but I would not call myself an expert on this field. I really need your advice Smile

The server:
    Dedicated Windows 2008 R2 (64 bit) server
    1 Gbit internet connection
    8 CPU cores
    32 RAM
    4 HDDs in RAID10 - operation system + hosting files for 50 Joomla sites, some of the sites are big
    2 SSDs in RAID1 - hosting the MySQL databases and Apache disk cache (that I still have to learn more about)

Apache + PHP:
    Apache/2.4.1 (Win32) mod_fcgid/2.3.7
    php-5.4.0-nts-Win32-VC9-x86


My Apache config file:
Removed

My PHP config file:
Removed

Apache log:
Removed


I think that Apache went offline at 06:55:26 (perhaps was too slowly in the hours before) and I restarted Apache at 08:37:32.

The log copy/paste starts at 01:03:47 where the problem keeps building up and... this is night time so there should be only a few people browsing the sites.

While pasting this in I saw that PHP's max_execution_time and max_input_time were way too high. I must have used the values from my dev computer. So I set them down to (reflected in the php.ini copy/paste above):

Code:
max_execution_time = 300
max_input_time = 300


Could the long PHP timeout get apache to go offline as it did? The Apache time has been on 300 all the time.

Another error is that the browsers' loading icon (hourglass) keeps running even though the page seems to be fully loaded (and can be used). It does this on some of sites and in all browser types, but worst in FF. The pages are loaded very quickly but something seems to be hanging. If I reload the page than the browsers’ loading icon stops and I can browser as I like (until it happens again on another page). It did not do this before I got this new server (and I changed to PHP 5.4 and started using the Apache version from Apache Lounge).

Thank you very much for all your advice Smile

Admin note: conf's and logs removed, see Forum Rules in the left menu.
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 2857
Location: Hilversum, NL, EU

PostPosted: Sat 31 Mar '12 14:49    Post subject: Reply with quote

Worth to try Apache 2.2.22. The 2.4 is oh so new, special in the mpm, disk cache area.

Steffen
Back to top
glsmith
Moderator


Joined: 16 Oct 2007
Posts: 2251
Location: Sun Diego, USA

PostPosted: Sat 31 Mar '12 18:26    Post subject: Reply with quote

I happened to see the stuff before it got removed.

I wouldn't say (or saw no evidence) that you are running out of threads. What I saw was that you reached the MaxConnectionsPerChild and Apache closed the old and spawned new child processes. You do this cause of memory leaks so your note said, but how much leaking are you actually seeing?

At 10,000 connections per child it must not take too long to run through them, there is only once child on Windows IIRC.

I saw fcgid errors that I thought were gone in 2.3.7-dev, that might have something to do with what is going on, why child processes are getting forced killed when the mpm spawns a new child process.

I saw something to do with memory set (MaxMemFree) for 1GB with a note about too high? Possible.

I have absolutely no suggestions, just thought I'd summarize what I remember seeing at 2am last night.

Well, I'd probably raise the MaxConnectionsPerChild at least if for no other reason than to lessen the log pollution every time Apache hits it, and spawns a new child. I'd probably leave MaxMemFree at it's default.

Joomla is php, and mod_fcgid is dealing with it, I'm not fcgid savvy so cannot say much on that subject, nor do I remember the settings you had. There are however, suggestions on mod_fcgid settings here in this forum, could probably search them out easily.

There's a performance tuning Apache on Server 2008 Word document in the "Additional" downloads section, you might want to check it out if you haven't already.
Back to top
Sunburst



Joined: 09 Mar 2012
Posts: 20

PostPosted: Sat 31 Mar '12 18:56    Post subject: Reply with quote

Hi Steffen

Thank you for your advice. I will try downgrading if the problem remains.

Hi glsmith

Thank you for your advice. I will lower MaxMemFree and increase MaxConnectionsPerChild and see what happens Smile

I have re-posted my conf + log here: http://apache.pastebin.ca/2134322
Back to top
Sunburst



Joined: 09 Mar 2012
Posts: 20

PostPosted: Sat 31 Mar '12 19:03    Post subject: Reply with quote

Here is a screenshot from this morning (my time) when the server was down. Perhaps it can be usefull Smile

Back to top
Sunburst



Joined: 09 Mar 2012
Posts: 20

PostPosted: Sun 01 Apr '12 0:31    Post subject: Reply with quote

Nearly every 10'th minute Apache creates this error in the Windows Server Event Log:

Code:
Faulting application name: php-cgi.exe, version: 5.4.0.0, time stamp: 0x4f4e6d23
Faulting module name: php5.dll, version: 5.4.0.0, time stamp: 0x4f4e7c5f
Exception code: 0xc0000005
Fault offset: 0x0002760b
Faulting process id: 0x2b94
Faulting application start time: 0x01cd0f7edf36b0a4
Faulting application path: C:\Storage\Webserver\Apache1\PHP\php-5.4.0-nts-Win32-VC9-x86\php-cgi.exe
Faulting module path: C:\Storage\Webserver\Apache1\PHP\php-5.4.0-nts-Win32-VC9-x86\php5.dll
Report Id: 17ccb434-7b73-11e1-b61e-00259049bbd5


Can this be of any help?

UPDATE:

Found that error myself. It was a newsletter script that ran every 10'th minute via a batch file. I wounder if this added more to the 100%-cpu-error.


Last edited by Sunburst on Sun 01 Apr '12 9:15; edited 1 time in total
Back to top
minibayit



Joined: 01 Apr 2012
Posts: 2

PostPosted: Sun 01 Apr '12 9:13    Post subject: Reply with quote

I know this thread has been resolved, but why has no-one suggested "server-status" and "server-info".

This is how I discovered I had a similar issue with a php calendar that the google bot managed to crawl till the year -5068 or something!
Back to top
Sunburst



Joined: 09 Mar 2012
Posts: 20

PostPosted: Thu 19 Apr '12 8:29    Post subject: Reply with quote

The problem came back but I finally got it to work by setting the LogLevel to Error so it didn't spend resources building up EXTREMELY BIG log file (a 60 GB log file just for one site).

So I am on to the next problem... this gigantic security hole in my settings: http://www.apachelounge.com/viewtopic.php?t=4683
Back to top


Reply to topic   Topic: Apache 2.4 Server ran out of threads to serve requests View previous topic :: View next topic
Post new topic   Forum Index -> Apache