logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: scoreboard is full, not at MaxRequestWorkers.Increase Server
Author
guanggevwu



Joined: 19 Feb 2024
Posts: 4

PostPosted: Tue 20 Feb '24 18:12    Post subject: scoreboard is full, not at MaxRequestWorkers.Increase Server Reply with quote

The system and software information are here:
https://drive.google.com/file/d/1_ghH_vWvXGGAqOe2T6Db12msfrMqm-8j/view?usp=sharing
The settings for mpm_event is:
https://drive.google.com/file/d/1Kj8dqNWa-t9vm5jX4-clR42MS_M3Frtc/view?usp=sharing
As can be seen from the image, there are a lot of PIDs "Stopping; yes (old gen)". In details, every day, it generates 4 PIDs but the old ones seemed not killed.
Since I set the ServerLimit to 32, eventually it should lead to the error in the title. I restarted apache 6 days ago and thus occupied 24 slots/PID (4 PIDs each day * 6 days).
Any solutions?
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 312
Location: UK

PostPosted: Sat 24 Feb '24 14:11    Post subject: Reply with quote

Your post details don't provide any real clues as to why the "old gen" processes have stopped accepting connections. Presume you've checked the Apache error log for clues (noting you're using mod_wsgi). Are the processes blocking on some connection or resource that's timed out, or needed keepalive?

Other than timeout settings, I'd consider changing MaxConnectionPerChild to something non-zero, to cause new child processes to be launched periodically, and see what if anything changes.
Back to top
guanggevwu



Joined: 19 Feb 2024
Posts: 4

PostPosted: Mon 26 Feb '24 17:22    Post subject: Reply with quote

tangent, thanks for looking into my problem.

Your post details don't provide any real clues as to why the "old gen" processes have stopped accepting connections.
I actually don't know much about how apache works, but I checked the error log and somehow at midnight everyday in the error log file:
[Mon Feb 26 00:00:01.049613 2024] [mpm_event:notice] [pid 1021636:tid 140429992023936] AH00493: SIGUSR1 received. Doing graceful restart
Does this cause apache2 to generate new PIDs everyday?I think so.
I Just did some test. I manually did a "systemctl restart apache2" and then kept doing "apachectl -k graceful", the old PIDs were killed successfully by monitoring the server-status page. Howevers, as long as I set up a connection to my website (by visiting the index page and then close the page), "apachectl -k graceful" can't kill all the 4 old PIDs (usually it kills 3 of them).
In summary, it seems the log configuration caused a graceful restart everyday. But the reason why the old PIDs can't be killed is unclear.

Are the processes blocking on some connection or resource that's timed out, or needed keepalive?
As can be seen from the image links, I have 4 slots (25 threads each) accepting requests, and most of them are idle (the one that is not idle I suppose is the server-status page itself). This means I don't have a lot of visitors and there should not be any connections that refuse to close.

Other than timeout settings, I'd consider changing MaxConnectionPerChild to something non-zero, to cause new child processes to be launched periodically, and see what if anything changes.
Can you please explain more about the timeout settings? Is there a way Apache can hard stop an old PID automatically?
Is there a suggested MaxConnectionPerChild value?
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 312
Location: UK

PostPosted: Mon 26 Feb '24 22:18    Post subject: Reply with quote

Your Apache log entry showing a graceful restart at midnight, suggests someone's set up a cron task to do this, and if so hopefully they've added a comment to the cron entry explaining why. One possible reason for this might be to force Apache to pick up WSGI application code changes that would otherwise be cached in process memory. This depends on your various Apache mod_wsgi configuration settings, which you've not disclosed.

Either way, your simple index page access test rather confirms the Apache process is holding onto a resource, which is preventing the graceful restart from killing the process.

Since your status page is showing there are no open/pending network connections, I'd consider this to be WSGI resource related, so for me that's where I'd start looking further.

Other things I'd consider:

a) Raise the Apache loglevel to debug or higher and repeat your index page graceful restart test.
Code:
LogLevel debug

b) Check the memory use of your stuck Apache processes to see if there's anything untoward going on.
Code:
# ps uax | egrep 'VSZ|apache2'

c) If acceptable to your service, consider going round the problem by changing the graceful restart to a hard restart in the wee small hours?
Back to top
guanggevwu



Joined: 19 Feb 2024
Posts: 4

PostPosted: Wed 28 Feb '24 19:34    Post subject: Reply with quote

Thanks, tangent. It seems this post has the same problem as mine.
https://github.com/GrahamDumpleton/mod_wsgi/issues/383#issuecomment-449555949
First, the reason the "graceful" happens in midnight is because of logrotate.
Second, the reason why the "graceful stop" doesn't work correctly is because the information can't be passed to the WSGIDaemonProcess.
However, I don't quite know how to fix the problem. Fixing either of the two issues should solve my problem. For the second issue, I don't know how to set the "display-name" even I read the documentation (https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html).
For the first issue, I don't think I have made changes to use an external log script. Maybe it is the Ubuntu default. Is there a way I can use Apache's own rotatelog?
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 312
Location: UK

PostPosted: Fri 01 Mar '24 22:37    Post subject: Reply with quote

I can't help with the WSGI side of things, but you could reconfigure the logging setup on Ubuntu to avoid the graceful restart triggered through /etc/logrotate.d/apache2.

With some effort you could change the Apache configuration to using the rotatelogs utility (or third party cronolog), both of which will rotate log files automatically based date and time, e.g.

Code:
ErrorLog  "| /usr/bin/rotatelogs ${APACHE_LOG_DIR}/error_%Y-%m-%d.log 86400"
CustomLog "| /usr/bin/rotatelogs ${APACHE_LOG_DIR}/access_%Y-%m-%d.log 86400" combined

You can then comment out the "invoke-rc.d apache2 reload" line in /etc/logrotate.d/apache2, to prevent the restart attempt. You'll also need to change the 'rotate' to 'maxage', since log rotation is now managed elsewhere.

Historically, I personally favour the cronolog utility (which you'd have to install with "apt install cronolog"), since it lets you use a symlink (soft or hard) to reference the latest log file, e.g. for error.log or access.log
Code:
ErrorLog  "| /usr/bin/cronolog --hardlink ${APACHE_LOG_DIR}/error.log ${APACHE_LOG_DIR}/error_%Y-%m-%d.log"
CustomLog "| /usr/bin/cronolog --hardlink ${APACHE_LOG_DIR}/access.log ${APACHE_LOG_DIR}/access_%Y-%m-%d.log" combined

I'd recommend setting this up on a test system, until you're happy with the configuration and log file changes.
Back to top
guanggevwu



Joined: 19 Feb 2024
Posts: 4

PostPosted: Wed 06 Mar '24 17:18    Post subject: Reply with quote

After disabling the default logrotate and using rotatelogs, the server didn't try to graceful restart in the midnight. Although the real problem should be in the mod_wsgi side which causes a graceful restart can not be completed, I believe this solved the "scoreboard is full" problem. Thank you again, tangent.
Back to top


Reply to topic   Topic: scoreboard is full, not at MaxRequestWorkers.Increase Server View previous topic :: View next topic
Post new topic   Forum Index -> Apache