logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: mod_fcgid latency on high concurrency
Author
nono303



Joined: 20 Dec 2016
Posts: 207
Location: Lille, FR, EU

PostPosted: Wed 20 May '20 12:22    Post subject: mod_fcgid latency on high concurrency Reply with quote

Hi @all!

I open this topic to (re?)start a discussion around the mod_fcgid latency in case of high concurrency.
This subject has already been discussed (cf. https://bz.apache.org/bugzilla/show_bug.cgi?id=53693) with a revert in mod_fcgid 2.3.9
Being faced with the problem (thumb gallery whose images are generated in php via mod_fcgid involving concurrency from several tens of processes without lazy-load) I have randomly latencies of (exactly) one second on 1 to several images.
This is explained in the mod_fcgid code if all the processes are busy:
Code:
apr_sleep (apr_time_from_sec (1));

https://github.com/pagespeed/mod_fcgid/blob/master/modules/fcgid/fcgid_bridge.c#L456

In your point of view, what would be the best way to go around this bottleneck?
    1) Reducing the waiting time apr_sleep (apr_time_from_msec (XX)); maybe with XX externalized as the time to spawn a new process depends on many parameters
    2) Directly force a new process spawn and try to handle request to it more than 2 times (see patch https://www.mail-archive.com/dev@httpd.apache.org/msg55268.html which I am currently testing)
    3) Other ideas? On configuration side?
Many thanks for your advices!
++NoNo
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 207
    Location: Lille, FR, EU

    PostPosted: Wed 20 May '20 13:47    Post subject: Reply with quote

    ...and here is my config:
    Code:
    FcgidInitialEnv PHPRC "C:/serveur/php7"
    <Files ~ (\.php)>
      FcgidWrapper "C:/serveur/php7/php-cgi.exe -c C:/serveur/php7/php.ini" .php
    </Files>
    FcgidInitialEnv TEMP "R:/php-httpd_temp"
    FcgidInitialEnv TMP "R:/php-httpd_temp"
    FcgidInitialEnv windir "C:\WINDOWS"
    FcgidInitialEnv SystemRoot "C:\Windows"
    FcgidInitialEnv SystemDrive "C:"
    FcgidInitialEnv PROCESSOR_ARCHITECTURE "AMD64"
    FcgidInitialEnv PROCESSOR_IDENTIFIER "Intel64 Family 6 Model 42 Stepping 7, GenuineIntel"
    FcgidInitialEnv PROCESSOR_LEVEL "6"
    FcgidInitialEnv PROCESSOR_REVISION "2a07"
    FcgidInitialEnv NUMBER_OF_PROCESSORS 4
    FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 0
    FcgidInitialEnv PHP_FCGI_CHILDREN 0
    FcgidFixPathinfo 1
    FcgidWin32PreventOrphans On
    FcgidMaxRequestsPerProcess 0
    FcgidMaxProcesses 40
    FcgidMaxProcessesPerClass 40
    FcgidMinProcessesPerClass 16
    FcgidIdleScanInterval 1
    FcgidErrorScanInterval 1
    FcgidZombieScanInterval 1
    FcgidIdleTimeout 60
    FcgidProcessLifeTime 60
    FcgidConnectTimeout 1
    FcgidIOTimeout 600
    FcgidBusyTimeout 600
    FcgidBusyScanInterval 600
    FcgidMaxRequestLen 134217728
    FcgidOutputBufferSize 0
    FcgidPassHeader Authorization
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 7374
    Location: Germany, Next to Hamburg

    PostPosted: Wed 20 May '20 16:33    Post subject: Reply with quote

    The re-spawn takes time. I noticed that, too.

    You set FcgidProcessLifeTime to one minute. I set it to 0 (endless). Every minute mod_fcgid kills the spawned php-cgi process.

    Why do you let the module kill the php process so early? That also causes a log flood[1]. It has only been fixed in trunk. But nobody dares a release of 2.3.10. I run the .10 for years without any issues.

    --- edit ---

    My config[2]

    --- /edit ---

    [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=54597
    [2] https://www.apachelounge.com/viewtopic.php?t=2394
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 207
    Location: Lille, FR, EU

    PostPosted: Sun 24 May '20 14:53    Post subject: Reply with quote

    Thanks @James Blond for your feedback!

    Yes, FcgidProcessLifeTime set at 1 minute is really too short but setting it to endless it’s not possible for me with php in a multiple vhost context (different kind of php usage that may have memory issue like big persistent footprint or leaks)
    I tried it and all php-cgi process were zombified after some hours… like with fcgi-proxy)

    I also run mod_fcgid built from the trunk (2.3.10 as I understand…) and never had issue, just this frequent "1 second latency" on high concurrency.

    I’ll try to make a FcgidMinSpareProcesses patch… for fun ^^
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 7374
    Location: Germany, Next to Hamburg

    PostPosted: Sun 24 May '20 20:41    Post subject: Reply with quote

    Well I use the Thread Safe version (TS) and don't have such issues at all.
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 207
    Location: Lille, FR, EU

    PostPosted: Fri 05 Jun '20 9:44    Post subject: Reply with quote

    After 1 week testing TS instead of NTS (but process spawned by fcgid doesn't share context except opcache, so...) nothing changes for me with long FcgidProcessLifeTime value : some child are not reachable after some long time.
    I know that some of my php script are quite dirty.

    Fyi, after testing the 3 patches (see https://bz.apache.org/bugzilla/show_bug.cgi?id=53693 & https://www.mail-archive.com/dev@httpd.apache.org/msg55262.html) the one from Merijn van den Kroonenberg seems to be the best compromise without any 503 and lower latencies
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 7374
    Location: Germany, Next to Hamburg

    PostPosted: Fri 05 Jun '20 15:38    Post subject: Reply with quote

    I'm a bit out of context after such a long time looking that the code of mod_fcgid

    basicly what that patches removes is
    Code:

    if (i > 0 || j > 0 || count_busy_processes(r, &fcgi_request)) {
    -                apr_sleep(apr_time_from_sec(1));


    removing i (connected ipc handle) seems okay.
    removing j (FCGID_APPLY_TRY_COUNT) also okay

    The condition also calls count_busy_processes which sets a lock. That might cause a bottle neck.

    So maybe only removing that function call and lowering the sleep time to re reasonable might be it. This is only my thinking with you trying Wink
    Back to top
    federicoemartinez



    Joined: 13 May 2023
    Posts: 3
    Location: Argentina

    PostPosted: Sat 13 May '23 14:03    Post subject: Reply with quote

    nono303 wrote:
    After 1 week testing TS instead of NTS (but process spawned by fcgid doesn't share context except opcache, so...) nothing changes for me with long FcgidProcessLifeTime value : some child are not reachable after some long time.
    I know that some of my php script are quite dirty.

    Fyi, after testing the 3 patches (see https://bz.apache.org/bugzilla/show_bug.cgi?id=53693 & https://www.mail-archive.com/dev@httpd.apache.org/msg55262.html) the one from Merijn van den Kroonenberg seems to be the best compromise without any 503 and lower latencies


    Would it be possible to you to publish the patched version with the patch by Merijn van den Kroonenberg so I can try it? I think I have hit the same issue but hasn't been able to configure a windows machine to build mod_fcgid yet.
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 207
    Location: Lille, FR, EU

    PostPosted: Sun 14 May '23 10:46    Post subject: Reply with quote

    vc15 vs16 vs17 for x86 & x64 binaries v2.3.10.3 :
    https://github.com/nono303/mod_fcgid/tree/2.3.10.3/bin
    Back to top
    bagu



    Joined: 06 Jan 2011
    Posts: 193
    Location: France

    PostPosted: Sun 14 May '23 21:02    Post subject: Reply with quote

    nono303 wrote:
    vc15 vs16 vs17 for x86 & x64 binaries v2.3.10.3 :
    https://github.com/nono303/mod_fcgid/tree/2.3.10.3/bin


    Apache don't load fcgid module if i replace mod_fcgid.so 2.3.10 vs17 by yours.
    Is there something else to do than just replace ?
    Back to top
    federicoemartinez



    Joined: 13 May 2023
    Posts: 3
    Location: Argentina

    PostPosted: Mon 15 May '23 1:24    Post subject: Reply with quote

    nono303 wrote:
    vc15 vs16 vs17 for x86 & x64 binaries v2.3.10.3 :
    https://github.com/nono303/mod_fcgid/tree/2.3.10.3/bin


    Thanks a lot!
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 207
    Location: Lille, FR, EU

    PostPosted: Mon 15 May '23 7:49    Post subject: Reply with quote

    bagu wrote:
    Apache don't load fcgid module if i replace mod_fcgid.so 2.3.10 vs17 by yours.
    Is there something else to do than just replace ?


    As it has been built with standard CMake, nothing special on it...
    Just check dependencies (cf. https://github.com/nono303/mod_fcgid/blob/master/README.md) and get the good version for your installation (toolset version & arch cf. https://github.com/nono303/PHP-memcache-dll#how-to-get-the-good-version forgetting #3)
    Back to top
    bagu



    Joined: 06 Jan 2011
    Posts: 193
    Location: France

    PostPosted: Mon 15 May '23 19:50    Post subject: Reply with quote

    Ok, it was a download problem. Thanks Wink
    Back to top


    Reply to topic   Topic: mod_fcgid latency on high concurrency View previous topic :: View next topic
    Post new topic   Forum Index -> Apache