logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in  RSS Apache Lounge  


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.



Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: mod_fcgid latency on high concurrency
Author
nono303



Joined: 20 Dec 2016
Posts: 118
Location: France, Lille

PostPosted: Wed 20 May '20 12:22    Post subject: mod_fcgid latency on high concurrency Reply with quote

Hi @all!

I open this topic to (re?)start a discussion around the mod_fcgid latency in case of high concurrency.
This subject has already been discussed (cf. https://bz.apache.org/bugzilla/show_bug.cgi?id=53693) with a revert in mod_fcgid 2.3.9
Being faced with the problem (thumb gallery whose images are generated in php via mod_fcgid involving concurrency from several tens of processes without lazy-load) I have randomly latencies of (exactly) one second on 1 to several images.
This is explained in the mod_fcgid code if all the processes are busy:
Code:
apr_sleep (apr_time_from_sec (1));

https://github.com/pagespeed/mod_fcgid/blob/master/modules/fcgid/fcgid_bridge.c#L456

In your point of view, what would be the best way to go around this bottleneck?
    1) Reducing the waiting time apr_sleep (apr_time_from_msec (XX)); maybe with XX externalized as the time to spawn a new process depends on many parameters
    2) Directly force a new process spawn and try to handle request to it more than 2 times (see patch https://www.mail-archive.com/dev@httpd.apache.org/msg55268.html which I am currently testing)
    3) Other ideas? On configuration side?
Many thanks for your advices!
++NoNo
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 118
    Location: France, Lille

    PostPosted: Wed 20 May '20 13:47    Post subject: Reply with quote

    ...and here is my config:
    Code:
    FcgidInitialEnv PHPRC "C:/serveur/php7"
    <Files ~ (\.php)>
      FcgidWrapper "C:/serveur/php7/php-cgi.exe -c C:/serveur/php7/php.ini" .php
    </Files>
    FcgidInitialEnv TEMP "R:/php-httpd_temp"
    FcgidInitialEnv TMP "R:/php-httpd_temp"
    FcgidInitialEnv windir "C:\WINDOWS"
    FcgidInitialEnv SystemRoot "C:\Windows"
    FcgidInitialEnv SystemDrive "C:"
    FcgidInitialEnv PROCESSOR_ARCHITECTURE "AMD64"
    FcgidInitialEnv PROCESSOR_IDENTIFIER "Intel64 Family 6 Model 42 Stepping 7, GenuineIntel"
    FcgidInitialEnv PROCESSOR_LEVEL "6"
    FcgidInitialEnv PROCESSOR_REVISION "2a07"
    FcgidInitialEnv NUMBER_OF_PROCESSORS 4
    FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 0
    FcgidInitialEnv PHP_FCGI_CHILDREN 0
    FcgidFixPathinfo 1
    FcgidWin32PreventOrphans On
    FcgidMaxRequestsPerProcess 0
    FcgidMaxProcesses 40
    FcgidMaxProcessesPerClass 40
    FcgidMinProcessesPerClass 16
    FcgidIdleScanInterval 1
    FcgidErrorScanInterval 1
    FcgidZombieScanInterval 1
    FcgidIdleTimeout 60
    FcgidProcessLifeTime 60
    FcgidConnectTimeout 1
    FcgidIOTimeout 600
    FcgidBusyTimeout 600
    FcgidBusyScanInterval 600
    FcgidMaxRequestLen 134217728
    FcgidOutputBufferSize 0
    FcgidPassHeader Authorization
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 6799
    Location: Germany, Next to Hamburg

    PostPosted: Wed 20 May '20 16:33    Post subject: Reply with quote

    The re-spawn takes time. I noticed that, too.

    You set FcgidProcessLifeTime to one minute. I set it to 0 (endless). Every minute mod_fcgid kills the spawned php-cgi process.

    Why do you let the module kill the php process so early? That also causes a log flood[1]. It has only been fixed in trunk. But nobody dares a release of 2.3.10. I run the .10 for years without any issues.

    --- edit ---

    My config[2]

    --- /edit ---

    [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=54597
    [2] https://www.apachelounge.com/viewtopic.php?t=2394
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 118
    Location: France, Lille

    PostPosted: Sun 24 May '20 14:53    Post subject: Reply with quote

    Thanks @James Blond for your feedback!

    Yes, FcgidProcessLifeTime set at 1 minute is really too short but setting it to endless it’s not possible for me with php in a multiple vhost context (different kind of php usage that may have memory issue like big persistent footprint or leaks)
    I tried it and all php-cgi process were zombified after some hours… like with fcgi-proxy)

    I also run mod_fcgid built from the trunk (2.3.10 as I understand…) and never had issue, just this frequent "1 second latency" on high concurrency.

    I’ll try to make a FcgidMinSpareProcesses patch… for fun ^^
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 6799
    Location: Germany, Next to Hamburg

    PostPosted: Sun 24 May '20 20:41    Post subject: Reply with quote

    Well I use the Thread Safe version (TS) and don't have such issues at all.
    Back to top
    nono303



    Joined: 20 Dec 2016
    Posts: 118
    Location: France, Lille

    PostPosted: Fri 05 Jun '20 9:44    Post subject: Reply with quote

    After 1 week testing TS instead of NTS (but process spawned by fcgid doesn't share context except opcache, so...) nothing changes for me with long FcgidProcessLifeTime value : some child are not reachable after some long time.
    I know that some of my php script are quite dirty.

    Fyi, after testing the 3 patches (see https://bz.apache.org/bugzilla/show_bug.cgi?id=53693 & https://www.mail-archive.com/dev@httpd.apache.org/msg55262.html) the one from Merijn van den Kroonenberg seems to be the best compromise without any 503 and lower latencies
    Back to top
    James Blond
    Moderator


    Joined: 19 Jan 2006
    Posts: 6799
    Location: Germany, Next to Hamburg

    PostPosted: Fri 05 Jun '20 15:38    Post subject: Reply with quote

    I'm a bit out of context after such a long time looking that the code of mod_fcgid

    basicly what that patches removes is
    Code:

    if (i > 0 || j > 0 || count_busy_processes(r, &fcgi_request)) {
    -                apr_sleep(apr_time_from_sec(1));


    removing i (connected ipc handle) seems okay.
    removing j (FCGID_APPLY_TRY_COUNT) also okay

    The condition also calls count_busy_processes which sets a lock. That might cause a bottle neck.

    So maybe only removing that function call and lowering the sleep time to re reasonable might be it. This is only my thinking with you trying Wink
    Back to top


    Reply to topic   Topic: mod_fcgid latency on high concurrency View previous topic :: View next topic
    Post new topic   Forum Index -> Apache