logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache reverse proxy Dotnet Upstream error
Author
tonysar



Joined: 01 Dec 2023
Posts: 4
Location: Canada

PostPosted: Sat 02 Dec '23 18:44    Post subject: Apache reverse proxy Dotnet Upstream error Reply with quote

Hello.
I am working on a site developed in Dotnet hosted on Linux Ubuntu vps , Apache 2 .
Apache is reverse proxy to dotnet core . Using Let's Encrypt SSL, Everything is working well with One single issue.

At times I have to start upstream server , being hosted on linux , init service script located on /etc/systemd/system will take about 10 seconds to restart upstream server .
because of this delay .. I get error 503
Proxy_http_error ; HTTP: failed to make connection to backend: 127.0.0.1.
back stream server assigned to localhost and listens on port 5000. this error due to stoped server.

I like to know , if there is way to fix this ? maybe somehow tell apache to wait for 10 seconds .
I found nothing in past 10 days on how to fix this little problem

Thanks.
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Sun 03 Dec '23 21:07    Post subject: Reply with quote

You haven't posted the relevant section of your reverse proxy configuration, but assuming you're using the ProxyPass directive, along with ProxyTimeout, what parameter options have you configured?

ProxyTimeout defaults to the Apache Timeout setting, which is 60 seconds by default, so I'm surprised your back end server restarts are causing a problem, unless this figure has been changed.

Specifically for ProxyPass, have you set timeout, connectiontimeout and possibly retry? I'd also consider setting keepalive to on.

See https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass and https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxytimeout for details.
Back to top
tonysar



Joined: 01 Dec 2023
Posts: 4
Location: Canada

PostPosted: Sun 03 Dec '23 22:31    Post subject: Reply with quote

Thanks.
here is the config of vhost .

Code:
<VirtualHost *.*>
RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
</VirtualHost>



<VirtualHost *:80>
   ServerAdmin admin@MYDOMAIN.ca
   DocumentRoot /var/www/solution/html
   ProxyPreserveHost On
        ProxyPass / http://127.0.0.1:5000/ retry=1 acquire=3000 timeout=600 Keepalive=On
        ProxyPassReverse / http://127.0.0.1:5000/
        ServerName MYDOMAIN.ca
        ServerAlias www.MYDOMAIN.ca

   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined

SSLProxyEngine on
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.MYDOMAIN.ca [OR]
RewriteCond %{SERVER_NAME} =MYDOMAIN.ca
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>


I am not very familiar with proxy setting . this configuration comes from MSDN .. and Added few from searching the net for this issue.

i have added retry , timeout to the ProxyPass , that has never worked.

Mod note: removed not needed config comments
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Mon 04 Dec '23 16:33    Post subject: Reply with quote

In your configuration, the retry parameter is set to 1 second (the default is 60 seconds). If your backend service restart takes 10 seconds, then I'd set retry to be slightly greater than that time. This is what the mod_proxy_http docs say about the retry parameter.
    retry 60 Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache httpd will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.
You've also set the proxy (socket) timeout to 600 seconds, which means you're prepared to wait 10 mins for proxied content response from the backend. This may be perfectly reasonable, depending on what processing the backend is actually doing, e.g. report generation, but unless needed, it will cause Apache to hold on to resources that could otherwise be released sooner.

That said, your configuration looks somewhat confusing over the VirtualHost sections. Apart from the proxy entries, your port 80 section contains rewrite rules to redirect the client to an https connection on port 443. I'd therefore expect to see the proxy entries, and bulk of the configuration settings, in a port 443 virtualhost section.

If the above parameter changes don't help solve the problem, then up the loglevel for mod_proxy to see what's going on, e.g.

Code:
LogLevel warn proxy:trace6
Back to top
tonysar



Joined: 01 Dec 2023
Posts: 4
Location: Canada

PostPosted: Tue 05 Dec '23 0:37    Post subject: Reply with quote

Thank you so much for your help
I increased the retry time , didnt't work . as you have suggested, changed loglevel to show proxy . nothing really there to show reason behind this. except connection refused during backend boot .

As for vhost.
I used certbot to generate ssl , Certbot created second vhost for 443 post that is identical to what I have posted here .

other then that one issue everything seems to be working and there are no errors in log that would be of any concerns . now the question is why I can not make apache / Proxt wait for 10 seconds, is just Odd .
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Tue 05 Dec '23 16:16    Post subject: Reply with quote

Ok, so it would seem the retry setting applies to existing connections, rather than creating new ones.

Even if the core Apache timeout hasn't been changed elsewhere in your configuration, I'd still explicitly set the ProxyTimeout, to localise the setting for your proxy service.

I'd also add connectiontimeout and ping parameters to your ProxyPass directive. If you read the documentation over ping, it causes Apache to test the connection first (waiting for the specified time), so this should help get over the backend restart problem, albeit for a slight increase in service overhead.

Adjust the parameter times to suit your system setup, e.g.
Code:
ProxyPreserveHost On
ProxyTimeout 600
ProxyPass / http://127.0.0.1:5000/ acquire=3000 connectiontimeout=20 keepalive=on ping=20 retry=20 timeout=600
ProxyPassReverse / http://127.0.0.1:5000/
Back to top


Reply to topic   Topic: Apache reverse proxy Dotnet Upstream error View previous topic :: View next topic
Post new topic   Forum Index -> Apache