logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in  RSS Apache Lounge  


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.



Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Compare ConnectionTimeOut, ProxyTimeOut & KeepAliveTimeo
Author
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Wed 25 Nov '20 6:01    Post subject: Compare ConnectionTimeOut, ProxyTimeOut & KeepAliveTimeo Reply with quote

I am trying to fine-tune a few of our servers at work for performance measures. I read a lot that KeepAliveTimeout can have a very profound effect on performance issues. At times when large files get downloaded our system gives a ProxyTimeOut error, and then we have a bunch of ReverseProxy load balancer that apache has a ConnectionTimeOut parameter for it. What's the difference between all these. How do I combine all of these parameters together to get the best performance?
Back to top
tangent



Joined: 16 Aug 2020
Posts: 39
Location: UK

PostPosted: Wed 25 Nov '20 22:13    Post subject: Reply with quote

This is a tricky subject indeed.

It might help to view keepalives separate from the various timeout parameters. Moreover, when thinking about setting the various proxy timeout parameters, ensure you maintain this sequence:

Timeout (core) > ProxyTimeout (proxy) > ConnectionTimeOut (proxypass)

Note, if not explicitly set, ProxyTimeout and ConnectionTimeOut default to Timeout.

Core Timeout defaults to 60 seconds, but in practice, for some sites I increase this to 300 secs, though this obviously depends on your server capacity and throughput, since you'll be keeping more connections open for longer.

Turning to keepalives, some years ago I too faced issues with connections failing. After investigation, this turned out to be down to some clients not following the RFC's for socket handling, them sending an RST rather than FIN when timing out on a socket connection (notably Internet Explorer). This MS page rather confirms that's still the case https://docs.microsoft.com/en-us/troubleshoot/browsers/change-keep-alive-time-out, noting it says after the timeout period IE "resets" the connection.

If the client sends an RST rather than a FIN, and the server no longer has a socket connection to accept the RST, the client browser will essentially hang, the connection state between the client and service being broken. It's possible this might be triggering some of your proxy timeout error messages, in which case we need to adjust the server KeepAliveTimeout.

The keepalive timeout within IE appears to be 60 seconds whilst the default for Apache is only 5. So to try and cover this eventuality, I increase the Apache KeepAliveTimeout to 61 seconds, meaning there should be a valid socket for existing clients to RST to.

In addition, when using persistent or lengthy connections, there's a viewpoint that the MaxKeepAliveRequests figure can be increased to improve socket reuse, or even set to 0 to allow an unlimited number of requests over the same connection. However, considering the potential misuse of sockets, it's probably unwise to set a value of zero. A non-zero value at least ensures new socket requests do get opened and closed on a regular basis, rather than assuming they can be reused ad infinitum. In the past I've set MaxKeepAliveRequests to 1000.

Finally, regarding the proxy side of things, have you set the proxypass Keepalive to on (it defaults to off)?

These factors may not directly be the cause of your ProxyTimeOut errors, but as the saying goes, sometimes you can't see the wood for the trees.

Hope this helps
Back to top
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Thu 03 Dec '20 19:02    Post subject: Reply with quote

Thanks Tangent for your help.
I am following this article for keepalivetimeout:

https://ioflood.com/blog/2020/02/21/what-is-apache-keepalive-timeout-how-to-optimize-this-critical-setting/

In here It says put keepalivetimeout to 1 second. I am not quite sure that when you talk about IE appearing to be 60 second and Apache is 5 second should I increase from 5 towards 60 or keep it at 5.

I dont have the proxypass keepalive as On, I thought that if you set keepalive setting to On outside the proxy, that setting covers the proxy. Most of my proxies are reverseproxy servers that act as a load balancer.

The other thing that I find mesmerizing is that our Firewall has a feature called "Health Check" and it kind of sends a request to the apache server once every 2 seconds or so, I am finding that with these keepalive settings the log files are increasing in size quite fast. I dont know whether this really has to do with the keepalive settings and health check on the Web Access Firewall or not.

Our company uses F5 appliances as firewall.

Appreciate your comments on this
Back to top
tangent



Joined: 16 Aug 2020
Posts: 39
Location: UK

PostPosted: Thu 03 Dec '20 22:55    Post subject: Reply with quote

That article makes a good case for lowering the KeepaliveTimeout to a very low figure, but is based on the assumption that most web pages are made of multiple requests that load sequentially over a short period of time.

Your situation is different though, in that you've described a design where much of your content is served from back end servers via a proxy configuration, with large file downloads in particular causing occasional proxy timeouts.

That implies the connection state between the client browser and server is sometimes getting broken, and my suggestion of increasing the Apache KeepaliveTimeout is simply aimed at keeping a TCP connection state on the server for longer, specifically because delivery of some of your client content is dependent on the performance (and hence latency) of your back-end proxy servers. Indeed, you don't say what type of content your users are accessing over this proxied service. Are they client/server applications in the traditional sense, that may entail some database query latency?

That Microsoft URL I posted says IE (as well as some other browsers) have a KeepAliveTimeout of 60 seconds, after which they basically give up a connection and reset it using an RST (there's a lot of discussion on the net over closing connections with RST vs FIN). So I'm reasoning if you have a low Apache KeepAliveTimout, then the TCP connection will definitely close and a new connection will have to be created. Hence, an existing proxy connection delivering content may well become orphaned (and possibly cause your timeout message).

Regarding proxypass keepalive, unless your client code is specifically probing or refreshing content from your back end servers, then any external connection keepalive will only be at the socket level, and so wouldn't extend to the proxy connection.

Can I suggest you simply suck it and see, by increasing the KeepAliveTimeout figure to see if it makes any difference, changing one thing at a time, and then monitor things keeping a track of connection counts and status using "netstat -an".

You also mention load balanced proxy connections, so I'm assuming you've got session stickyness covered for those proxies?

Finally, regarding your F5 firewall health check probes (which are nothing to do with Apache keepalive settings), you can use the following dodge to prevent certain site requests being logged. Adjust the Request_URI to suit your F5 probe.

Code:
# Define URI entries for which we don't want to create log entries.
#
SetEnvIfNoCase Request_URI (?i)^/favicon.ico$ DontLog
SetEnvIfNoCase Request_URI ^/$ DontLog

CustomLog logs/access_log common env=!DontLog


Hope this helps.
Back to top
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Fri 04 Dec '20 0:42    Post subject: Reply with quote

Thanks Again Tangent
I will give more details about the back end that we have shortly, but I had one question.

In a earlier post you say:

Timeout (core) > ProxyTimeout (proxy) > ConnectionTimeOut (proxypass)

Does this mean that lets say
Value of Timeout =300
therefore
Value of ProxyTimeout should be 200 (as an example)
and hence the value of ConnectionTimeOut should be 100 (lets say)

so 300 > 200 > 100

Thanks
Back to top
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Fri 04 Dec '20 3:56    Post subject: Reply with quote

Our system is a application running on JBOSS that serves HTML pages, this application runs on multiple servers and a internal domain points to a apache server. This apache server acts as a reverse proxy server redirecting the calls to 5 other jboss application servers. The actual domain name points to the apache server.
Thats one configuration,

The other configuration that we have is the JBOSS application talking to third party vendors.

Again an apache server is defined between the jboss application and the vendor where it acts as a proxy server. The jboss calls a domain that is defined as a virtualhost on the apache server, The apache server in turn redirects that call to the third party vendor. It uses a forward proxy to do that.

So that's how our Reverse Proxy and Forward Proxies are configured.
Back to top
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Fri 04 Dec '20 8:21    Post subject: Reply with quote

I was able to implement your suggestion about logging, and the virtualhost customlog is now reduced in size.
Now my challenge is the main errorlog
Apparently apache 2.4.46 has ErrorLogFormat command and ErrorLog command that you cannot apply a setenfifNoCase to it. So how would I tackle that one.
The loglevel is set to error, but it still logs quite a bit.
Thanks
Back to top
tangent



Joined: 16 Aug 2020
Posts: 39
Location: UK

PostPosted: Fri 04 Dec '20 21:23    Post subject: Reply with quote

Re:
apishdad wrote:
Thanks Again Tangent

In a earlier post you say:

Timeout (core) > ProxyTimeout (proxy) > ConnectionTimeOut (proxypass)

Does this mean that lets say
Value of Timeout =300
therefore
Value of ProxyTimeout should be 200 (as an example)
and hence the value of ConnectionTimeOut should be 100 (lets say)

so 300 > 200 > 100

Yes, that's the idea, though strictly speaking I should have said:

Timeout (core) >= ProxyTimeout (proxy) >= ConnectionTimeOut (proxypass)

I personally wouldn't change ProxyTimeout and ConnectionTimeOut from their defaults unless there's a specific reason to do so in relation to your back end services. Depending on what clues your error messages are telling you, the forward proxy to the remote third party vendor might be a candidate.

If you do choose to change the timeouts from their defaults, then just stick to that sequence.

In your next post you mention there are multiple JBoss servers behind the reverse proxy, so assume you've set up session stickiness in your proxy balancer configuration? I personally prefer to use Apache based cookies for session stickiness, rather than using JSESSIONID or the like from the back end application servers, since this decouples Apache from potential changes in the application. See example code here: https://httpd.apache.org/docs/2.4/mod/mod_proxy_balancer.html#example

In your final post over error logging, there is as you say no conditional variable option, but then error conditions can occur at any point in request / response handling and need to be recorded. Personally, I would try to catagorise the errors to understand what's causing them, and then work with the application developers if needbe to resolve the worst problems.

I'm guessing many of them are application related, either due to missing content, connectivity, performance, or servlet timeout conditions.

You may be able to use ErrorDocument entries to trap certain classes of error, so providing a more user friendly responses, e.g.

Code:
<LocationMatch (?i)^/(apple-touch-icon.*.png|favicon.ico)$>
  ErrorDocument 404 "Icon file does not exist"
</LocationMatch>
and
Code:
# Redirect servlet errors to custom error page
ProxyErrorOverride On
ErrorDocument 500 /error.html
Back to top
apishdad



Joined: 01 Jul 2019
Posts: 16
Location: Canada, Toronto

PostPosted: Mon 07 Dec '20 19:01    Post subject: Reply with quote

Hi Tangent,
Just a token of thanks and appreciation for your help. You answered all my concerns and questions.
thanks again.
Back to top


Reply to topic   Topic: Compare ConnectionTimeOut, ProxyTimeOut & KeepAliveTimeo View previous topic :: View next topic
Post new topic   Forum Index -> Apache