Topic: Compare ConnectionTimeOut, ProxyTimeOut & KeepAliveTimeo

[apishdad]

I am trying to fine-tune a few of our servers at work for performance measures. I read a lot that KeepAliveTimeout can have a very profound effect on performance issues. At times when large files get downloaded our system gives a ProxyTimeOut error, and then we have a bunch of ReverseProxy load balancer that apache has a ConnectionTimeOut parameter for it. What's the difference between all these. How do I combine all of these parameters together to get the best performance?

[tangent]

This is a tricky subject indeed.

It might help to view keepalives separate from the various timeout parameters. Moreover, when thinking about setting the various proxy timeout parameters, ensure you maintain this sequence:

Timeout (core) > ProxyTimeout (proxy) > ConnectionTimeOut (proxypass)

Note, if not explicitly set, ProxyTimeout and ConnectionTimeOut default to Timeout.

Core Timeout defaults to 60 seconds, but in practice, for some sites I increase this to 300 secs, though this obviously depends on your server capacity and throughput, since you'll be keeping more connections open for longer.

Turning to keepalives, some years ago I too faced issues with connections failing. After investigation, this turned out to be down to some clients not following the RFC's for socket handling, them sending an RST rather than FIN when timing out on a socket connection (notably Internet Explorer). This MS page rather confirms that's still the case https://docs.microsoft.com/en-us/troubleshoot/browsers/change-keep-alive-time-out, noting it says after the timeout period IE "resets" the connection.

If the client sends an RST rather than a FIN, and the server no longer has a socket connection to accept the RST, the client browser will essentially hang, the connection state between the client and service being broken. It's possible this might be triggering some of your proxy timeout error messages, in which case we need to adjust the server KeepAliveTimeout.

The keepalive timeout within IE appears to be 60 seconds whilst the default for Apache is only 5. So to try and cover this eventuality, I increase the Apache KeepAliveTimeout to 61 seconds, meaning there should be a valid socket for existing clients to RST to.

In addition, when using persistent or lengthy connections, there's a viewpoint that the MaxKeepAliveRequests figure can be increased to improve socket reuse, or even set to 0 to allow an unlimited number of requests over the same connection. However, considering the potential misuse of sockets, it's probably unwise to set a value of zero. A non-zero value at least ensures new socket requests do get opened and closed on a regular basis, rather than assuming they can be reused ad infinitum. In the past I've set MaxKeepAliveRequests to 1000.

Finally, regarding the proxy side of things, have you set the proxypass Keepalive to on (it defaults to off)?

These factors may not directly be the cause of your ProxyTimeOut errors, but as the saying goes, sometimes you can't see the wood for the trees.

Hope this helps

[apishdad]

Thanks Tangent for your help.
I am following this article for keepalivetimeout:

https://ioflood.com/blog/2020/02/21/what-is-apache-keepalive-timeout-how-to-optimize-this-critical-setting/

In here It says put keepalivetimeout to 1 second. I am not quite sure that when you talk about IE appearing to be 60 second and Apache is 5 second should I increase from 5 towards 60 or keep it at 5.

I dont have the proxypass keepalive as On, I thought that if you set keepalive setting to On outside the proxy, that setting covers the proxy. Most of my proxies are reverseproxy servers that act as a load balancer.

The other thing that I find mesmerizing is that our Firewall has a feature called "Health Check" and it kind of sends a request to the apache server once every 2 seconds or so, I am finding that with these keepalive settings the log files are increasing in size quite fast. I dont know whether this really has to do with the keepalive settings and health check on the Web Access Firewall or not.

Our company uses F5 appliances as firewall.

Appreciate your comments on this

[tangent]

That article makes a good case for lowering the KeepaliveTimeout to a very low figure, but is based on the assumption that most web pages are made of multiple requests that load sequentially over a short period of time.

Your situation is different though, in that you've described a design where much of your content is served from back end servers via a proxy configuration, with large file downloads in particular causing occasional proxy timeouts.

That implies the connection state between the client browser and server is sometimes getting broken, and my suggestion of increasing the Apache KeepaliveTimeout is simply aimed at keeping a TCP connection state on the server for longer, specifically because delivery of some of your client content is dependent on the performance (and hence latency) of your back-end proxy servers. Indeed, you don't say what type of content your users are accessing over this proxied service. Are they client/server applications in the traditional sense, that may entail some database query latency?

That Microsoft URL I posted says IE (as well as some other browsers) have a KeepAliveTimeout of 60 seconds, after which they basically give up a connection and reset it using an RST (there's a lot of discussion on the net over closing connections with RST vs FIN). So I'm reasoning if you have a low Apache KeepAliveTimout, then the TCP connection will definitely close and a new connection will have to be created. Hence, an existing proxy connection delivering content may well become orphaned (and possibly cause your timeout message).

Regarding proxypass keepalive, unless your client code is specifically probing or refreshing content from your back end servers, then any external connection keepalive will only be at the socket level, and so wouldn't extend to the proxy connection.

Can I suggest you simply suck it and see, by increasing the KeepAliveTimeout figure to see if it makes any difference, changing one thing at a time, and then monitor things keeping a track of connection counts and status using "netstat -an".

You also mention load balanced proxy connections, so I'm assuming you've got session stickyness covered for those proxies?

Finally, regarding your F5 firewall health check probes (which are nothing to do with Apache keepalive settings), you can use the following dodge to prevent certain site requests being logged. Adjust the Request_URI to suit your F5 probe.

[apishdad]

Thanks Again Tangent
I will give more details about the back end that we have shortly, but I had one question.

In a earlier post you say:

Timeout (core) > ProxyTimeout (proxy) > ConnectionTimeOut (proxypass)

Does this mean that lets say
Value of Timeout =300
therefore
Value of ProxyTimeout should be 200 (as an example)
and hence the value of ConnectionTimeOut should be 100 (lets say)

so 300 > 200 > 100

Thanks

[apishdad]

Our system is a application running on JBOSS that serves HTML pages, this application runs on multiple servers and a internal domain points to a apache server. This apache server acts as a reverse proxy server redirecting the calls to 5 other jboss application servers. The actual domain name points to the apache server.
Thats one configuration,

The other configuration that we have is the JBOSS application talking to third party vendors.

Again an apache server is defined between the jboss application and the vendor where it acts as a proxy server. The jboss calls a domain that is defined as a virtualhost on the apache server, The apache server in turn redirects that call to the third party vendor. It uses a forward proxy to do that.

So that's how our Reverse Proxy and Forward Proxies are configured.

[apishdad]

I was able to implement your suggestion about logging, and the virtualhost customlog is now reduced in size.
Now my challenge is the main errorlog
Apparently apache 2.4.46 has ErrorLogFormat command and ErrorLog command that you cannot apply a setenfifNoCase to it. So how would I tackle that one.
The loglevel is set to error, but it still logs quite a bit.
Thanks

[tangent]

Re:

[apishdad]

Hi Tangent,
Just a token of thanks and appreciation for your help. You answered all my concerns and questions.
thanks again.