| Author |  | 
| pidfile 
 
 
 Joined: 25 Dec 2013
 Posts: 4
 
 
 | 
|  Posted: Wed 25 Dec '13 16:31    Post subject: Apache 2.4.3 mod_proxy not stable in Production |   |  
| 
 |  
| Merry Xmas   
 Have run through a bunch of threads on mod_proxy race conditions, possible causes, and potential solutions suggested by Apache core devs.
 
 What is the status? Is there a fix for this, maybe in 2.4.7?
 
 On 2.4.3 in production we literally have 10K errors in the past 2 months along the lines of:
 
  	  | Code: |  	  | AH00898: Error reading from remote server returned by... | 
 
 Application server (JVM webapp) issues no errors at the time of mod_proxy error occurring. It seems then that mod_proxy is sporadically broken, which is horrible for end users accessing an otherwise rock solid production application.
 
 Would love to stay on Apache but we're going to have to switch to a different solution if no fix has been made since 2.4.3
 
 Thanks
 |  | 
| Back to top |  | 
| jraute 
 
  
 Joined: 13 Sep 2013
 Posts: 188
 Location: Rheinland, Germany
 
 | 
|  Posted: Wed 25 Dec '13 16:59    Post subject: |   |  
| 
 |  
| What is the complete error line and what configuration you are using? |  | 
| Back to top |  | 
| Steffen Moderator
 
 
 Joined: 15 Oct 2005
 Posts: 3131
 Location: Hilversum, NL, EU
 
 | 
|  Posted: Wed 25 Dec '13 17:00    Post subject: |   |  
| 
 |  
| When does it happen ? 
 Assuming that you use ProxyPass directive.
 
 What about the keepalive timeout, front and back the same ? Good habbit is to make different,  1-2 seconds different (not recall front or back)
 
 Also the default for Timeout has been reduced in 2.4 from 300 to 60. The developers said that if
 there are complaints from users, the new value may be too low and we should maybe reconsider the new value.
 
 So check and play with the settings:
 
 TimeOut 300
 KeepAlive On
 MaxKeepAliveRequests 100
 KeepAliveTimeOut 15
 
 Running Windows ?
 You can try:
 
 AcceptFilter http none
 AcceptFilter https none
 EnableSendfile off
 EnableMMAP off
 |  | 
| Back to top |  | 
| pidfile 
 
 
 Joined: 25 Dec 2013
 Posts: 4
 
 
 | 
|  Posted: Wed 25 Dec '13 21:37    Post subject: |   |  
| 
 |  
| Thanks guys. 
 Linux box (CentOS 6.3)
 
 *******************
 # conf.d/foo.conf
 <Location />
 ProxyPass http://172.16.xx.xx:9001/ max=60 retry=10
 ProxyPassReverse  http://172.16.xx.xx:9001/
 SetOutputFilter DEFLATE
 SetEnv proxy-sendchunks 1
 </Location>
 
 # httpd.conf
 KeepAlive On
 KeepAliveTimeout 2500ms
 EnableSendfile On
 ********************
 
 Timeout shouldn't come into play, page requests typically served in < 400ms, not the 60s default.
 
 Linux default KeepAlive timeout is 2 hours.
 
 Again, apache dev lists do indicate that there is an as yet unresolved (as of 2.4.3 at any rate) issue with mod_proxy and keepalive race conditions.
 
 Not sure what the underlying cause is, but getting on average at least 150 proxy errors per day _in production_ not good.
 |  | 
| Back to top |  | 
| jraute 
 
  
 Joined: 13 Sep 2013
 Posts: 188
 Location: Rheinland, Germany
 
 | 
|  Posted: Wed 25 Dec '13 23:05    Post subject: |   |  
| 
 |  
| Although you didn't post the complete error log line  i am quite sure that the problem still exists. Reading the bug blog there seems to be only one solution which is based on deactivating keep alives and i remember such errors even with 2.4.7 in our logs too although the clients do not have any problems.
 
 Have you tried the settings Steffen mentioned?
 |  | 
| Back to top |  | 
| pidfile 
 
 
 Joined: 25 Dec 2013
 Posts: 4
 
 
 | 
|  Posted: Thu 26 Dec '13 0:43    Post subject: |   |  
| 
 |  
| Thanks, settings are the same as Stefan's, just shorter TTL. 
 Could turn off KeepAlive, but prefer performance gain, not loss.
 
 Will dig around some more, perhaps there's something specific to my setup if clients are affected by the errors while in your case apparently they are not.
 |  | 
| Back to top |  | 
| jraute 
 
  
 Joined: 13 Sep 2013
 Posts: 188
 Location: Rheinland, Germany
 
 | 
|  Posted: Thu 26 Dec '13 11:26    Post subject: |   |  
| 
 |  
| There's one advice in the blog to use 
 
  	  | Code: |  	  | SetEnv proxy-initial-not-pooled | 
 
 Have you tried that?
 We have this in use for many configurations with positive effects.
 
 Btw proxy-sendchunks is only working reliably with backend servers supporting HTTP 1.1
 |  | 
| Back to top |  | 
| pidfile 
 
 
 Joined: 25 Dec 2013
 Posts: 4
 
 
 | 
|  Posted: Thu 26 Dec '13 12:44    Post subject: |   |  
| 
 |  
| See http://www.gossamer-threads.com/lists/apache/dev/428261 for all the gory details. From the thread: 
 
  	  | Quote: |  	  | As Covener pointed out to me in IRC, using mod_proxy_http's env variable "proxy-initial-not-pooled" does
 offer a solution to the problem albeit at the cost of performance.
 | 
 
 Thread has been running for over 2 years, last updated a couple of weeks ago with a bunch of dev activity; there is as yet no solution that does not impact performance.
 
 Our JVM app speaks HTTP 1.1, no problem there.
 |  | 
| Back to top |  | 
| jraute 
 
  
 Joined: 13 Sep 2013
 Posts: 188
 Location: Rheinland, Germany
 
 | 
|  Posted: Fri 27 Dec '13 12:44    Post subject: |   |  
| 
 |  
| Ok, no solution so far. 
 (Honestly spoken in our setting the "performance problem" is minimal - up to 30000 page requests per day and the performance and especially the response time is ok. But you are right, it depends on the things you want to do.)
 
 Greets
 |  | 
| Back to top |  |