| Author | 
  | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Fri 14 Jun '24 8:36    Post subject: Apache Proxy errors AH01102 + AH00898 - Request too long | 
     | 
 
  | 
 
Hello,
 
 
I have an error on Apache + proxy with balancer.
 
 
When an url exceeds a certain length, I have an Proxy error 502 :
 
 
 	  | Code: | 	 		  Proxy Error
 
 
The proxy server received an invalid response from an upstream server.
 
The proxy server could not handle the request
 
 
Reason: Error reading from remote server | 	  
 
 
With this message in error log : 
 
 
 	  | Code: | 	 		  [Fri Jun 14 08:18:33.010605 2024] [proxy_http:error] [pid 47783:tid 139883905533632] (104)Connection reset by peer: [client 84.103.4.88:59302] AH01102: error reading status line from remote server 192.168.202.41:80
 
[Fri Jun 14 08:18:33.010830 2024] [proxy:error] [pid 47783:tid 139883905533632] [client 84.103.4.88:59302] AH00898: Error reading from remote server returned by /test.php
 
 | 	  
 
 
Apache log :
 
 	  | Code: | 	 		  84.103.xxx.xxx - - [14/Jun/2024:08:18:33 +0200] "GET /favicon.ico HTTP/1.1" 404 2005 "https://dev.xxxxxx.cloud/test.php?code=0.AQwAJDMu3tBPA0i7lCy2LEWy9QpYbsxwWF5Hrz2Vu76zvO4MANc.AgABBAIAAADnfolhJpSnRYB1SVj-Hgd8AgDs_wUA9P_3Y6zJdaJIrc4scFbbFErN7BCueQ-2bx3G7cA-6dLRBgyIe_23b8xAx1wIQIWBNf45-5b67bkHSc56oWE5XdIP0BJffyrMJAOrTIhaHO95tQ8Rpmnluhzx3zqBjgi-Q3U6LBd__e49k0Qe48L0RGuIIysOJ2UcpRrgpmIp2AeLez51JjqfEKKsxB4jvhIxH5YMtTqnXxoFKuIiG8XzDFpXGPDxSWeZ1zhKcKgfB3yLsP2GtxJ0g97JSKLyFtft5CfYs0UHP0r_kduwGw--dDpnq1tmTQE1vqZhiX9mebWtALoqaM1ykbeD2RDL8bvVAy3Yu3vKmwOu4RwJjEXLojR1hrf8nsDLSTskxLRZWbAKFi7rZjxo30xaxTRPKXpv5Xrnd9MMrBQa8xKU52IdLUMvgEKD1mkvpKH871s1K2mRoB5LqVDamIgKcMpRqrfECIPCmVBDPfDjZJU12wzi5-S3965qkWWb1dIJNgQ4J0ar9THVBxCICrH-I2CgHfFuJ3Y7JXJIHG9tLcpyDDHWeXe5bKUBfvGRNEx0rsEp9L-ZpMeknx2ReCqToLFu_j-SL8LGYMGqSULtNJt2y5heRnns4vboAV5k_GX51y_fSvgPX-FhKqj9ERfirvXLQSwPJqpVdr88mGX9_9yKd-PP9MCg4LD6Mkd2oD8mpcOa6WmRN3EahoeMVM5qkS2Bt5Q4ATOOeWhJ_U_LPva_DRToswcKaTINSff6wWH5GCl7_be3Hk-gbZHtPNl-NaJsjuDqZcy8bjx5eU2mf3IARZWUFM7DeuOniPh3cP2t6MMT5Q1rqVmnJ2P_Me9xHFFMPqVt3-PIU5xlK7OpHO-tMVnYrVUBEHcG5zhlIy3oIeQSMgzhtkXdQF0b-r6yd6Db2ZHvMxn7eLedJtj_zoD111111111111111" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"
 
 | 	  
 
 
My Apache : Apache/2.4.57 
 
OS : Debian 12
 
 
I'm desperate to find a solution to this problem...
 
 
Thanks for your help | 
 
  | 
| Back to top | 
 | 
tangent Moderator
 
  Joined: 16 Aug 2020 Posts: 400 Location: UK
  | 
 Posted: Fri 14 Jun '24 16:43    Post subject:  | 
     | 
 
  | 
 
You don't say what your upstream load balanced servers are running, but presume you've checked to see what corresponding error is being recorded there?
 
 
Your request URL alone is over 1K in length, and request headers in general have grown in size over time (multiple cookies, authentication tokens, X-headers, etc.), so is it possible you've exceeded some buffer size for some of your client requests?
 
 
In the past I've seen this problem with Tomcat backend servers, where we needed to increase maxHttpHeaderSize in server.xml.  Other backend services will equally have some finite defaults.
 
 
Also, at the Apache level, check out ProxyIOBufferSize which defaults to 8192 bytes. | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Sat 15 Jun '24 8:17    Post subject: RE | 
     | 
 
  | 
 
Hello,
 
 
Sorry. I have an other apache serveur behind.
 
 
If I test directly the url with local IP in replacement of domain, it works.
 
 
Errors are logged on Proxy server, not on server behind.
 
 
I've already tested ProxyIOBufferSize directive.
 
It doesn't change anything. | 
 
  | 
| Back to top | 
 | 
James Blond Moderator
  
  Joined: 19 Jan 2006 Posts: 7443 Location: EU, Germany, Next to Hamburg
  | 
 Posted: Mon 17 Jun '24 10:53    Post subject:  | 
     | 
 
  | 
 
You may set  	  | Code: | 	 		  | connectiontimeout=[time in seconds] timeout=[time in seconds] | 	  
 
 
for example 60 seconds (1 min)
 
 	  | Code: | 	 		  
 
ProxyPass / http://localhost:4382/ connectiontimeout=60 timeout=60 | 	 
  | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Wed 19 Jun '24 8:53    Post subject:  | 
     | 
 
  | 
 
| Also tested without any result... | 
 
  | 
| Back to top | 
 | 
James Blond Moderator
  
  Joined: 19 Jan 2006 Posts: 7443 Location: EU, Germany, Next to Hamburg
  | 
 Posted: Fri 21 Jun '24 8:48    Post subject:  | 
     | 
 
  | 
 
| Please post your reverse proxy config. | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Fri 21 Jun '24 10:32    Post subject:  | 
     | 
 
  | 
 
Below the config
 
 
 	  | Code: | 	 		  
 
<VirtualHost *:443>
 
  ServerName www.xxxxxx.cloud
 
  ServerAlias xxxxxx.cloud
 
 
  ErrorLog /var/log/apache2/error.xxxxxx.cloud.log
 
  CustomLog /var/log/apache2/xxxxxx.cloud.log combined
 
 
  <Directory /var/www/xxxxxx.cloud/>
 
    Options Indexes FollowSymLinks
 
    AllowOverride All
 
    DirectoryIndex index.htm index.php index.html
 
  </Directory>
 
 
  Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
 
 
  <Proxy balancer://xxxxxx_prod>
 
    include /etc/apache2/xxxxxx_prod.conf
 
    ProxySet lbmethod=byrequests
 
    ProxySet stickysession=ROUTEID
 
    ProxySet nofailover=On
 
  </Proxy>
 
 
  ProxyPass "/"  "balancer://xxxxxx_prod/"
 
  ProxyPassReverse "/"  "balancer://xxxxxx_prod/"
 
 
  <Location /balancer-manager>
 
    SetHandler balancer-manager
 
    Order deny,allow
 
    Allow from all
 
  </Location>
 
 
  ProxyRequests Off
 
  ProxyPreserveHost On
 
 
  Include /etc/letsencrypt/options-ssl-apache.conf
 
  ServerAlias beta.xxxxxx.cloud
 
  ServerAlias oweba.xxxxxx.cloud
 
  SSLCertificateFile /etc/letsencrypt/live/www.xxxxxx.cloud/fullchain.pem
 
  SSLCertificateKeyFile /etc/letsencrypt/live/www.xxxxxx.cloud/privkey.pem
 
</VirtualHost>
 
 | 	  
 
 
And balancer settings
 
 
 	  | Code: | 	 		  
 
    BalancerMember "http://192.168.202.101" loadfactor=1 route=route1
 
    BalancerMember "http://192.168.202.102" loadfactor=1 route=route2
 
    BalancerMember "http://192.168.202.103" loadfactor=1 route=route3
 
 | 	 
  | 
 
  | 
| Back to top | 
 | 
tangent Moderator
 
  Joined: 16 Aug 2020 Posts: 400 Location: UK
  | 
 Posted: Fri 21 Jun '24 21:32    Post subject:  | 
     | 
 
  | 
 
Looking at your proxy configuration, and the error log detail in the original post, I notice the error relates to remote server 192.168.202.41:80 whereas your balancer member IP's are 192.168.202.101, 102 and 103. Is this relevant, or did you change the balancer IP's for the posted config?
 
 
Otherwise, if you've already tried increasing the buffer sizes, I suspect the problem lies somewhere with the proxied headers being fragmented / truncated, or relates to compression, but to analyse this further believe you need more detail to crack where the exact problem lies.
 
 
There are two things I'd try next here.
 
 
Firstly try removing the Accept-Encoding request header, so that the back end servers send an uncompressed response to Apache. Put the following in your virtual host block or global configuration (if acceptable).
 
 	  | Code: | 	 		  
 
RequestHeader unset Accept-Encoding    | 	  	
 
Secondly, I'd use network capture tools such as tcpdump on your Debian platforms (Wireshark if the backend servers are Windows based), to see exactly what's going on here during the client request.
 
 
Per chance, is there a firewall between your Apache and the backend servers? | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Tue 25 Jun '24 9:35    Post subject:  | 
     | 
 
  | 
 
We have 1 proxy and 3 environments (prod / dev /beta).
 
The configuration given is about Prod.
 
 
IP address 192.168.202.41 is for dev environment.
 
 
I've added "RequestHeader ..." but it doesn't change anything.
 
 
I'm checking with tcpdump but not simple to analyze. | 
 
  | 
| Back to top | 
 | 
tangent Moderator
 
  Joined: 16 Aug 2020 Posts: 400 Location: UK
  | 
 Posted: Tue 25 Jun '24 12:55    Post subject:  | 
     | 
 
  | 
 
Ok, so your one proxy covers multiple environments. Presumably, this error occurs across all three, albeit inconsistently?
 
 
Assuming you can narrow the problem to say the dev environment, you should be able to craft a filter for tcpdump to restrict the captured traffic (written to a pcap file). I'd personally just capture and analyse the traffic between the proxy and corresponding/potential back-end servers (hence the recommendation to remove the Accept-Encoding request header, so the proxied content isn't compressed).
 
 
I tend to transfer the pcap files over to a Windows platform, where it's easier to analyse them with Wireshark.
 
 
Let us know how you get on. | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Tue 25 Jun '24 14:12    Post subject:  | 
     | 
 
  | 
 
Yes it's the same on each environment.
 
 
I try your method and give you feedback. | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Wed 26 Jun '24 10:31    Post subject:  | 
     | 
 
  | 
 
We found a workaround.
 
 
We use proxy server in "normal web mode" for one subdomain.
 
We store parameters then we redirect to other script that recover the paramaters.
 
 
Thanks for your help | 
 
  | 
| Back to top | 
 | 
tangent Moderator
 
  Joined: 16 Aug 2020 Posts: 400 Location: UK
  | 
 Posted: Wed 26 Jun '24 21:03    Post subject:  | 
     | 
 
  | 
 
Not sure I fully follow your explantion, but am glad you've resolved the problem with a workaround.
 
 
Apache Lounge has numerous subscribers with many years of experience. Don't hesitate to post queries for help, or equally your expertise in solving problems. | 
 
  | 
| Back to top | 
 | 
Renaud SALVIGNAC
 
 
  Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
  | 
 Posted: Thu 27 Jun '24 8:14    Post subject:  | 
     | 
 
  | 
 
I will not miss it. 
 
 
Thanks for your advices. | 
 
  | 
| Back to top | 
 |