logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache Proxy errors AH01102 + AH00898 - Request too long
Author
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Fri 14 Jun '24 8:36    Post subject: Apache Proxy errors AH01102 + AH00898 - Request too long Reply with quote

Hello,

I have an error on Apache + proxy with balancer.

When an url exceeds a certain length, I have an Proxy error 502 :

Code:
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server


With this message in error log :

Code:
[Fri Jun 14 08:18:33.010605 2024] [proxy_http:error] [pid 47783:tid 139883905533632] (104)Connection reset by peer: [client 84.103.4.88:59302] AH01102: error reading status line from remote server 192.168.202.41:80
[Fri Jun 14 08:18:33.010830 2024] [proxy:error] [pid 47783:tid 139883905533632] [client 84.103.4.88:59302] AH00898: Error reading from remote server returned by /test.php


Apache log :
Code:
84.103.xxx.xxx - - [14/Jun/2024:08:18:33 +0200] "GET /favicon.ico HTTP/1.1" 404 2005 "https://dev.xxxxxx.cloud/test.php?code=0.AQwAJDMu3tBPA0i7lCy2LEWy9QpYbsxwWF5Hrz2Vu76zvO4MANc.AgABBAIAAADnfolhJpSnRYB1SVj-Hgd8AgDs_wUA9P_3Y6zJdaJIrc4scFbbFErN7BCueQ-2bx3G7cA-6dLRBgyIe_23b8xAx1wIQIWBNf45-5b67bkHSc56oWE5XdIP0BJffyrMJAOrTIhaHO95tQ8Rpmnluhzx3zqBjgi-Q3U6LBd__e49k0Qe48L0RGuIIysOJ2UcpRrgpmIp2AeLez51JjqfEKKsxB4jvhIxH5YMtTqnXxoFKuIiG8XzDFpXGPDxSWeZ1zhKcKgfB3yLsP2GtxJ0g97JSKLyFtft5CfYs0UHP0r_kduwGw--dDpnq1tmTQE1vqZhiX9mebWtALoqaM1ykbeD2RDL8bvVAy3Yu3vKmwOu4RwJjEXLojR1hrf8nsDLSTskxLRZWbAKFi7rZjxo30xaxTRPKXpv5Xrnd9MMrBQa8xKU52IdLUMvgEKD1mkvpKH871s1K2mRoB5LqVDamIgKcMpRqrfECIPCmVBDPfDjZJU12wzi5-S3965qkWWb1dIJNgQ4J0ar9THVBxCICrH-I2CgHfFuJ3Y7JXJIHG9tLcpyDDHWeXe5bKUBfvGRNEx0rsEp9L-ZpMeknx2ReCqToLFu_j-SL8LGYMGqSULtNJt2y5heRnns4vboAV5k_GX51y_fSvgPX-FhKqj9ERfirvXLQSwPJqpVdr88mGX9_9yKd-PP9MCg4LD6Mkd2oD8mpcOa6WmRN3EahoeMVM5qkS2Bt5Q4ATOOeWhJ_U_LPva_DRToswcKaTINSff6wWH5GCl7_be3Hk-gbZHtPNl-NaJsjuDqZcy8bjx5eU2mf3IARZWUFM7DeuOniPh3cP2t6MMT5Q1rqVmnJ2P_Me9xHFFMPqVt3-PIU5xlK7OpHO-tMVnYrVUBEHcG5zhlIy3oIeQSMgzhtkXdQF0b-r6yd6Db2ZHvMxn7eLedJtj_zoD111111111111111" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"


My Apache : Apache/2.4.57
OS : Debian 12

I'm desperate to find a solution to this problem...

Thanks for your help
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 333
Location: UK

PostPosted: Fri 14 Jun '24 16:43    Post subject: Reply with quote

You don't say what your upstream load balanced servers are running, but presume you've checked to see what corresponding error is being recorded there?

Your request URL alone is over 1K in length, and request headers in general have grown in size over time (multiple cookies, authentication tokens, X-headers, etc.), so is it possible you've exceeded some buffer size for some of your client requests?

In the past I've seen this problem with Tomcat backend servers, where we needed to increase maxHttpHeaderSize in server.xml. Other backend services will equally have some finite defaults.

Also, at the Apache level, check out ProxyIOBufferSize which defaults to 8192 bytes.
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Sat 15 Jun '24 8:17    Post subject: RE Reply with quote

Hello,

Sorry. I have an other apache serveur behind.

If I test directly the url with local IP in replacement of domain, it works.

Errors are logged on Proxy server, not on server behind.

I've already tested ProxyIOBufferSize directive.
It doesn't change anything.
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7329
Location: Germany, Next to Hamburg

PostPosted: Mon 17 Jun '24 10:53    Post subject: Reply with quote

You may set
Code:
connectiontimeout=[time in seconds] timeout=[time in seconds]


for example 60 seconds (1 min)
Code:

ProxyPass / http://localhost:4382/ connectiontimeout=60 timeout=60
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Wed 19 Jun '24 8:53    Post subject: Reply with quote

Also tested without any result...
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7329
Location: Germany, Next to Hamburg

PostPosted: Fri 21 Jun '24 8:48    Post subject: Reply with quote

Please post your reverse proxy config.
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Fri 21 Jun '24 10:32    Post subject: Reply with quote

Below the config

Code:

<VirtualHost *:443>
  ServerName www.xxxxxx.cloud
  ServerAlias xxxxxx.cloud

  ErrorLog /var/log/apache2/error.xxxxxx.cloud.log
  CustomLog /var/log/apache2/xxxxxx.cloud.log combined

  <Directory /var/www/xxxxxx.cloud/>
    Options Indexes FollowSymLinks
    AllowOverride All
    DirectoryIndex index.htm index.php index.html
  </Directory>

  Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

  <Proxy balancer://xxxxxx_prod>
    include /etc/apache2/xxxxxx_prod.conf
    ProxySet lbmethod=byrequests
    ProxySet stickysession=ROUTEID
    ProxySet nofailover=On
  </Proxy>

  ProxyPass "/"  "balancer://xxxxxx_prod/"
  ProxyPassReverse "/"  "balancer://xxxxxx_prod/"

  <Location /balancer-manager>
    SetHandler balancer-manager
    Order deny,allow
    Allow from all
  </Location>

  ProxyRequests Off
  ProxyPreserveHost On

  Include /etc/letsencrypt/options-ssl-apache.conf
  ServerAlias beta.xxxxxx.cloud
  ServerAlias oweba.xxxxxx.cloud
  SSLCertificateFile /etc/letsencrypt/live/www.xxxxxx.cloud/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/www.xxxxxx.cloud/privkey.pem
</VirtualHost>


And balancer settings

Code:

    BalancerMember "http://192.168.202.101" loadfactor=1 route=route1
    BalancerMember "http://192.168.202.102" loadfactor=1 route=route2
    BalancerMember "http://192.168.202.103" loadfactor=1 route=route3
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 333
Location: UK

PostPosted: Fri 21 Jun '24 21:32    Post subject: Reply with quote

Looking at your proxy configuration, and the error log detail in the original post, I notice the error relates to remote server 192.168.202.41:80 whereas your balancer member IP's are 192.168.202.101, 102 and 103. Is this relevant, or did you change the balancer IP's for the posted config?

Otherwise, if you've already tried increasing the buffer sizes, I suspect the problem lies somewhere with the proxied headers being fragmented / truncated, or relates to compression, but to analyse this further believe you need more detail to crack where the exact problem lies.

There are two things I'd try next here.

Firstly try removing the Accept-Encoding request header, so that the back end servers send an uncompressed response to Apache. Put the following in your virtual host block or global configuration (if acceptable).
Code:

RequestHeader unset Accept-Encoding   

Secondly, I'd use network capture tools such as tcpdump on your Debian platforms (Wireshark if the backend servers are Windows based), to see exactly what's going on here during the client request.

Per chance, is there a firewall between your Apache and the backend servers?
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Tue 25 Jun '24 9:35    Post subject: Reply with quote

We have 1 proxy and 3 environments (prod / dev /beta).
The configuration given is about Prod.

IP address 192.168.202.41 is for dev environment.

I've added "RequestHeader ..." but it doesn't change anything.

I'm checking with tcpdump but not simple to analyze.
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 333
Location: UK

PostPosted: Tue 25 Jun '24 12:55    Post subject: Reply with quote

Ok, so your one proxy covers multiple environments. Presumably, this error occurs across all three, albeit inconsistently?

Assuming you can narrow the problem to say the dev environment, you should be able to craft a filter for tcpdump to restrict the captured traffic (written to a pcap file). I'd personally just capture and analyse the traffic between the proxy and corresponding/potential back-end servers (hence the recommendation to remove the Accept-Encoding request header, so the proxied content isn't compressed).

I tend to transfer the pcap files over to a Windows platform, where it's easier to analyse them with Wireshark.

Let us know how you get on.
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Tue 25 Jun '24 14:12    Post subject: Reply with quote

Yes it's the same on each environment.

I try your method and give you feedback.
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Wed 26 Jun '24 10:31    Post subject: Reply with quote

We found a workaround.

We use proxy server in "normal web mode" for one subdomain.
We store parameters then we redirect to other script that recover the paramaters.

Thanks for your help
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 333
Location: UK

PostPosted: Wed 26 Jun '24 21:03    Post subject: Reply with quote

Not sure I fully follow your explantion, but am glad you've resolved the problem with a workaround.

Apache Lounge has numerous subscribers with many years of experience. Don't hesitate to post queries for help, or equally your expertise in solving problems.
Back to top
Renaud SALVIGNAC



Joined: 13 Jun 2024
Posts: 8
Location: FRANCE - Cognac

PostPosted: Thu 27 Jun '24 8:14    Post subject: Reply with quote

I will not miss it.

Thanks for your advices.
Back to top


Reply to topic   Topic: Apache Proxy errors AH01102 + AH00898 - Request too long View previous topic :: View next topic
Post new topic   Forum Index -> Apache