logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Apache mod_autoindex doesn't do gzip compression
Author
EIKA



Joined: 22 Jan 2019
Posts: 43
Location: US
PostPosted: Fri 17 Jul '20 23:04    Post subject: Apache mod_autoindex doesn't do gzip compression Reply with quote
I come across with issue I can't fix. I have Apache 2.4 with mod_autoindex (showing directory listings instead of index.html) and with mod_deflate.

I was able to get content gzipped, but only if they are files with extensions (e.g. .css, .html, etc.) But when I want to get gzipped page coming from mod_autoindex, I can't get it. E.g. https://domain.dom, https://domain.dom/somedir/, https://domain.dom/test/, etc. These URLs comes without Content-encoding: gzip header.

More details: https://stackoverflow.com/questions/62942077/apache-mod-autoindex-and-mod-deflate-gzip-cant-get-pages-compressed (I hope that external links are OK here).

Please advise.

Thanks!
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7294
Location: Germany, Next to Hamburg
PostPosted: Mon 05 Oct '20 10:10    Post subject: Reply with quote
The following works

Code:

<IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/html text/plain text/xml text/php text/css text/js text/javascript text/javascript-x application/x-javascript font/truetype
        AddOutputFilterByType DEFLATE application/javascript
                AddOutputfilterByType DEFLATE application/vnd.mozilla.xul+xml
        DeflateCompressionLevel 9
</IfModule>
Back to top
EIKA



Joined: 22 Jan 2019
Posts: 43
Location: US
PostPosted: Mon 05 Oct '20 10:14    Post subject: Reply with quote
This works for what? For directory listings via mod_autoindex? I don't think so, because this is very standard config that covers only files with extensions.

Can you confirm that it will for for e.g. for URI "/" generated by mod_autoindex. I bet that you cannot!
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7294
Location: Germany, Next to Hamburg
PostPosted: Mon 05 Oct '20 10:29    Post subject: Reply with quote
I did that on my server.

I can prove it see https://imgur.com/cng61kp

The code that I posted is at the end of my httpd.conf

Sure you mod_brotli encoding, since I prefer that Wink
Code:

<IfModule mod_deflate.c>
        AddOutputFilterByType BROTLI;DEFLATE text/html text/plain text/xml text/php text/css text/js text/javascript text/javascript-x application/x-javascript font/truetype
        AddOutputFilterByType BROTLI;DEFLATE application/javascript
                AddOutputfilterByType BROTLI;DEFLATE application/vnd.mozilla.xul+xml
        DeflateCompressionLevel 9
</IfModule>
Back to top
EIKA



Joined: 22 Jan 2019
Posts: 43
Location: US
PostPosted: Mon 05 Oct '20 11:03    Post subject: Reply with quote
Sadly I see no your healers in full. And I can't locate content-encoding hear in your screenshot.

Could you please copy/paste your headers in text where I can see content-encoding: gzip for directory?
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7294
Location: Germany, Next to Hamburg
PostPosted: Mon 05 Oct '20 11:57    Post subject: Reply with quote
Code:
access-control-allow-origin
   *
content-encoding
   br
content-length
   172
content-security-policy
   default-src 'self' 'unsafe-inline' data:; font-src 'self' 'unsafe-inline' fonts.gstatic.com data:; style-src 'self' 'unsafe-inline' fonts.googleapis.com; script-src 'self' 'unsafe-inline' 'unsafe-eval';
content-type
   text/html;charset=ISO-8859-1
date
   Mon, 05 Oct 2020 09:55:09 GMT
expect-ct
   max-age=86400, enforce
feature-policy
   geolocation 'none'; midi 'none'; camera 'none'; usb 'none'; magnetometer 'none'; accelerometer 'none'; vr 'none'; speaker 'none'; ambient-light-sensor 'none'; gyroscope 'none'; microphone 'none'
server
   mario
strict-transport-security
   max-age=15553000; preload
vary
   Accept-Encoding
x-content-type-options
   nosniff
X-Firefox-Spdy
   h2
x-frame-options
   SAMEORIGIN
x-xss-protection
   1; mode=block


https://imgur.com/a/JPvSNX7

I added some red color around it in the screen shot. Note that br is brotli endoding
Back to top
EIKA



Joined: 22 Jan 2019
Posts: 43
Location: US
PostPosted: Mon 05 Oct '20 12:53    Post subject: Reply with quote
Thanks, now it's more clear. But I have no ideas what is the BR. I am using gzip on site-wide basis and want to add it to mod_indexed directories. Without changing encoding type/compressor module.

Do you have ideas how to run gzip for mod_indexed directories?

I tried to as on Apache mail list as well and got a few suggestions, but was unable to implement them (maybe that I did something wrong and/or suggestion was wrong). I don't know.

Here they are:

Quote:
> Hi folks!
>
> I come across with issue I can't fix. I have Apache/2.4.43
> OpenSSL/1.1.1g with mod_autoindex (showing directory listings instead
> of index.html) and with mod_deflate.
>
> I was able to get content gzipped, but only if they are files with
> extensions (e.g. .css, .html, etc.) But when I want to get gzipped
> page coming from mod_autoindex, I can't get it. E.g.
> https://domain.dom, https://domain.dom/somedir/,
> https://domain.dom/test/, etc. These URLs comes without Content-encoding: gzip header.
>
> I think that I am not far from the reason why, because I found that
> directory indexes sent by chunks (transfer-encoding: chunked). But
> files with extensions came with these headers:

I suspect that you have simply configured mod_gzip to be only active for URLs ending in .html etc, and the transfer-encoding just happens to correlate with that because it's not a local "file" but generated dynamically. You should check the outgoing content-type instead of the URL. We've been using the following for quite some time:

FilterProvider buffer BUFFER "%{CONTENT_TYPE} =~ m|^text/|"
FilterProvider buffer BUFFER "%{CONTENT_TYPE} =~ m|^model/|"
[... more content types ]
FilterProvider gzip_compression DEFLATE "%{CONTENT_TYPE} =~ m|^text/|"
FilterProvider gzip_compression DEFLATE "%{CONTENT_TYPE} =~ m|^model/|"
[...]


Quote:
Thanks for your response.

Could you please help a bit more?

Here is my Apache Deflate configured:

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/php
AddOutputFilterByType DEFLATE image/x-icon
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xml

And these directives works like charm for html, css, js, files, etc.

I tried to add:

FilterDeclare COMPRESS
FilterProvider COMPRESS DEFLATE "%{Content_Type} = 'text/html'"
FilterProvider COMPRESS DEFLATE "%{Content_Type} = 'text\/html.*$'"
FilterProvider COMPRESS DEFLATE "%{Content_Type} = 'text/plain'"
FilterProvider COMPRESS DEFLATE "%{Content_Type} = 'text/php'"
FilterChain COMPRESS gzip_compression
FilterProtocol COMPRESS DEFLATE change=yes;byteranges=no

But still getting chunked responses. I am sending requests to URI '/' and '/TEST/'. And Content-Type for these addresses are "text/html;charset=UTF-8".

Also, I have no ideas why Content-Type is 'text/html;charset=UTF-8', but not just 'text/html'.

Here is header example from LRT plugin for Chrome:

------------------------------------------------------
[
{
"url": "https://domain.dom/",
"redirect": false,
"redirectType": "",
"statusCode": 200,
"statusLine": "HTTP/1.1 200 OK",
"headers": [
{
"name": "Date",
"value": "Mon, 03 Aug 2020 19:18:19 GMT"
},
{
"name": "Server",
"value": "Apache/2.4"
},
{
"name": "Accept-Ranges",
"value": "none"
},
{
"name": "Vary",
"value": "Accept-Encoding,User-Agent"
},
{
"name": "Cache-Control",
"value": "max-age=86400, private, must-revalidate"
},
{
"name": "X-Permitted-Cross-Domain-Policies",
"value": "none"
},
{
"name": "X-Content-Type-Options",
"value": "nosniff"
},
{
"name": "X-XSS-Protection",
"value": "1; mode=block"
},
{
"name": "X-Frame-Options",
"value": "deny"
},
{
"name": "Content-Language",
"value": "en"
},
{
"name": "Keep-Alive",
"value": "timeout=5, max=95"
},
{
"name": "Connection",
"value": "Keep-Alive"
},
{
"name": "Transfer-Encoding",
"value": "chunked"
},
{
"name": "Content-Type",
"value": "text/html;charset=UTF-8"
}
],
}
]
------------------------------------------------------

Please advise what could be wrong.


Quote:
I don't think you can use regular expressions with just '=', you'll have to use '=~'

search 'regex' in the documentation at
https://httpd.apache.org/docs/2.4/expr.html for the exact syntax required.


Quote:
This is what I tried:

FilterDeclare COMPRESS
FilterProvider COMPRESS DEFLATE "%{CONTENT_TYPE} =~ m#text/html#"
FilterProvider COMPRESS DEFLATE "%{CONTENT_TYPE} =~ m#text\/html.*$#"
FilterProvider COMPRESS DEFLATE "%{CONTENT_TYPE} =~ m#text\/html\.*$#"
FilterChain COMPRESS gzip_compression
FilterProtocol COMPRESS DEFLATE change=yes;byteranges=no

And still the same. I.e. chunked content Sad

Is my syntax incorrect and doesn't cover [Content-Type is 'text/html;charset=UTF-8'] header?


And discussion has stopped with no any updates during a month or even more.

Can anybody check regexp syntax here?

Code:
FilterDeclare   COMPRESS   
FilterProvider   COMPRESS   DEFLATE "%{CONTENT_TYPE} =~ m#text/html#"
FilterProvider   COMPRESS   DEFLATE "%{CONTENT_TYPE} =~ m#text\/html.*$#"
FilterProvider   COMPRESS   DEFLATE "%{CONTENT_TYPE} =~ m#text\/html\.*$#"
FilterChain      COMPRESS   gzip_compression
FilterProtocol   COMPRESS   DEFLATE change=yes;byteranges=no
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7294
Location: Germany, Next to Hamburg
PostPosted: Mon 05 Oct '20 13:26    Post subject: Reply with quote
EIKA wrote:
But I have no ideas what is the BR.

As I wrote above BR is brotli. https://httpd.apache.org/docs/current/mod/mod_brotli.html

EIKA wrote:

I am using gzip on site-wide basis and want to add it to mod_indexed directories. Without changing encoding type/compressor module.


I think you mean mod_autoindex.

EIKA wrote:

Do you have ideas how to run gzip for mod_indexed directories?


As I posted above. Add this at the end of the httpd.conf file.
Code:

<IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/html text/plain text/xml text/php text/css text/js text/javascript text/javascript-x application/x-javascript font/truetype
        AddOutputFilterByType DEFLATE application/javascript
                AddOutputfilterByType DEFLATE application/vnd.mozilla.xul+xml
        DeflateCompressionLevel 9
</IfModule>


Make sure mod_deflate is loaded.

in doubt run httpd.exe -M
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 312
Location: UK
PostPosted: Thu 08 Oct '20 21:54    Post subject: Reply with quote
I've had fun with filters in the past, and they do take a bit of getting used to.

In the Apache documentation for filters https://httpd.apache.org/docs/2.4/filter.html, the Using Filters section explains the difference between Simple and Dynamic filtering. It also says although the directive AddOutputFilterByType is still supported, it's deprecated, and to use Dynamic filtering instead.

Dynamic filtering can only be used for output filters, but that's all EIKA needs here.

This sample code seems to work as requested for directory listings (on my test server), using Brotli if the browser Accept-Encoding header requests it, else gzip from the deflate module.

Code:
# Load extra modules needed.
#
<IfModule !brotli_module>
    LoadModule brotli_module modules/mod_brotli.so
</IfModule>

<IfModule !deflate_module>
    LoadModule deflate_module modules/mod_deflate.so
</IfModule>

<IfModule !filter_module>
    LoadModule filter_module modules/mod_filter.so
</IfModule>

# Fancy directory listings
#
Include conf/extra/httpd-autoindex.conf

# Brotli filter.
#
FilterDeclare Brotli
FilterProvider Brotli Brotli_Compress "%{CONTENT_TYPE} =~ m%^application/(javascript||xml)%i"
FilterProvider Brotli Brotli_Compress "%{CONTENT_TYPE} =~ m%^text/(css|html|javascript|php|plain|xml)%i"
FilterProtocol Brotli change=yes;byteranges=no
FilterChain Brotli

# Deflate (gzip) filter.
#
FilterDeclare Compress
FilterProvider Compress Deflate "%{CONTENT_TYPE} =~ m%^application/(javascript||xml)%i"
FilterProvider Compress Deflate "%{CONTENT_TYPE} =~ m%^text/(css|html|javascript|php|plain|xml)%i"
FilterProtocol Compress change=yes;byteranges=no
FilterChain Compress


Since FilterProvider supports regex (through ap_expr), we can simplify the content type logic to the lines as shown. Also, since the regex is a match substring tied to the begining of the variable, it should catch text/html;charset=UTF-8, as well as text/html.

You can easily add more content types to the regex as required.

Finally, if it helps anyone with local/legacy testing, you can force Firefox to support Brotli for http requests rather than just for https, by using about:config and search for accept-encoding.
Back to top


Reply to topic   Topic: Apache mod_autoindex doesn't do gzip compression View previous topic :: View next topic
Post new topic   Forum Index -> Apache