| Author |
|
arakpt
Joined: 25 Jul 2021
Posts: 9
Location: jakarta
|
 Posted: Sun 25 Jul '21 14:37 Post subject: Trying to add X-Robots-Tag HTTP header on Content-Type: appl |
 |
|
hi
i have feed pages that i want to set noindex using X-Robots-Tag
the permalinks is something like this
| Code: | http://domain/feed
http://domain/post-about/feed |
its an rss page with permalinks made by cms
i add this code
| Code: | <FilesMatch "\.(xml|rss)$">
Header set X-Robots-Tag "noindex"
</Files> |
but doesnt work, response header return no X-robot-Tag
| Code: | Final response
< HTTP/1.1 200 OK
< Date: Sun, 25 Jul 2021 10:27:42 GMT
< Content-Type: application/rss+xml; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/7.3.10
< Last-Modified: Sun, 25 Jul 2021 10:28:01 GMT
< Set-Cookie: pvc_visits[0]=1627295281b60; expires=Mon, 26-Jul-2021 10:28:01 GMT; Max-Age=86400; path=/; secure; HttpOnly
< Link: ; rel="https://api.w.org/", ; rel="alternate"; type="application/json", ; rel=shortlink
< Cache-Control: max-age=3600
< Expires: Sun, 25 Jul 2021 11:28:00 GMT
< Vary: Accept-Encoding,User-Agent
< CF-Cache-Status: DYNAMIC
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< NEL: {"report_to":"cf-nel","max_age":604800}
< X-Content-Type-Options: nosniff
< Server: cloudflare
< CF-RAY: 6744b
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400 |
how do i match/set robots tag with Content-Type: application/rss+xml; charset=UTF-8?
thank you |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 397
Location: UK
|
| Posted: Sun 25 Jul '21 20:45 Post subject: |
|
|
Try the following line (no <FilesMatch> directive needed).
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss+xml#" |
|
|
| Back to top |
|
arakpt
Joined: 25 Jul 2021
Posts: 9
Location: jakarta
|
| Posted: Mon 26 Jul '21 1:58 Post subject: |
|
|
Hi tangent
i add the code in my htaccess but apparently still no X-Robots-Tag appear |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 397
Location: UK
|
| Posted: Mon 26 Jul '21 17:56 Post subject: |
|
|
I tested this header set directive before posting, adding the following to the global section of the default httpd.conf file, and it worked.
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#" |
Can you try this rather than using an .htaccess file, and if it works then revise the match string to your required content type? |
|
| Back to top |
|
arakpt
Joined: 25 Jul 2021
Posts: 9
Location: jakarta
|
| Posted: Tue 27 Jul '21 7:34 Post subject: |
|
|
| tangent wrote: | I tested this header set directive before posting, adding the following to the global section of the default httpd.conf file, and it worked.
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#" |
Can you try this rather than using an .htaccess file, and if it works then revise the match string to your required content type? |
ok i edit it on global httpd.conf ...this one is worked, | Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#" |
Requesting: http://localhost/mit
| Code: | Final response
< HTTP/1.1 200 OK
< Date: Tue, 27 Jul 2021 05:28:14 GMT
< Server: Apache/2.4.41 (Win64) OpenSSL/1.1.1c PHP/7.3.10
< X-Robots-Tag: noindex
< X-Powered-By: PHP/7.3.10
< Link: ; rel="https://api.w.org/", ; rel="alternate"; type="application/json", ; rel=shortlink
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8 |
but rss+xml does not work | Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss+xml#" |
Requesting: http://localhost/mit/feed
| Code: | Final response
< HTTP/1.1 200 OK
< Date: Tue, 27 Jul 2021 05:22:40 GMT
< Server: Apache/2.4.41 (Win64) OpenSSL/1.1.1c PHP/7.3.10
< Upgrade: h2,h2c
< Connection: Upgrade
< X-Powered-By: PHP/7.3.10
< Last-Modified: Fri, 23 Jul 2021 05:40:53 GMT
< ETag: "467b59d5bd802cdc31a85787da531b30"
< Link: ; rel="https://api.w.org/"
< Transfer-Encoding: chunked
< Content-Type: application/rss+xml; charset=UTF-8 |
content type for feed atom are application/rss+xml right?
could it be it bacause i dont have any mime type for rss+xml in mime_module
| Code: | <IfModule mime_module>
TypesConfig conf/mime.types
AddType application/x-gzip .tgz
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
#AddHandler cgi-script .cgi .pl
#For type maps (negotiated resources):
#AddHandler type-map var
#AddType text/html .shtml
#AddOutputFilter INCLUDES .shtml
</IfModule> |
====edit===
this rule worked
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss#" |
thank you @tangent |
|
| Back to top |
|
mraddi
Joined: 27 Jun 2016
Posts: 152
Location: Schömberg, Baden-Württemberg, Germany
|
| Posted: Tue 27 Jul '21 8:56 Post subject: |
|
|
Good morning,
as you are working with regular expressions I guess you have to escape the + with a \ so it reads
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#" |
because the + is a special character in regular expressions (= the character to its left to it appears once or multiple times) |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 397
Location: UK
|
| Posted: Tue 27 Jul '21 15:59 Post subject: |
|
|
Thanks, mraddi, my mistake.
I should have escaped the '+' symbol in the regex. |
|
| Back to top |
|
arakpt
Joined: 25 Jul 2021
Posts: 9
Location: jakarta
|
| Posted: Fri 30 Jul '21 7:41 Post subject: |
|
|
| mraddi wrote: | Good morning,
as you are working with regular expressions I guess you have to escape the + with a \ so it reads
| Code: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#" |
because the + is a special character in regular expressions (= the character to its left to it appears once or multiple times) |
hi mraddi
another question , on apache 2.4 the code above works
but in older version apache it throw error and apache unable to start
| Code: | Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar |
|
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7442
Location: EU, Germany, Next to Hamburg
|
| Posted: Fri 30 Jul '21 10:04 Post subject: |
|
|
| arakpt wrote: |
but in older version apache it throw error and apache unable to start
| Code: | Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar |
|
You may post that line. We are not fortune tellers  |
|
| Back to top |
|
arakpt
Joined: 25 Jul 2021
Posts: 9
Location: jakarta
|
| Posted: Fri 30 Jul '21 13:09 Post subject: |
|
|
| James Blond wrote: | | arakpt wrote: |
but in older version apache it throw error and apache unable to start
| Code: | Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar |
|
You may post that line. We are not fortune tellers |
hi James
line 1007 on my httpd.conf are this code
| Quote: | | Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#" |
but this one are different server and still using older version apache, the previous one are on windows server
| Code: | <VirtualHost *:80>
ServerAdmin webmaster@test.com
DocumentRoot "/opt/test"
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#"
ServerName www.test.com
ServerAlias test.com
ErrorLog "/var/logs/test.com-error_log"
CustomLog "/var/logs/test.com-access_log" common
</VirtualHost> |
|
|
| Back to top |
|
Jan-E
Joined: 09 Mar 2012
Posts: 1283
Location: Amsterdam, NL, EU
|
| Posted: Fri 30 Jul '21 15:01 Post subject: |
|
|
| James Blond wrote: | | arakpt wrote: |
but in older version apache it throw error and apache unable to start
| Code: | Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar |
|
You may post that line. We are not fortune tellers |
And tell what version is “older version”. |
|
| Back to top |
|
tangent
Moderator
Joined: 16 Aug 2020
Posts: 397
Location: UK
|
| Posted: Fri 30 Jul '21 18:01 Post subject: |
|
|
With Apache 2.2 series, the mod_headers module does not support 'expr' qualifiers (which can pick up %{CONTENT_TYPE}); it only supports conditionals based on environment variables.
Further, in Apache 2.2 SetEnvIf only sets environment variables based on request characteristics, not the response, so it can't pick up on the response content type and use it to set a conditional variable.
The only option I can think of (other forum members may suggest another approach), would be to write a custom output filter if you're up for it, activated with the appropriate content type, which could then add the X-Robots-Tag header. e.g.
| Code: | <IfVersion < 2.4>
FilterDeclare X-Robots-Tag
FilterProvider X-Robots-Tag MyCustomFilter resp=Content-Type $application/rss
FilterChain X-Robots-Tag
</IfVersion> |
Alternatively, could you not update the older Apache instance, seeing as you already have a working 2.4 series configuration? |
|
| Back to top |
|
