logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Other Software View previous topic :: View next topic
Reply to topic   Topic: Trying to add X-Robots-Tag HTTP header on Content-Type: appl
Author
arakpt



Joined: 25 Jul 2021
Posts: 9
Location: jakarta

PostPosted: Sun 25 Jul '21 14:37    Post subject: Trying to add X-Robots-Tag HTTP header on Content-Type: appl Reply with quote

hi

i have feed pages that i want to set noindex using X-Robots-Tag

the permalinks is something like this

Code:
http://domain/feed
http://domain/post-about/feed


its an rss page with permalinks made by cms

i add this code

Code:
<FilesMatch "\.(xml|rss)$">
Header set X-Robots-Tag "noindex"
</Files>


but doesnt work, response header return no X-robot-Tag

Code:
Final response
< HTTP/1.1 200 OK
< Date: Sun, 25 Jul 2021 10:27:42 GMT
< Content-Type: application/rss+xml; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/7.3.10
< Last-Modified: Sun, 25 Jul 2021 10:28:01 GMT
< Set-Cookie: pvc_visits[0]=1627295281b60; expires=Mon, 26-Jul-2021 10:28:01 GMT; Max-Age=86400; path=/; secure; HttpOnly
< Link: ; rel="https://api.w.org/", ; rel="alternate"; type="application/json", ; rel=shortlink
< Cache-Control: max-age=3600
< Expires: Sun, 25 Jul 2021 11:28:00 GMT
< Vary: Accept-Encoding,User-Agent
< CF-Cache-Status: DYNAMIC
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< NEL: {"report_to":"cf-nel","max_age":604800}
< X-Content-Type-Options: nosniff
< Server: cloudflare
< CF-RAY: 6744b
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400


how do i match/set robots tag with Content-Type: application/rss+xml; charset=UTF-8?

thank you
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Sun 25 Jul '21 20:45    Post subject: Reply with quote

Try the following line (no <FilesMatch> directive needed).

Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss+xml#"
Back to top
arakpt



Joined: 25 Jul 2021
Posts: 9
Location: jakarta

PostPosted: Mon 26 Jul '21 1:58    Post subject: Reply with quote

Hi tangent

i add the code in my htaccess but apparently still no X-Robots-Tag appear
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Mon 26 Jul '21 17:56    Post subject: Reply with quote

I tested this header set directive before posting, adding the following to the global section of the default httpd.conf file, and it worked.

Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#"

Can you try this rather than using an .htaccess file, and if it works then revise the match string to your required content type?
Back to top
arakpt



Joined: 25 Jul 2021
Posts: 9
Location: jakarta

PostPosted: Tue 27 Jul '21 7:34    Post subject: Reply with quote

tangent wrote:
I tested this header set directive before posting, adding the following to the global section of the default httpd.conf file, and it worked.

Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#"

Can you try this rather than using an .htaccess file, and if it works then revise the match string to your required content type?


ok i edit it on global httpd.conf ...this one is worked,
Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^text/html#"


Requesting: http://localhost/mit

Code:
Final response
< HTTP/1.1 200 OK
< Date: Tue, 27 Jul 2021 05:28:14 GMT
< Server: Apache/2.4.41 (Win64) OpenSSL/1.1.1c PHP/7.3.10
< X-Robots-Tag: noindex
< X-Powered-By: PHP/7.3.10
< Link: ; rel="https://api.w.org/", ; rel="alternate"; type="application/json", ; rel=shortlink
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8



but rss+xml does not work
Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss+xml#"


Requesting: http://localhost/mit/feed
Code:
Final response
< HTTP/1.1 200 OK
< Date: Tue, 27 Jul 2021 05:22:40 GMT
< Server: Apache/2.4.41 (Win64) OpenSSL/1.1.1c PHP/7.3.10
< Upgrade: h2,h2c
< Connection: Upgrade
< X-Powered-By: PHP/7.3.10
< Last-Modified: Fri, 23 Jul 2021 05:40:53 GMT
< ETag: "467b59d5bd802cdc31a85787da531b30"
< Link: ; rel="https://api.w.org/"
< Transfer-Encoding: chunked
< Content-Type: application/rss+xml; charset=UTF-8


content type for feed atom are application/rss+xml right?

could it be it bacause i dont have any mime type for rss+xml in mime_module

Code:
<IfModule mime_module>

    TypesConfig conf/mime.types

    AddType application/x-gzip .tgz

    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #AddHandler cgi-script .cgi .pl

#For type maps (negotiated resources):
    #AddHandler type-map var

    #AddType text/html .shtml
    #AddOutputFilter INCLUDES .shtml
</IfModule>



====edit===

this rule worked

Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss#"


thank you @tangent
Back to top
mraddi



Joined: 27 Jun 2016
Posts: 149
Location: Schömberg, Baden-Württemberg, Germany

PostPosted: Tue 27 Jul '21 8:56    Post subject: Reply with quote

Good morning,

as you are working with regular expressions I guess you have to escape the + with a \ so it reads
Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#"

because the + is a special character in regular expressions (= the character to its left to it appears once or multiple times)
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Tue 27 Jul '21 15:59    Post subject: Reply with quote

Thanks, mraddi, my mistake.

I should have escaped the '+' symbol in the regex.
Back to top
arakpt



Joined: 25 Jul 2021
Posts: 9
Location: jakarta

PostPosted: Fri 30 Jul '21 7:41    Post subject: Reply with quote

mraddi wrote:
Good morning,

as you are working with regular expressions I guess you have to escape the + with a \ so it reads
Code:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#"

because the + is a special character in regular expressions (= the character to its left to it appears once or multiple times)


hi mraddi

another question , on apache 2.4 the code above works

but in older version apache it throw error and apache unable to start
Code:
Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7288
Location: Germany, Next to Hamburg

PostPosted: Fri 30 Jul '21 10:04    Post subject: Reply with quote

arakpt wrote:

but in older version apache it throw error and apache unable to start
Code:
Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar


You may post that line. We are not fortune tellers Wink
Back to top
arakpt



Joined: 25 Jul 2021
Posts: 9
Location: jakarta

PostPosted: Fri 30 Jul '21 13:09    Post subject: Reply with quote

James Blond wrote:
arakpt wrote:

but in older version apache it throw error and apache unable to start
Code:
Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar


You may post that line. We are not fortune tellers Wink


hi James

line 1007 on my httpd.conf are this code

Quote:
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#"



but this one are different server and still using older version apache, the previous one are on windows server
Code:
<VirtualHost *:80>
    ServerAdmin webmaster@test.com
    DocumentRoot "/opt/test"
Header always set X-Robots-Tag "noindex" "expr=%{CONTENT_TYPE} =~ m#^application/rss\+xml#"
   ServerName www.test.com
    ServerAlias test.com
    ErrorLog "/var/logs/test.com-error_log"
    CustomLog "/var/logs/test.com-access_log" common
</VirtualHost>
Back to top
Jan-E



Joined: 09 Mar 2012
Posts: 1248
Location: Amsterdam, NL, EU

PostPosted: Fri 30 Jul '21 15:01    Post subject: Reply with quote

James Blond wrote:
arakpt wrote:

but in older version apache it throw error and apache unable to start
Code:
Starting httpd: Syntax error on line 1007 of /etc/httpd/conf/httpd.conf:
error: envclause should be in the form env=envar

You may post that line. We are not fortune tellers Wink
And tell what version is “older version”.
Back to top
tangent
Moderator


Joined: 16 Aug 2020
Posts: 305
Location: UK

PostPosted: Fri 30 Jul '21 18:01    Post subject: Reply with quote

With Apache 2.2 series, the mod_headers module does not support 'expr' qualifiers (which can pick up %{CONTENT_TYPE}); it only supports conditionals based on environment variables.

Further, in Apache 2.2 SetEnvIf only sets environment variables based on request characteristics, not the response, so it can't pick up on the response content type and use it to set a conditional variable.

The only option I can think of (other forum members may suggest another approach), would be to write a custom output filter if you're up for it, activated with the appropriate content type, which could then add the X-Robots-Tag header. e.g.

Code:
<IfVersion < 2.4>
  FilterDeclare X-Robots-Tag
  FilterProvider X-Robots-Tag MyCustomFilter resp=Content-Type $application/rss
  FilterChain X-Robots-Tag
</IfVersion>


Alternatively, could you not update the older Apache instance, seeing as you already have a working 2.4 series configuration?
Back to top


Reply to topic   Topic: Trying to add X-Robots-Tag HTTP header on Content-Type: appl View previous topic :: View next topic
Post new topic   Forum Index -> Other Software