logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: blocking Applebot
Author
Doug22



Joined: 02 Jun 2013
Posts: 57
Location: Houston TX

PostPosted: Thu 12 Jan '23 5:13    Post subject: blocking Applebot Reply with quote

So I'm trying to block web bots, and my robots.txt file seems to be doing a pretty good job of that EXCEPT for Applebot. I've put

Code:
User-agent: Applebot
Disallow: /


in my robots.txt file,but Applebot seems to be grabbing my stuff anyway. In fact, when I search my archive for robots.txt, Applebot never seems to be accessing it. Googlebot and Yandex at least do so regularly.

What's going on? Does Applebot check on robots.text once a week,or once a month,or what?
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg

PostPosted: Thu 12 Jan '23 13:40    Post subject: Reply with quote

If it doesn't you can block it by the user agent

Code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT}  ^.*Applebot.*$
RewriteRule . - [R=403,L]
Back to top
Doug22



Joined: 02 Jun 2013
Posts: 57
Location: Houston TX

PostPosted: Thu 12 Jan '23 15:24    Post subject: Reply with quote

Yes, thank you. I understand that is a straightforward possibility. But Apple formally says that its AppleBot obeys robots.txt. Not seeing it.
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg

PostPosted: Thu 12 Jan '23 17:09    Post subject: Reply with quote

Doug22 wrote:
But Apple formally says that its AppleBot obeys robots.txt. Not seeing it.


So what is your question now?
Back to top
Doug22



Joined: 02 Jun 2013
Posts: 57
Location: Houston TX

PostPosted: Thu 12 Jan '23 19:56    Post subject: Reply with quote

Just pointing out that Apple says that AppleBot pays attention to robots.txt, but it really doesn't. I was originally asking if anyone knew something I didn't know about why that seems to be.
Back to top


Reply to topic   Topic: blocking Applebot View previous topic :: View next topic
Post new topic   Forum Index -> Apache