logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Webmaster Tools & Utilities View previous topic :: View next topic
Reply to topic   Topic: web crawler
Author
sailor



Joined: 17 Apr 2015
Posts: 82
Location: US

PostPosted: Tue 14 May '24 14:50    Post subject: web crawler Reply with quote

I need a web crawler that I can run from Windows or maybe a RedHat Linux 9 to prime a content security policy report.

It seems like curl and wget (which shows last updated in 2008), might work, but not sure if they are good enough. Also, some of the sites require a logon.
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7360
Location: Germany, Next to Hamburg

PostPosted: Wed 15 May '24 9:53    Post subject: Reply with quote

What are the specifications? Do you want to see the web headers? Download the website? Test SSL?
Back to top
sailor



Joined: 17 Apr 2015
Posts: 82
Location: US

PostPosted: Thu 16 May '24 15:38    Post subject: Reply with quote

I have content-security-policy setup in Apache with a report to url and want to prime the reporting location (and not wait many many days). I don't think I need the headers. It would be good to crawl the entire site. Yes, definitely by SSL.
Back to top


Reply to topic   Topic: web crawler View previous topic :: View next topic
Post new topic   Forum Index -> Webmaster Tools & Utilities