ray/scripts
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update blc script to specify headers as strings

Browse source 
This makes the code a bit more readable. We then map over the list of
strings to produce the parsed format the web client requires.
This commit is contained in:
Ray Miller 2025-01-31 15:12:54 +00:00
parent 0f09a25063
commit 8c4df89483
1 changed files with 7 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
guile/broken-link-checker.scm
View file

@ -23,6 +23,7 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
(web client) (web client)
(web response) (web response)
(web uri) (web uri)
((web http) #:select (parse-header))
((htmlprag) #:select (html->sxml)) ((htmlprag) #:select (html->sxml))
((sxml xpath) #:select (sxpath)) ((sxml xpath) #:select (sxpath))
(srfi srfi-1) (srfi srfi-1)
@ -177,10 +178,12 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
;; Some sites return 403 errors for bot requests, these headers make ;; Some sites return 403 errors for bot requests, these headers make
;; us look more like a real browser. ;; us look more like a real browser.
(define request-headers '((accept . ((text/html) (application/xhtml+xml) (application/xml (q . 900)) (*/* (q . 800)))) (define request-headers
(accept-encoding . ((1000 . "gzip") (1000 . "deflate") (1000 . "br") (1000 . "zstd"))) (map (match-lambda ((k . v) (cons k (parse-header k v))))
(accept-language . ((1000 . "en-GB") (500 . "en"))) '((accept . "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
(user-agent . "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"))) (accept-encoding . "gzip, deflate, br, zstd")
(accept-language . "en-GB,en;q=0.5")
(user-agent . "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"))))
;; A memoized version of http-get. This allows us to quickly check a URL we have ;; A memoized version of http-get. This allows us to quickly check a URL we have
;; seen before (when it is linked from multiple different pages) without generating ;; seen before (when it is linked from multiple different pages) without generating