Update libcurl-impersonate usage

[wrobelda]

Overhaul the usage of libcurl-impersonate, which was not, in fact, being used properly, as the code was overriding the headers set by it to prevent detection.

• update the libcurl-impersonate to an actively managed lexiforest fork
• impersonate Chrome 131
• move the defaultHttpHeaders to http.php, where it belongs
• only set defaultHttpHeaders if curl-impersonate is not detected
• make useragent ini setting optional and disabled by default
• add necessary documentation updates

[dvikan]

i was hoping we could keep CurlHttpClient cleanly separated by treating it as a thirdparty http client

this change removes the http user agent for non-curl-impersonator usage which might break some fetching. (default ua is Curl v1.2.3 or simply the empty string i dont remember)

this change breaks the usage of custom http headers for non-curl-impersonate uses (no it doesnt, i read code wrong)

[wrobelda]

i was hoping we could keep CurlHttpClient cleanly separated by treating it as a thirdparty http client

I can see that, but you could also make a point that getContents() in contents.php could be kept clean of the actual underlying anti-bot measures, which is the purpose of the default headers you had there already.

Also, I would imagine any alternative client would need to incorporate some of their own anti-bot measures, possibly with another set of headers, so if we consider the current set of default headers as curl-specific, I think it makes sense to keep them all sealed off in CurlHttpClient.

It would be hard to fix the libcurl-impersonate usage without the refactoring I did.

this change removes the http user agent for non-curl-impersonator usage which might break some fetching. (default ua is Curl v1.2.3 or simply the empty string i dont remember)

Indeed, fixed it.

[dvikan]

thanks for helping to keep this up to date.