Comments (6)
I see extra headers for Connection:, Cache-Control:, and also DNT
I just tested Chrome 103 with Wireshark and the SSLKEYLOGFILE
env var set and browsed to www.wikipedia.org. I didn't get these headers. It might have something to do with the local socat or the use of HTTP/1.1 (Connection
for example is not used in HTTP/2). Anyway I wouldn't worry about adding them unless it looks absolutely necessary for fetching the website you need.
Also the sec-ch-ua: string seems to be a bit different
Yes it changes every release.
the Connection: and Cache-Control: headers end up at the end, despite my array passing them first
You are correct, this is a bug with my implementation. The user-supplied headers are always added after the built-in list of headers that curl-impersonate uses. I'll open a separate issue for that.
I tried editing the curl_chrome101 script to add these 2 headers in there, and same behavior: they end up at the bottom of the headers instead of top.
I just tried that and the order looks fine, same as supplied in the script. Can you verify again?
from curl-impersonate.
Opened #90
from curl-impersonate.
Here is what I get when manually editing the curl_chrome101 script:
- sudo nano /usr/local/bin/curl_chrome101
- add in the header lines so the script looks like this:
#!/bin/bash
# Find the directory of this script
dir=${0%/*}
# The list of ciphers can be obtained by looking at the Client Hello message in
# Wireshark, then converting it using this reference
# https://wiki.mozilla.org/Security/Cipher_Suites
"$dir/curl-impersonate-chrome" \
--ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY130
5_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-A
ES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE
-RSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-SHA,ECDHE-RSA-AES256-SHA,AES128-GCM-SHA256,AES256-GCM-SHA384,AES128-SHA,AES256-SHA \
-H 'Connection: keep-alive' \
-H 'Cache-Control: max-age=0' \
-H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "Windows"' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: none' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Accept-Language: en-US,en;q=0.9' \
--http2 --false-start --compressed \
--tlsv1.2 --no-npn --alps \
--cert-compression brotli \
"$@"
- run socat in one shell: socat -v openssl-listen:8443,reuseaddr,fork,cert=server.pem,verify=0 echo
- In another shell, connect: curl_chrome101 -k https://localhost:8443
- result:
> 2022/07/24 09:45:05.293892 length=689 from=0 to=688
GET / HTTP/1.1\r
Host: localhost:8443\r
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"\r
sec-ch-ua-mobile: ?0\r
sec-ch-ua-platform: "Windows"\r
Upgrade-Insecure-Requests: 1\r
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/101.0.4951.67 Safari/537.36\r
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/w
ebp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r
Sec-Fetch-Site: none\r
Sec-Fetch-Mode: navigate\r
Sec-Fetch-User: ?1\r
Sec-Fetch-Dest: document\r
Accept-Encoding: gzip, deflate, br\r
Accept-Language: en-US,en;q=0.9\r
Connection: keep-alive\r
Cache-Control: max-age=0\r
from curl-impersonate.
I just tried exactly the same and I get the correct order. Is it possible that you have the CURL_IMPERSONATE
environment variable set as well? Because that could cause that (i.e. the bug in libcurl-impersonate pops up).
from curl-impersonate.
Ahh, yes you're right I had forgotten I added that env var into my .bashrc a while back. Ok that explains it.
from curl-impersonate.
Alright, I'll close this issue. We have #90 open for the libcurl-impersonate bug with the headers order.
from curl-impersonate.
Related Issues (20)
- Homebrew recipe? HOT 1
- JA4+ HOT 2
- Homebrew unofficial receipt available HOT 1
- Ubuntu 22:04 libcurl-impersonate-chrome.so cannot be preloaded HOT 3
- Websocket on Windows HOT 6
- Pre shared key extension
- make chrome-build fails HOT 1
- Fiddler Intergration / Cloudfare bypass
- WS/WSS Protocol in Docker Image HOT 3
- compile error on boringssl - [-Werror,-Wunused-but-set-variable] HOT 3
- Content-Encoding may be gzip but the response body is uncompressed
- Safari http2 fingerprints are not correct
- make chrome-build failed HOT 7
- Bypassing Cloudflare HOT 6
- Impersonation no longer good enough?
- recompile php with curl-impersonate HOT 1
- curl: (56) HTTP/2 stream 1 was reset
- Namespace wrapping for firefox and chrome in one?
- Use Different browsers, ... , But get same fingerprints
- Precompiled binary doesn't work, but docker version works well for chatGPT.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from curl-impersonate.