Git Product home page Git Product logo

Comments (6)

lwthiker avatar lwthiker commented on May 18, 2024

I see extra headers for Connection:, Cache-Control:, and also DNT

I just tested Chrome 103 with Wireshark and the SSLKEYLOGFILE env var set and browsed to www.wikipedia.org. I didn't get these headers. It might have something to do with the local socat or the use of HTTP/1.1 (Connection for example is not used in HTTP/2). Anyway I wouldn't worry about adding them unless it looks absolutely necessary for fetching the website you need.

Also the sec-ch-ua: string seems to be a bit different

Yes it changes every release.

the Connection: and Cache-Control: headers end up at the end, despite my array passing them first

You are correct, this is a bug with my implementation. The user-supplied headers are always added after the built-in list of headers that curl-impersonate uses. I'll open a separate issue for that.

I tried editing the curl_chrome101 script to add these 2 headers in there, and same behavior: they end up at the bottom of the headers instead of top.

I just tried that and the order looks fine, same as supplied in the script. Can you verify again?

from curl-impersonate.

lwthiker avatar lwthiker commented on May 18, 2024

Opened #90

from curl-impersonate.

A-Posthuman avatar A-Posthuman commented on May 18, 2024

Here is what I get when manually editing the curl_chrome101 script:

  1. sudo nano /usr/local/bin/curl_chrome101
  2. add in the header lines so the script looks like this:
#!/bin/bash

# Find the directory of this script
dir=${0%/*}

# The list of ciphers can be obtained by looking at the Client Hello message in
# Wireshark, then converting it using this reference
# https://wiki.mozilla.org/Security/Cipher_Suites
"$dir/curl-impersonate-chrome" \
    --ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY130
5_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-A
ES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE
-RSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-SHA,ECDHE-RSA-AES256-SHA,AES128-GCM-SHA256,AES256-GCM-SHA384,AES128-SHA,AES256-SHA \
    -H 'Connection: keep-alive' \
    -H 'Cache-Control: max-age=0' \
    -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"' \
    -H 'sec-ch-ua-mobile: ?0' \
    -H 'sec-ch-ua-platform: "Windows"' \
    -H 'Upgrade-Insecure-Requests: 1' \
    -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36' \
    -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
    -H 'Sec-Fetch-Site: none' \
    -H 'Sec-Fetch-Mode: navigate' \
    -H 'Sec-Fetch-User: ?1' \
    -H 'Sec-Fetch-Dest: document' \
    -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Accept-Language: en-US,en;q=0.9' \
    --http2 --false-start --compressed \
    --tlsv1.2 --no-npn --alps \
    --cert-compression brotli \
    "$@"
  1. run socat in one shell: socat -v openssl-listen:8443,reuseaddr,fork,cert=server.pem,verify=0 echo
  2. In another shell, connect: curl_chrome101 -k https://localhost:8443
  3. result:
> 2022/07/24 09:45:05.293892  length=689 from=0 to=688
GET / HTTP/1.1\r
Host: localhost:8443\r
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"\r
sec-ch-ua-mobile: ?0\r
sec-ch-ua-platform: "Windows"\r
Upgrade-Insecure-Requests: 1\r
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
 like Gecko) Chrome/101.0.4951.67 Safari/537.36\r
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/w
ebp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r
Sec-Fetch-Site: none\r
Sec-Fetch-Mode: navigate\r
Sec-Fetch-User: ?1\r
Sec-Fetch-Dest: document\r
Accept-Encoding: gzip, deflate, br\r
Accept-Language: en-US,en;q=0.9\r
Connection: keep-alive\r
Cache-Control: max-age=0\r

from curl-impersonate.

lwthiker avatar lwthiker commented on May 18, 2024

I just tried exactly the same and I get the correct order. Is it possible that you have the CURL_IMPERSONATE environment variable set as well? Because that could cause that (i.e. the bug in libcurl-impersonate pops up).

from curl-impersonate.

A-Posthuman avatar A-Posthuman commented on May 18, 2024

Ahh, yes you're right I had forgotten I added that env var into my .bashrc a while back. Ok that explains it.

from curl-impersonate.

lwthiker avatar lwthiker commented on May 18, 2024

Alright, I'll close this issue. We have #90 open for the libcurl-impersonate bug with the headers order.

from curl-impersonate.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.