Spoofed TLS/JA3 fingerprint is detected by some sites about cycletls HOT 6 OPEN

Incredibly useful info, I was noticing similar issues. Technically speaking that 771 at the start of the token designates tls 1.3 so if the outward request is not reflective of that that is a bug. Looking into this now, will update once I know more.

from cycletls.

Update:

I have been using https://tls13.1d.pw/ to test the tls 1.3 handshake and have discovered a few more variables that need to be set dynamically. As mentioned in this ticket there are a few extensions which cannot be parsed from the ja3 token and need to be manually set.

Problem 1

For tls 1.3 the main issue lies in extension 51 the KeyShareExtension and the SupportedCurvesExtension. Basically (and you can test this on tls13 by hitting it a few times or inspecting normal outgoing requests in wireshark)

Expected tls 1.3 behavior

1. The server will select a supported curve and request it from the KeyShareExtension list ->
2. If the KeyShareExtension curve is not supported on that tls version the server will send a HELLO_RETRY_REQUEST ->
3. The client will (should) send another CLIENT_HELLO ->
4. The server will select a different KeyShareExtension (server should be able to process this one)

you can see this on on tls13 or wireshark

Requests will sometimes just work because the correct KeyShareExtension will be selected.

Solution 1

I believe the fix for this is just to handle this HELLO_RETRY_REQUEST or only send specific curves based on the tls version of the server. Currently we are not sending another CLIENT_HELLO when asked for a a HELLO_RETRY_REQUEST

Problem 2

There is a fundamental detection problem here where the KeyShareExtensions are not set from the token and thus are detectable. I do NOT believe this is why the 403 error you are facing is happening above. I believe the reason for that is because we are failing the handshake and rather than erroring out like the tls13 site does the website just returns a 403.

Solution 2

Occasionally we also get errors when the SupportedCurvesExtension do not include the grease_placeholder extension. This is unique to chrome browser but again cannot be parsed from the ja3 token making if difficult to determine when to set it. A possible solution is parsing the User Agent and if Chrome is within it send the grease_placeholder data.

Notes

The tls13 as well as http2 provide decent external testing but the response formats make it hard to debug what is missing/what should be included. Ja3er also has a known bug which sometimes does not return the correct ja3 extensions. I will have a fix out which should (hopefully) allow you to consistently hit tls 1.3 servers although long term I believe developing the server component of this (http2/tls1.3) with declarative json formatted extension lists as well as the ja3 token/User Agent is important for making sure this repo is robust. Along with this maybe some documentation on how to parse ja3 tokens and how the server handshakes work to make debugging this easier in the future.

from cycletls.

Reverted/fixed some issues causing a failed TLS 1.3 handshake. It still has some intermittent request failures but I need to inspect/fix the Utls library to implement a permanent fix. With the default parrots defined in Utls e.g. utls.HelloChrome_83 or utls.HelloFirefox_65 we will still get the below error intermittently.

tls: server selected unsupported group

I would test detection on the current 0.0.14 release and if there are still issue let me know. I also added in custom error handling related to tls 1.3 so if that error appears let me know.

from cycletls.

Followup, upon further investigation this appears related to this fix . It seems 771 actually designates tls version 1.2 so the handshake fails, the fix would be reverting this and allowing all extension types. It appears though in my testing that sometimes the server selects version 1.2 (utls forces us to have a min version that can't be 1.3) and this causes the handshake to intermittently fail. Once I figure this out there should be a fix out.

from cycletls.

An update from my side: I've tried reverting af9b002 and you were right, now the used protocol was TLS v1.3 indeed. However, and this is very odd, CycleTLS still got detected. So regardless of this, it looks like the TLS version is not the (only) way of the spoofing detection.

from cycletls.

Great job on fixing the issue, thank you for your work!
However, something still has to be different, as even with the same headers and the same TLS version, I still get 403 Forbidden. I do see some differences in the Client Hello in Wireshark, but unfortunately I don't know the meaning/reason of them.

from cycletls.