AWS EC2 Linux Server Detected Despite Manipulated Headers Using Undetected Chromedriver about undetected-chromedriver HOT 9 OPEN

Updating just the user-agent isn't the only way to detect the actual platform of the browser.

Consider utilising 'sec' overrides.

sec_ch_ua = '"Examplary Browser"; v="73", ";Not?A.Brand"; v="27"' # for example

options = Options()
options.add_argument(f'--sec-ch-ua={sec_ch_ua}')

from undetected-chromedriver.

In a datacenter (what AWS is), you are detected per definition

from undetected-chromedriver.

Thank you for your answers! I am still wondering what possibilities there are for the server not to be detected. I already use proxies.

from undetected-chromedriver.

Who is detecting you? They may detect your IP. They be doing TLS fingerprint which is happening more and more

"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.0.0 Safari/537.36",

if this is your UA, then of course they are going to detect you, you are telling them you are using headless chrome

from undetected-chromedriver.

In general, you are right. But what surprises me is that I'm not detected with my computer, even though I use the same configs and proxies. My computer for comparison:

"request": {
            "headers": {
                "Upgrade-Insecure-Requests": "1",
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.0.0 Safari/537.36",
                "sec-ch-ua": "\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"",
                "sec-ch-ua-mobile": "?0",
                "sec-ch-ua-platform": "\"Windows\""
            }

My assumption is therefore that detecting Linux is the problem.

from undetected-chromedriver.

There's a thousand things they'd be flagging you for just Linux.

As @ usr741852 said, there's probably some level of fingerprinting at play, and if you're using a more typical base EC2 machine, that'd be the easiest to encounter blocks on.

Easy to test though, just spin up an instance with Windows on.

from undetected-chromedriver.

Sounds like a good idea. I'll let you know as soon as I have the results

from undetected-chromedriver.

I have now run the bot on an AWS Windows server. Unfortunately, I had the same experience, although I started it once with and once without headless. It probably has nothing to do with the operating system, but with the fact that these are EC2 instances. Is there a way to bypass this?

from undetected-chromedriver.

To better understand the problem, here is some context:
The bot scrapes the data of the particular matches. When I start the bot with my computer, it finds them all. If I start it with the server, the website returns a limited offer of all matches. So, I'm not blocked but only limited because the bot is detected.

from undetected-chromedriver.