go-rod / stealth Goto Github PK

View Code? Open in Web Editor NEW

220.0 220.0 27.0 118 KB

anti-bot-detection with rod

Home Page: https://pkg.go.dev/github.com/go-rod/stealth

License: MIT License

Go 100.00%

bot-detection rod stealth

stealth's People

Contributors

Stargazers

Watchers

stealth's Issues

求教:这个库怎么使用

比如打开：https://tly38.com/
怎么绕过Cloudflare,example没看明白怎么使用的

current version has old version of navigator.plugins and navigator.mimeTypes, that is the problem, i think...
current stealth version have:

but new version chromium browser have default values in different object structure:

i was test it in different detect sites (https://bot.sannysoft.com/ and other) and i think, just need to update navigator.plugins object to new version and navigator.mimeTypes in this package and after it will work fine becouse everything else is still fine

canvas & ja3

Hi
We spoke on: https://github.com/go-rod/rod/issues/162

A. Do you plan on adding canvas evasion?
B. JA3 fingerprinting evasion (it becomes easier and easier to validate it, and going forward I think this fingerprinting will be used by a lot of sites)
C. What else is different in this repo from: https://github.com/berstend/puppeteer-extra/tree/master/packages/extract-stealth-evasions#readme

在遇到瑞数的情况，stealth无法起作用,将一直卡住

复现代码如下

package main

import (
	"crypto/md5"
	"fmt"
	"github.com/go-rod/rod/lib/launcher"
	"time"

	"github.com/go-rod/rod"
	"github.com/go-rod/stealth"
)

func main() {
	path, _ := launcher.LookPath()
	url := launcher.New().Bin(path).Headless(false).MustLaunch()
	browser := rod.New().
		ControlURL(url).Timeout(time.Minute).MustConnect()
	defer browser.MustClose()

	// You can also use stealth.JS directly without rod
	fmt.Printf("js: %x\n\n", md5.Sum([]byte(stealth.JS)))

	page := stealth.MustPage(browser)

	page.MustNavigate("http://lib.cqvip.com/")
	time.Sleep(10 * time.Minute)
	
}

Headless(false) using stealth

When using this stealth plugin, it seems its not able to open/show the browser window - which is possible when not using this.

I am using the launcher like

	l := launcher.New().
		Headless(false).
		Devtools(true)
	defer l.Cleanup() // remove launcher.FlagUserDataDir
	url := l.MustLaunch()

however with stealth it seems to ignore the Headless(false) parameter?

Tip: extract-stealth-evasions

FYI there's https://www.npmjs.com/package/extract-stealth-evasions now, which is part of the puppeteer-extra repo.

How to use stealth?

Is here any complete example code?

I guess stealth.min.js is detectable using the Default Device, wondering if we could change webgl, etc, through the API

When I visit https://amiunique.org/fp and https://coveryourtracks.eff.org/ using:

	page := stealth.MustPage(browser)
	page.MustEmulate(devices.Device{
		Title: "Custom Linux Desktop",
		UserAgent:      "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0)",
		AcceptLanguage: "en",
	})
	page.MustNavigate("https://amiunique.org/fp")

AmIUnique always shows Intel Iris as a graphics card and EFF says my browser's fingerprint is unique and detectable. If I visit using my regular desktop Brave browser it says: your browser has a randomized fingerprint. So, is there any way to customize those through go-rod/stealth?

It seems there might be ways to change or randomize those using JavaScript here and here for example. But, I couldn't find a way in the API to allow me to do that.

So relevant issue on berstend/puppeteer-extra that might explain the issue better.

In a regular browser either Firefox or Chrome if you look for Fingerprint extensions there are some extensions that do a good job at randomizing the Fingerprint. They even notify you when an attempt is made by the website you are visiting Fingerprint you. If we could load these extensions, using rod they might be able to do the job. Sadly, it seems load-extension not working in my Firefox tests, haven't tested with chrome yet. But, I remember chromedp had no way to do that. Especially if I load an existing user data-dir. If I recall correctly, I saw some comments in the rod doc that NewUserMode might not be usable in such scenarios as well.

I appreciate it if you head me in the right direction. Thank you!

import stealth shows checksum mismatch error

main imports
github.com/go-rod/stealth: github.com/go-rod/[email protected]: verifying module: checksum mismatch
downloaded: h1:kKjZSB6osxJInG/2ywC8vt1Y0Oig/UnUfkcd/IRMGDM=
sum.golang.org: h1:JU5MgM1TZ1t1r+JLkKyf3EhnxMNCLHjcfQUFe/Vxta8=

Compress the js

Large js may cause cdp crash on some machines.

Add some doc in the readme to explain how this repo works

Help beginners to understand it to reduce misuse.

Can you add multi parameter to page.EvalOnNewDocument(JS)

can you add multi parameter to page.EvalOnNewDocument(JS)
something like page.Eval(js string, jsArgs ...interface{}).
it can better pass parameters to JS

And bypass can use this code for handle all new open target

         wait := browser.EachEvent(func(e *proto.TargetTargetCreated) {
		// if it's not a page return
		if e.TargetInfo.Type != proto.TargetTargetInfoTypePage {
			fmt.Println("not a page", e.TargetInfo.Type)
			return
		}
		fmt.Println("page created")
		// create a page from the page id
		page := browser.MustPageFromTargetID(e.TargetInfo.TargetID)
		_, err = page.EvalOnNewDocument(JS)
		if err != nil {
			fmt.Println(err)
		}
	})
	go wait()

Hide fingerprint and prevent bot detection

go-rod/rod#29 (comment)

Blocking fingerprinting is more like a "cat and mouse" game, I won't say there could be a common way, so it'll be a long-term thing to do.

We need to figure out the ways they collect unique information from the browser, then use js to hook them.

Fortunately, we can get all source codes from the client-side, even if they're obfuscated.

Some information:

Got error when using npx

Run `npm install` to download the correct Chromium revision

Remove the cache of the npx will resolve it rm ~/.npm/_npx

Questions: Cloudflare Waiting Room

hello, I'm trying to bypass cloudflare with your example_file but i'm not finding any documentation to wait until the pages is fully loaded.
As for example on aw-lab.com you have to wait like 5-6 seconds before been redirect in the home page.
I tried to use page.Timeout(10 * time.Second).MustWaitLoad() but it didnt wait 10 seconds

Can't bypass Login With Google

Try to automate login with google process

it gives me error message:
"This browser or app may not be secure"

Here's my code so far:

package main

import (
	"github.com/go-rod/rod"
	"github.com/go-rod/rod/lib/utils"
	"github.com/go-rod/stealth"
)

func main() {
	const email = "<my_gmail_account>"
	browser := rod.New().MustConnect()
	page := stealth.MustPage(browser)
	page.MustNavigate("https://accounts.google.com/ServiceLogin")
	page.MustElement("#identifierId").MustInput(email).MustType()
	page.MustElement("#identifierNext").MustClick()
	utils.Pause()
}

someone report the same issue here:
berstend/puppeteer-extra#668

anybody, can give me directions?

WebDriver present with a remote launcher

I'm trying to use bypass with a remote launcher in a docker image, but it fails the WebDriver test. Any chance I can make it work this way?

func main() {
	client := launcher.MustNewRemote("ws://localhost:9222")
	client.Headless(true)

	browser := rod.New().Timeout(time.Minute*2).Client(client.Client()).MustConnect()
	...

Emulate CPU/ GPU + more.

Hello :),

There are many things that need to be spoofed in order to pass bot protection nowadays.

Most important things are custom fingerprints, CPU, GPU and several other things which can't be emulated yet in stealth.

Is it possible to add it somehow, because I do get fingerprinted easily on some sites with heave bot protection?

If not is there any way to emulate this whole fingerprint I posted down here?

{
  "navigator": {
    "appCodeName": "Mozilla",
    "appName": "Netscape",
    "appVersion": "5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36",
    "language": "fr-FR",
    "languages": [
      "fr-FR",
      "fr",
      "en-US",
      "en"
    ],
    "userAgent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36",
    "vendor": "Google Inc.",
    "product": "Gecko",
    "productSub": "20030107",
    "platform": "Win32",
    "vibrate": true,
    "getBattery": true,
    "credentials": true,
    "appMinorVersion": false,
    "bluetooth": true,
    "storage": true,
    "getGamepads": true,
    "getStorageUpdates": false,
    "hardwareConcurrency": true,
    "mediaDevices": true,
    "mozAlarms": false,
    "mozConnection": false,
    "mozIsLocallyAvailable": false,
    "mozPhoneNumberService": false,
    "msManipulationViewsEnabled": false,
    "permissions": true,
    "registerProtocolHandler": true,
    "requestMediaKeySystemAccess": true,
    "requestWakeLock": false,
    "sendBeacon": true,
    "serviceWorker": true,
    "storeWebWideTrackingException": false,
    "webkitGetGamepads": false,
    "webkitTemporaryStorage": true,
    "cookieEnabled": true,
    "javaEnabled": false,
    "doNotTrack": -1,
    "plugins": []
  },
  "window": {
    "ActiveXObject": false,
    "innerHeight": 888,
    "innerWidth": 1862,
    "outerWidth": 1862,
    "outerHeight": 888,
    "devicePixelRatio": 2,
    "addEventListener": true,
    "XMLHttpRequest": true,
    "XDomainRequest": false,
    "orientation": "undefined",
    "DeviceOrientationEvent": true,
    "DeviceMotionEvent": true,
    "TouchEvent": true,
    "chrome": {
      "app": {
        "isInstalled": false,
        "InstallState": {
          "DISABLED": "disabled",
          "INSTALLED": "installed",
          "NOT_INSTALLED": "not_installed"
        },
        "RunningState": {
          "CANNOT_RUN": "cannot_run",
          "READY_TO_RUN": "ready_to_run",
          "RUNNING": "running"
        }
      }
    },
    "prototype_bind": true,
    "PointerEvent": true,
    "sessionStorage": true,
    "localStorage": true,
    "indexedDB": true,
    "FileReader": true,
    "HTMLElement": false,
    "webRTC": true,
    "mozInnerScreenY": 0
  },
  "document": {
    "documentMode": "undefined",
    "webdriver": false,
    "driver": false,
    "selenium": false,
    "hidden": false,
    "webkitHidden": false
  },
  "other": {
    "CC_ON": false,
    "InstallTrigger": false,
    "prototype_forEach": true,
    "imul": true,
    "parseInt": true,
    "hypot": true,
    "value1": false,
    "XPathResult": true
  },
  "performance": {
    "timeOrigin": 1672836946172.5,
    "timing": {
      "connectStart": 1672836946239,
      "navigationStart": 1672836946171,
      "secureConnectionStart": 1672836946283,
      "fetchStart": 1672836946179,
      "domContentLoadedEventStart": 1672836946872,
      "responseStart": 1672836946588,
      "domInteractive": 1672836946872,
      "domainLookupEnd": 1672836946239,
      "responseEnd": 1672836946589,
      "redirectStart": 0,
      "requestStart": 1672836946553,
      "unloadEventEnd": 0,
      "unloadEventStart": 0,
      "domLoading": 1672836946594,
      "domComplete": 1672836947114,
      "domainLookupStart": 1672836946239,
      "loadEventStart": 1672836947114,
      "domContentLoadedEventEnd": 1672836946872,
      "loadEventEnd": 1672836947115,
      "redirectEnd": 0,
      "connectEnd": 1672836946553
    },
    "navigation": {
      "type": 0,
      "redirectCount": 0
    }
  },
  "canvas": {
    "value1": "420217769",
    "value2": "1243744842"
  },
  "fonts_optm": "12,14,15,16,17,27,3,38,40,44,7,9",
  "fonts": "12,14,15,16,17,27,3,38,40,44,7,9",
  "rCFP": "-158760887",
  "ssh": "e54e4dcbd557a1fe86616568abcd0650295cf8d2f959bd8c84fec71923f1a445",
  "mr": "20,20,20,20,20,20,20,20,20,0,0,0,0,100,",
  "brave": "0",
  "wv": "Google Inc. (NVIDIA)",
  "wr": "ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Ti Direct3D11 vs_5_0 ps_5_0, D3D11)",
  "weh": "fab37783e5c3bc89f876774bac442ad3e613f7dc5b769b19a8d4d8c6b80ef077",
  "wl": 29,
  "fmh": "082f3e287baebaac8dfa0e53be71bed004ee57a16b3dffd269f318915d405c81",
  "fpValstr": "420217769;1243744842;dis;;true;true;true;-60;true;24;24;true;false;-1",
  "np": "11321144241322243122",
  "screen": {
    "availHeight": 888,
    "availLeft": 0,
    "availTop": 0,
    "availWidth": 1862,
    "colorDepth": 24,
    "height": 888,
    "pixelDepth": 24,
    "width": 1862
  },
  "__v": 0
}

Browser Being Bot Detected

Looks like the PageSetBypassCSP no longer works, see below

But looks like there is also a bypass in the puppeteer-extra plugin
berstend/puppeteer-extra#668 (comment)

If I am wrong and this function is already available, how can we use ?

Thanks

Rename bypass to stealth

I think stealth is a better name for this project.

Cloudflare detection false

Hello, I'm trying to bypass cloudflare with code:

func main() {
	lc := launcher.New().
		Set(flags.UserDataDir, ".chrome").
		Headless(false)

	ctlURL := lc.MustLaunch()

	browser := rod.New().ControlURL(ctlURL).MustConnect()
	defer browser.MustClose()
	
	page := stealth.MustPage(browser)
	page.MustNavigate("https://nowsecure.nl/").MustWaitLoad()

	utils.Pause()
}

Not passed whether to enable Headless or not.

stealth can not bypass cloudflare bot detection

Description

I want to use stealth to bypass Cloudflare's latest bot detection. This feature was great last year, but recently when I wanted to use stealth, it didn't work like it used to. Stealth cannot bypass Cloudflare's bot detection mechanism.

How to reproduce

package main

import (
	"context"
	"os"
	"time"

	"github.com/go-rod/rod"
	"github.com/go-rod/rod/lib/launcher"
	"github.com/go-rod/stealth"
)

func main() {
	l := launcher.NewUserMode().
		Leakless(true).
		UserDataDir("tmp/t").
		Set("disable-default-apps").
		Set("no-first-run").
		Headless(false)
	defer l.Kill()

	browser := rod.New().ControlURL(l.MustLaunch()).MustConnect()
	defer browser.MustClose()

	page := stealth.MustPage(browser)
	page.MustNavigate("https://nowsecure.nl").MustWaitLoad()

	time.Sleep(time.Minute)
}

What I want

I want to see the actual pages behind Cloudflare protection.

What I got

The page is stuck on the Cloudflare bot detection page.

webdriver is true on the new page

Click on a label to open a new page, and webdriver is true on the new page.

What methods are there to deal with?

headless 问题

使用stealth库，headless设置为true，就无法获取网页信息，但是设置为false，页面返回是403，这是啥原因。