Git Product home page Git Product logo

chromedp-undetected's People

Contributors

alexgredn avatar davincible avatar dependabot[bot] avatar mihaiav avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

chromedp-undetected's Issues

Many Core files are generated in the container

Step

package chromedpundetected

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/chromedp/chromedp"
	"github.com/hashicorp/go-multierror"
)

var n = 3

func TestChromedpundetected(t *testing.T) {
	testRun(t,
		n,
		NewConfig(
			WithTimeout(time.Hour),
			WithHeadless(),
		),
		func(ctx context.Context) error {
			return chromedp.Run(ctx,
				chromedp.Navigate("https://nowsecure.nl"),
				chromedp.WaitVisible(`//div[@class="notfound"]`),
			)
		},
	)

	t.Logf("Undetected!")
}

func testRun(t *testing.T, n int, cfg Config, run func(ctx context.Context) error) {
	var gerr error
	var success bool

	// Attempt to run the tests multiple times, as in CI they are unstable.
	for i := 0; i < n; i++ {
		t.Logf("Attempt %d/%d", i+1, n)

		ctx, cancel, err := New(cfg)
		if err != nil {
			gerr = multierror.Append(gerr, fmt.Errorf("create context: %w", err))
			continue
		}

		if err = run(ctx); err != nil {
			gerr = multierror.Append(gerr, fmt.Errorf("chromedp run: %w", err))

			// Close Chrome instance.
			ctxN, cancelN := context.WithTimeout(ctx, time.Second*10)
			if err := chromedp.Cancel(ctxN); err != nil {
				t.Logf("failed to cancel: %v", err)
			}
			cancelN()
			cancel()

			continue
		}

		success = true
		if gerr != nil {
			t.Logf("attempt %d, errors: %s", i+1, gerr.Error())
		}

		cancel()
		break
	}

	if !success {
		t.Fatal(gerr)
	}
}
  • Modify the Dockerfile file to run the command
FROM golang:1.19

RUN apt-get update && apt-get -y upgrade && apt-get -y install gcc g++ ca-certificates chromium xvfb

WORKDIR /app

COPY . .

RUN go mod download

ENTRYPOINT [ "go", "test", "-run", "TestChromedpundetected" ]
  • Build the image and run

Wait a few minutes, then go into the container and look at it, you will see that some core files were created.

image

MacOS headless usage

Hi there,

First of all, thanks for making this package, it works perfectly for the protected site(s) I want to scrape!
My issue is that headless mode does not work on Mac. You already mentioned this in your README:

The headless option only works on linux, and requires Xvfb to be installed. Could theoretically work on Mac OS with xquartz but I don't have a Mac to test with, so feel free to PR.

I did install xquartz and your example only works by removing the headless option.

You mentioned you can't test it on here so I'm willing to do so but I would need some pointers on where to start if possible.

Thanks!

Can it support windows or macos

Hi Davincible
I made an error when compiling windows or macos. According to the error message, it seems that I am using a linux-specific field?

../../go/pkg/mod/github.com/!davincible/[email protected]/chrome.go:153:21: cmd.SysProcAttr.Pdeathsig undefined (type *syscall.SysProcAttr has no field or method Pdeathsig)
../../go/pkg/mod/github.com/!davincible/[email protected]/display.go:82:19: xvfb.SysProcAttr.Pdeathsig undefined (type *syscall.SysProcAttr has no field or method Pdeathsig)

Is there a way to be compatible with windows or macos

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.