My node.js version is 12.14.1. I just tried to install it and run the example script in the document, but I got such error.

2020-07-22T02:37:00.770Z ERROR UnhandledClientError {
clientError: TypeError: SecretAgent.createBrowser is not a function
at /media/mine/Work/fiverr/keadam/index.js:4:37
at Object. (/media/mine/Work/fiverr/keadam/index.js:9:3)
at Module._compile (internal/modules/cjs/loader.js:955:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)
at Module.load (internal/modules/cjs/loader.js:811:32)
at Function.Module._load (internal/modules/cjs/loader.js:723:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)
at internal/main/run_main_module.js:17:11
}

This ticket is to refactor the polyfills that backfill headed chrome functionality so that they are ready to go on windows and mac, not just docker.

Right now, when you run on windows or mac, the polyfills don't actually simulate a real browser, which can cause issues with aggressively blocking sites.

We need to allow configuration of the viewport that a session will use. Things we need to decide:

1. How is this configured?

• I think it's likely part of creating a session
• We should probably store this with userProfile

2. Do we have a list of default screen sizes that are rotated through? What about window positions/locations?

• If we randomize this, could it result in some unexpected behavior? Do we have fixed screen sizes, but randomized locations?
• Or should we tweak +- some number of pixels?

3. Ideally we would be capturing screen combinations in Double Agent. Maybe this should be part of our eventual "user sampling" to determine how common various features are with real users

By default SA should use a consistent emulatorID on every run. To randomly cycle through emulators, the user must explicitly set a default flag.

Proposal:

If emulatorId is null or undefined when calling SecretAgent.createBrowser() then default emulatorId will be used (for now, it will be chrome-80).

If emulatorId is set to "random" then a random emulator will be picked.

If emulatorId is set to a valid/known emulator then use that.

Otherwise (if emulatorId is invalid/unknown) then throw an error.

When running the Double Agent test suite, Secret Agent will experience timeouts for 3 or 4 sessions (out of ~40). We need to track down what exactly is happening here.

Things to note:

• Other requests are still going through, so this is not a full-stop of the system
• There's not consistency to "which" sessions fail to load
• The hang is frequently on the "run-page" - eg, http://a1.dlf.org:3001/run-page?sessionid=32. Some of the fetch requests don't complete (no response handler ever called, but double agent records a response)

[35.x] Error on http://a1.dlf.org:3001/run-page?sessionid=35 TimeoutError: Timeout waiting for element document,querySelector,#goto-results-page to be visible
    at Timer.throwIfExpired (/app/shared/commons/Timer.ts:37:13)
    at Window.waitForElement (/app/core/lib/Window.ts:381:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at SessionState.runCommand (/app/shared/session-state/index.ts:124:14)
    at Window.runCommand (/app/core/lib/Window.ts:188:16)
    at Core.waitForElement (/app/core/index.ts:68:5)
    at CoreClient.coreClient.pipeOutgoingCommand (/app/full-client/index.ts:17:18)
    at CoreCommandQueue.processQueue (/app/client/lib/CoreCommandQueue.ts:46:28)

We need to add documentation for AwaitedDOM to the website.

Hi,

This isn't an issue, feature request, or bug fix. I know that maintaining an open-source project can be daunting and challenging. Especially, without a lot of external validation or users willing to submit praise or feedback.

I am just saying thank you for your continued efforts in making scraping/automation an amazing experience!

I've not had a chance to use secret-agent yet but I am going to be using it in my next project.

Hopefully, this isn't spamming the issues feed. If it is, feel free to close/remove this issue but don't stop the good work.

All the best.

First of all, thank you for a great job! Keep it up!

Now to the problem. I'm trying to make a request to street-beat.ru, but I get this two errors:

Error 1
2020-12-23T10:27:10.171Z ERROR [node_modules/@secret-agent/core/index] UnhandledErrorOrRejection { clientError: 'TypeError [ERR_HTTP2_HEADER_SINGLE_VALUE]: Header field "x-content-type-options" must only have a single value', context: {} }

Error 2
2020-12-22T19:48:44.558Z ERROR [node_modules/@secret-agent/mitm/handlers/HttpRequestHandler] MitmHttpRequest.ProxyToServer.RequestHandlerError { request: 'GET: https://cdn1.imshop.io/assets/app/b2.min.css', error: 'Error: 2020/12/22 23:48:44 Error on tls handshake with 91.209.77.149:45785. Args main.ConnectArgs{Host:"92.223.124.254", Port:"443", IsSsl:true, Servername:"cdn1.imshop.io", RejectUnauthorized:true, ProxyUrl:"", ClientHelloId:"Chrome83", TcpTtl:0, TcpWindowSize:0, Debug:false}, tlsConn.Handshake error: x509: certificate signed by unknown authority', context: {} }

In Replay window page seems like fully loaded, but neither waitForAllContentLoaded nor waitForElement methods are not triggered...

The errors occurs with and without proxy, I am using example code from Usage Example section.
Secret-Agent v1.2.0-alpha.4, nodeJS v14.15.1, macOS Big Sur 11.0.1

When proxy designated in upstreamProxyUrl is unreachable for whatever reason - process dies instead of throwing error.

How to reproduce:

const SecretAgent = require('secret-agent');

(async () => {
    try {
        const agent = await new SecretAgent({
            showReplay: false,
            upstreamProxyUrl: 'http://127.0.0.1:65535'
        });
        await agent.goto('http://api.ipify.org/');
    }
    catch(err) {
        console.error(`Error: ${err}`)
    }
})();

Expected Behavior

Catchable error?

Current Behavior

Getting a couple of errors in the console log.

ts-node ./src/_playground.ts

2020-12-04T10:53:48.829Z ERROR [@secret-agent/core/index] UnhandledErrorOrRejection {
  clientError: '2020/12/04 13:53:48 Dial (proxy/remote) Error: dial tcp 127.0.0.1:65535: connect: connection refused',
  context: {}
}
2020-12-04T10:53:48.831Z ERROR [@secret-agent/core/index] UnhandledErrorOrRejection {
  clientError: "TypeError: Cannot create property 'stack' on string '2020/12/04 13:53:48 Dial (proxy/remote) Error: dial tcp 127.0.0.1:65535: connect: connection refused'",
  context: {}
}

Context

SecretAgent version: 1.2.0-alpha.2
ts-node: v9.1.0

Shadow DOM is not currently being recorded by our PageEventsRecorder, and therefor doesn't show up in Replay. This can make some sites appear unstyled (commonly those using React Styled Components).

The Chrome 79 emulator has several failures in it's polyfills - many are the result of things not being available in a non-secure environment. These are things like Bluetooth, registerProtocolHandler, etc.

Polyfills can access the window location, so they should be altered to handle secure vs non-secure pages. This will require new data for http property orders

For my particular case, I want to execute a jQuery AJAX request inside the browser for my target website. I'm sorry if this is mentioned in the documentation but I couldn't find anything related to it via google search.

At the moment all documented methods in AwaitedDOM show a generic ...args and a return type of null. This should be fixed to show their true value.

What is the equivalent to puppeteer's page.reload? Can't find specific method, should it be performed as agent.document.location.reload()?

Seems like such sites as brandshop.ru (with Variti bot protection) can detect this. First load is successful, but such reload does not go through...

The MDN documentation for CharacterData says that it is:

an abstract interface, meaning there aren't any object of type CharacterData: it is implemented by other interfaces, like Text, Comment, or ProcessingInstruction which aren't abstract.

Therefore it should not be a Super.

npm install not working, it shows errors and crashes

When Replay shows a previous script, it doesn't currently load or set the viewport that was used during the script. This means mouse positions could be "off" when they're reproduced in replay.

At the moment all property types are set to null. They should show their type.

I found a browser.closeTab function. Is there a way to open a new tab manually? How does secret-agent manage the opening of new tabs?

I tried adding cross-env SA_REPLAY_SKIP_BINARY_DOWNLOAD=true npm instal into package.json and ran it. It ignores the env and downloads replay.

We need to capture fatal errors in and set a close data on a session (or record the error and display that in Replay).

Secret agent scripts sometimes have the need to go "Back" to the previous url vs a new "goto" to the previous url. We should add the ability to navigate back in the navigation stack.

We need to create updated Chrome plugins. When we add these, we should add the ability for emulators to choose their own rendering engine. To consider:

1. Create a central repository that emulators can tap into. This would mean Emulators or SA is responsible for providing paths to different engines.
2. Or emulators package their own rendering engine. This might be preferable if, for example, someone wanted to use a patched version of a browser

How can i pass args like devtools: false to the underlying chromium?

Hello,

Thank you for your excellent work.

Is it possible to provide args like --no-sandbox to the browser?

Right now we're displaying the direct properties and methods of each Super, but since they inherit from a number of other classes, we're missing the bulk. Instead, we should just list and link to their inherited classes.

When running Secret Agent in a Docker, the Tcp settings to emulate Windows 7/10 don't take. Need to investigate if this has to do with running in Docker "on a mac", or if this is because the Go configuration is not working in Linux/Docker-slim

A few things:

1. Certain requests are hanging up in the Mitm. This appears to have been introduced once we started capturing postData.
2. JsPath is not found in the window after a number of pages have loaded. Appears to be a timing issue with running the NewDocument initialization scripts
3. Xhr Requests going through the Mitm are throwing an error that their "resourceType" cannot be found.

Search box on AwaitedDOM page is not currently implemented.

Right now, iframe content is being recorded during each session, but we don't replay anything other than the main frame. To get this to work, we need some way to link the frames created on the page to the domChange records (they're recorded in different contexts and so stored with overlapping ids and elements - ie, a body for each frame).

When you run a document.querySelectorAll, you should be able to get an index out of the resultset:

const results = await document.querySelectorAll('a'); const href = await results[0].getAttribute('href')

I want to block HTTP Requests that doesn't affect scrapping process like images, styles, etc. How can I do it? Does it increases the chances of detection of Secret Agent?

is it possible to set different proxies for different tabs using secret agent?

I want to add cache-control and pragma headers to every request with value 'no-cache'. It seems like my target site (street-beat.ru) somehow understands that I am using proxy, so I can't pass through to the actual site page. I thought that requests caching by proxy may be the problem.

At the moment we're not including properties and methods from inherited classes and/or mixins. They should be included directly on the class documentation page.

I'd like to see a mobile view for this library, I think it'll be awesome!

const SecretAgent = require('secret-agent');

(async () => {
const agent = new SecretAgent();
console.log(agent)
await agent.goto('https://example.org');
const title = await agent.document.title;
const intro = await agent.document.querySelector('p').textContent;
await agent.close();

console.log('Retrieved from https://example.org', {
title,
intro,
});
})();
I'm getting this error why?
UnhandledErrorOrRejection { clientError: 'Error: spawn ENOMEM', context: {} }

Got this error, when I tried to set cookies using setItem method (agent.activeTab.cookieStorage.setItem('test', 'test')):

clientError: 'Error: Storage.setCookies: Invalid cookie fields'

Used simple example code from docs. MacOS.

creepjs is able to detect secret-agent's lies.

Hello, how to use AwaitedDOM as specified element?

agent.document.querySelector("canvas") => returns ISuperElement

I want to call toDataURL() of HTMLCanvasElement but how would I do it ?

For consistency, AwaitedDOM documentation pages should use the same template as standard docs.

Should we add SecretAgent.shutdown to website examples? Or should we try to detect if we're the only thing sitting in the event loop and shutdown?

At the moment SA creates two directories (.sessions and .mitm-ca) in your CWD (current working directory) every time a script is run. This creates a confusing clutter of directories.

Both directories need to be organized under a single .secret-agent directory. The big question is, where should this new directory go?

A few options:

• Keep it in the CWD. Seems like a bad idea as the same script could have multiple .secret-agent directories depending on where it was run each time.

• Put it in the scriptEntrypoint directory. This is better than the previous option, but still seems a bit overkill (multiple .secret-agent directories if you have scripts in multiple directories). However, the bigger challenge is the data dir is created inside Core, which means it doesn't have access to scriptEntrypoint if working with remote client.

• Put it in the "project root". This would be ideal, except that determining the "project root" is tricky. It seems the project root can change substantially depending on how the script was launched, which in some circumstances, could make it frustrating to find the data directory. See https://stackoverflow.com/questions/10265798/determine-project-root-from-a-running-node-js-application

• Put it in /tmp. The pro is that it's consistently in the same place every time. The con is that data from ALL scripts and projects is going to be jumbled together into the same directory.

Thoughts?

In https://secretagent.dev/docs#installation it is mentioned that the download size is quite substantial and also why. But if this is all the case, why is it not simply recommended to download this instead as a global dependency (-g)? Is there a reason why you would want to have this as a local dependency?

I ask, as I cannot imagine that if you have 10 projects using it, that you want all of them using this dependency downloaded locally.

Is there any way to enforce using local DNS resolver?
In one of the projects, I must use a local DNS resolver (installed on the same host), and it seems that SecretAgent uses pre-configured DNS servers like Cloudflare over TLS, and I can't find a way to change that.

Updated Secret Agent to 1.3.0-alpha.0 and now getting this error. MacOS 11.1. The simplest example code.

TypeError: Cannot read property 'split' of undefined at extractOsVersion (/Users/dmz/.nvm/versions/node/v14.15.1/lib/node_modules/secret-agent/emulate-browsers/base/lib/getLocalOperatingSystemMeta.ts:27:21) at Object.getLocalOperatingSystemMeta [as default] (/Users/dmz/.nvm/versions/node/v14.15.1/lib/node_modules/secret-agent/emulate-browsers/base/lib/getLocalOperatingSystemMeta.ts:10:19) at Object.<anonymous> (/Users/dmz/.nvm/versions/node/v14.15.1/lib/node_modules/secret-agent/emulate-browsers/base/lib/DomDiffLoader.ts:5:48) at Module._compile (internal/modules/cjs/loader.js:1063:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10) at Module.load (internal/modules/cjs/loader.js:928:32) at Function.Module._load (internal/modules/cjs/loader.js:769:14) at Module.require (internal/modules/cjs/loader.js:952:19) at require (internal/modules/cjs/helpers.js:88:18) at Object.<anonymous> (/Users/dmz/.nvm/versions/node/v14.15.1/lib/node_modules/secret-agent/emulate-browsers/base/index.ts:8:1)

Got this warning, while recursively running (with timeout throttling) instance of Secret Agent.

Iteration goes like this:

1. Create Secret Agent instance with userProfile (if first iteration, userProfile = null);
2. agent.goto(url);
3. Scrape page;
4. agent.close();
5. Recursive call.

After some time this warning appears:
2021-01-28T11:57:09.227Z WARN [node_modules/@secret-agent/mitm/lib/MitmProxy] MitmProxy.RequestWithoutSessionId { isSSL: true, host: 'example.ru', url: '/', context: {} } 2021-01-28T11:57:09.331Z WARN [node_modules/@secret-agent/mitm/lib/MitmProxy] MitmProxy.RequestWithoutSessionId { isSSL: true, host: 'example.ru', url: '/example', context: {} }

Full iteration located in try/catch block, but it catches nothing. Iteration just freezes until I restart node process.

Secret agent v1.2.0-alpha.5. MacOS.

I am trying to edit some cookies but cannot find a method to execute code. And the document is read-only.
Let me know if I missed something or if this is not yet released

SecretAgent.createBrowser is
not a function

All properties and methods are currently be displayed in the AwaitedDOM documentation. We should hide the ones that are not yet implemented... or at least differentiate between them and the ones that are implemented.