chazkii / chromewhip Goto Github PK

View Code? Open in Web Editor NEW

116.0 9.0 16.0 1.93 MB

Scriptable Google Chrome™ as a HTTP service + asyncio driver

License: MIT License

Python 99.49% Shell 0.12% JavaScript 0.01% Makefile 0.01% Dockerfile 0.09% Jinja 0.28%

google-chrome devtools-protocol chrome-process asyncio-driver

chromewhip's People

Contributors

Stargazers

Watchers

Forkers

millen1m andreymeretsky cosive asahasrabuddhe ruairif onetwo1 prownsix spinda felciano hixio-mh zanachka huabinghuachaba kaliiiiiiiiii

chromewhip's Issues

Page.addScriptToEvaluateOnNewDocument

How can I use 'Page.addScriptToEvaluateOnNewDocument' for all new opened tabs?
If switching to a new tab init user by click link. Or use one injection for all new opened tabs

Implement `baseUrl` query param HTTP API

Send Network.enable
Send Network.setRequestInterceptionEnabled
For each Network.requestIntercepted event, send a continueInterceptedRequest with the new url.

PayloadMixin.convert_payload.convert() errors when payload is a typed list (e.g. [Metric])

Fairly new to this; installed 0.2.11 through PyPi. Tried putting a script together where I call sync_cmd(performance.Performance.getMetrics()), where sync_cmd is the function from the example. It errors out when the async response comes back. Performance.getMetrics() contains this:

cls.convert_payload({
                "metrics": {
                    "class": [Metric],
                    "optional": False
                },
            })

Here's the actual error:

  File "C:\Users\Chris\AppData\Local\Programs\Python\Python36-32\lib\site-packages\chromewhip\helpers.py", line 31, in convert
    if issubclass(expected_type_, ChromeTypeBase):
TypeError: issubclass() arg 1 must be a class

expected_type_ here is [Metric]. [Metric] is technically a list of classes so it tracks that issubclass would complain. I presume def convert should handle lists of classes, or the protocol generator should output a different class to represent a "typed list"?

For context, the example script works fine, as long as I don't push commands with typed list payloads.

TypeError: Cannot read property 'outerHTML' of null

I just started using chromewhip so I'm not sure if this is a bug or a problem from my side.

I started chromewhip service inside docker container and every time instead of the rendered html i get the error TypeError: Cannot read property 'outerHTML' of null

keep in mind my code is inside another docker container.

here's the full error

{ "error": { "reason": "Runtime.evalulate threw an error", "error": { "exceptionId": 1, "text": "Uncaught", "lineNumber": 0, "columnNumber": 25, "scriptId": "13", "url": null, "stackTrace": null, "exception": { "type": "object", "subtype": "error", "className": "TypeError", "description": "TypeError: Cannot read property 'outerHTML' of null\n at :1:26", "objectId": "{\"injectedScriptId\":1,\"id\":2}" }, "executionContextId": null } } }

my query GET http://127.0.0.1:8080/render.html?url=https://github.com

Example from Readme does not work

I am trying to launch code from your readme. Docker container was started using the command:
run --init -it --rm --shm-size=1024m -p 127.0.0.1:8080:8080 --cap-add=SYS_ADMIN chuckus/chromewhip

Error in line: loop.run_until_complete(c.connect())

I am using Native Docker for Windows, probably it can be a problem

Traceback:

File "C:\Users\user\AppData\Local\Programs\Python\Python36-32\lib\asyncio\base_events.py", line 467, in run_until_complete
return future.result()
File "F:\VirtualEnv\proj\lib\site-packages\chromewhip\chrome.py", line 315, in connect
await asyncio.wait_for(self.attempt_tab_fetch(), timeout=5)
File "C:\Users\user\AppData\Local\Programs\Python\Python36-32\lib\asyncio\tasks.py", line 358, in wait_for
return fut.result()
File "F:\VirtualEnv\proj\lib\site-packages\chromewhip\chrome.py", line 321, in attempt_tab_fetch
async with session.get(self._url + '/json') as resp:
File "F:\VirtualEnv\proj\lib\site-packages\aiohttp\client.py", line 637, in aenter
self._resp = yield from self._coro
File "F:\VirtualEnv\proj\lib\site-packages\aiohttp\client.py", line 241, in _request
yield from resp.start(conn, read_until_eof)
File "F:\VirtualEnv\proj\lib\site-packages\aiohttp\client_reqrep.py", line 559, in start
(message, payload) = yield from self._protocol.read()
File "F:\VirtualEnv\proj\lib\site-packages\aiohttp\streams.py", line 509, in read
yield from self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None

No way to generate multiple events produced by a ws

I may be missing the documentation or code section on this but there seemingly is no way to open a new tab and receive WS events indefinitely.

You can navigate to a page and await an individual response.
You can navigate to a page and await some arbitrary event and then pull out the events that happened during that time.
You seemingly can not setup a generator that will return network events or wait for the next event then return it.

Use case:
I am scraping a streaming site for network events sent to an overlay. These are sent json encoded to a websocket opened on page load. I need to consume these events in realtime without having to reload the page after each event. I can do this with PyChromeDevTools but they lack multi-tab support.

Optimally, I would like a way to create an async generator that i could yield from the events or be able to yield from network events from all tabs while being able to add or remove tabs as different streams come and go.

How to wait for non-hashable event "page.LoadEventFiredEvent" ?

setDeviceMetricsOverride() got an unexpected keyword argument 'fitWindow'

So, I started playing with chromewhip and I have it running with Docker (with the command from the README):

2017-09-11 10:07:50,411 chromewhip.chrome.ChromeTab                                      INFO     Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/81b27929-4426-4586-b06d-22c0b063d26f
2017-09-11 10:07:50,412 chromewhip.chrome.Chrome                                         DEBUG    Connected to Chrome! Found 1 tabs
2017-09-11 10:07:50,412 chromewhip.chrome.ChromeTab.recv_handler                         DEBUG    
Waiting for message...
======== Running on http://0.0.0.0 ========
(Press CTRL+C to quit)

And I make a request, let's say:

curl "http://0.0.0.0:8080/render.html?url=http://www.google.com"

This is the response:

{
    "error": "Traceback (most recent call last):
    File \"/usr/src/app/chromewhip/middleware.py\", line 20, in middleware_handler
    response = await handler(request)
    File \"/usr/src/app/chromewhip/views.py\", line 60, in render_html
    tab = await _go(request)
    File \"/usr/src/app/chromewhip/views.py\", line 45, in _go
    fitWindow=False)
    TypeError: setDeviceMetricsOverride() got an unexpected keyword argument 'fitWindow'"
}

Am I doing something wrong?

Example driver code don't work for me

I ran the example driver code and got TimeoutError

[9618:9655:0117/170740.834686:ERROR:browser_gpu_channel_host_factory.cc(108)] Failed to launch GPU process.
DEBUG:asyncio:Using selector: EpollSelector
INFO:chromewhip.chrome.ChromeTab:Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/39cff3b7-fed6-43e6-90da-12e1b003f291
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
INFO:chromewhip.chrome.ChromeTab:Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/83a74865-c5c1-4453-b5cc-b798becc9c5a
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
INFO:chromewhip.chrome.ChromeTab:Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/e70f45a0-793c-4a20-a5ce-2fb2cf45837a
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
INFO:chromewhip.chrome.ChromeTab:Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/61dbddaf-a646-4aa3-98d0-7a242f27ec03
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
INFO:chromewhip.chrome.ChromeTab:Connected to Chrome tab ws://127.0.0.1:9222/devtools/page/0780b1da-3bdb-42bd-aae9-762ca6fe95e2
DEBUG:chromewhip.chrome.Chrome:Connected to Chrome! Found 5 tabs
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
INFO:chromewhip.chrome.ChromeTab.send_handler:Sending command = {"method": "Page.enable", "params": {}, "id": 1}
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{"method": "Page.enable", "params": {}, "id": 1}')
DEBUG:chromewhip.chrome.ChromeTab.send_handler:Waiting for ack event set for id=1
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"id":1,"result":{}}')
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received message, processing...
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Notifying ack event with id=1
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
DEBUG:chromewhip.chrome.ChromeTab.send_handler:Received ack event set for id=1
INFO:chromewhip.chrome.ChromeTab.send_handler:Successfully sent command = {"method": "Page.enable", "params": {}, "id": 1}
INFO:chromewhip.chrome.ChromeTab.send_handler:Sending command = {"method": "Page.navigate", "params": {"url": "http://myip.dnsomatic.com/"}, "id": 2}
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{"method": "Page.navigate", "params": {"url": "http://myip.dnsomatic.com/"}, "id": 2}')
DEBUG:chromewhip.chrome.ChromeTab.send_handler:Waiting for ack event set for id=2
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.frameStartedLoading","params":{"frameId":"9704.1"}}')
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received message, processing...
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received event message!
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received a "Page.frameStartedLoading" event , storing against hash and name...
DEBUG:chromewhip.helpers:generated hash = Page.frameStartedLoading:frameId=9704.1
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Waiting for message...
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"commit","timestamp":197780.337573}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.frameNavigated","params":{"frame":{"id":"9704.1","loaderId":"9704.1","url":"http://myip.dnsomatic.com/","securityOrigin":"http://myip.dnsomatic.com","mimeType":"text/html"}}}')
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received message, processing...
DEBUG:chromewhip.chrome.ChromeTab.recv_handler:Received event message!
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.loadEventFired","params":{"timestamp":197780.340161}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"load","timestamp":197780.340161}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.frameStoppedLoading","params":{"frameId":"9704.1"}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.domContentEventFired","params":{"timestamp":197780.348346}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"DOMContentLoaded","timestamp":197780.348346}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"id":2,"result":{"frameId":"9704.1"}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"firstPaint","timestamp":197780.39997}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"firstContentfulPaint","timestamp":197780.399986}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"firstTextPaint","timestamp":197780.399987}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"firstMeaningfulPaintCandidate","timestamp":197780.399988}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"networkAlmostIdle","timestamp":197780.880721}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"method":"Page.lifecycleEvent","params":{"name":"networkIdle","timestamp":197781.097418}}')
ERROR:chromewhip.chrome.ChromeTab.send_handler:{"method": "Page.navigate", "params": {"url": "http://myip.dnsomatic.com/"}, "id": 2}
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/chrome.py", line 167, in _send
    await asyncio.wait_for(ack_event.wait(), timeout=TIMEOUT_S)  # recv
  File "/usr/lib/python3.6/asyncio/tasks.py", line 362, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mouse/.PyCharmCE2017.1/config/scratches/scratch_25.py", line 32, in <module>
    result = loop.run_until_complete(tab.send_command(cmd, await_on_event_type=await_on_event_type))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/chrome.py", line 264, in send_command
    return await self._send(*command, input_event_cls=input_event_type, trigger_event_cls=await_on_event_type)
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/chrome.py", line 241, in _send
    raise TimeoutError('Unknown cause for timeout to occurs for "%s" with id=%s' % (method, id_))
chromewhip.chrome.TimeoutError: Unknown cause for timeout to occurs for "Page.navigate" with id=2
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<ChromeTab.recv_handler() done, defined at /usr/local/lib/python3.6/dist-packages/chromewhip/chrome.py:71> exception=TypeError("LifecycleEventEvent unable to deserialise: __init__() missing 2 required positional arguments: 'frameId' and 'loaderId'",)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/helpers.py", line 91, in json_to_event
    result = event_cls(**payload['params'])
TypeError: __init__() missing 2 required positional arguments: 'frameId' and 'loaderId'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/chrome.py", line 97, in recv_handler
    event = helpers.json_to_event(result)
  File "/usr/local/lib/python3.6/dist-packages/chromewhip/helpers.py", line 93, in json_to_event
    raise TypeError('%s unable to deserialise: %s' % (event_cls.__name__, e))
TypeError: LifecycleEventEvent unable to deserialise: __init__() missing 2 required positional arguments: 'frameId' and 'loaderId'

Process finished with exit code 1

/usr/local/lib/python3.6/dist-packages/chromewhip/helpers.py
As you can see in debugger there is no frameId and loaderId in params and LifecycleEventEvent constructor needs it!

Is that an error in chromewhip or i understand something wrong?
This error also arrised when using docker and http api.

Edit Docker Container for Chrome Instance

Hi,.

This is less of an "issue" and more of a "I am seeking advice" :)

I wish to run the ChromeWhip Docker installation on a different machine to the Chrome instance running. I appreciate I will need to change the init script, but as a Docker idiot, am unsure how to do so, and then save it.

I have Chrome Dev Protocol running and able to access it on the machines I want it on, just unsure how to edit the docker image to make it work

could you advise me please ?

Thanks and apologies for a newbie level question :)

raise error "websocket.protocol has no attribute OPEN" if websocket connection has exception

Since in code, we just connect websocket without checking connect state, so we only met this error when send_commnd.
But in chromewhip/chrome.py:248:
if self._ws.state != websockets.protocol.OPEN: this would raise error since OPEN is in webscokets.protocol.State.
if self._ws.state != websockets.protocol.State.OPEN: should be right

How would you implement this function at the moment? I'd like to close the browser completely if that's possible?