Headless Chrome DevTools Protocol Client (RxJava3 + Kotlin)
License: Apache License 2.0
My name is Mateusz 'Serafin' Gajewski and I'm a Staff Software Engineer at Starburst Data which is a company behind the development of the Trino query engine (a.k.a. PrestoSQL)
You can find more details about me on my website wendigo.pl.
Feel free to reach me if you want to ask me about anything 🙂
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
Chromium can send frame with following structure {"id":7,"error":{"code":-32000,"message":"Inspected target navigated or closed"},"sessionId":"XXXXXX"}. In such a case client tries to deserialize this message to object RequestError. This deserialization ends up with following exception Caught WebSocket exception: kotlinx.serialization.MissingFieldException: Field 'data' is required for type with serial name 'pl.wendigo.chrome.protocol.websocket.RequestError', but it was missing. To fix this issue it should be enough to add default value null to this field. When value is marked as nullable it doesn't mean that it is not required during deserialization. It must have default value to be optional.
Could you fix it and release new version?
Thanks
If you came upon this project and have some wishes to API please post it here
I like to kindly inform that your usage examples are outdated ;)
I couldn't find protocol.headless("about:blank", 1280, 1024).blockingGet()
And I used protocol.session instead :)
But still thanks for awesome library 👍
After the closing browser object and creating new one I hit following exception
exception in thread "main" java.lang.RuntimeException: pl.wendigo.chrome.protocol.RequestFailed: request = RequestFrame(id=1, sessionId=null, method=Target.setDiscoverTargets, params={"discover":true}), error = Could not enqueue message {"discover":true}
How to reproduce
val browser1 = Browser.builder().withAddress("localhost:9222").build()
browser1.close()
val browser2 = Browser.builder().withAddress("localhost:9222").build()
browser2.close()
I see this issue as a bug but maybe I am missing something.
DebuggerFramesStream contains a ReplaySubject with response frames, which keeps all frames forever, and this leads to memory exhaustion as messages are going. This can be "fixed" by limiting size of ReplaySubject (thus disabling ability to replay event streams entirely - but I don't see any usage of this ability)
Browser class does not have any events like targetCreated/targetDestroyed
I need to use CDP directly for these events and the only way to do this is
Making this API public is easy and require almost no changes (I can even make PR by myself)
Can you do that?
Partial stacktrace when deserializing Node.
kotlinx.serialization.json.internal.JsonDecodingException: Encountered unknown key 'compatibilityMode'. Use 'ignoreUnknownKeys = true' in 'Json {}' builder to ignore unknown keys. Current input: .....le.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"} at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24) at pl.wendigo.chrome.api.dom.Node$$serializer.deserialize(Types.kt) at pl.wendigo.chrome.api.dom.Node$$serializer.deserialize(Types.kt:134) at pl.wendigo.chrome.api.dom.GetDocumentResponse$$serializer.deserialize(Domain.kt) at pl.wendigo.chrome.api.dom.GetDocumentResponse$$serializer.deserialize(Domain.kt:862) at pl.wendigo.chrome.protocol.websocket.FrameMapper.deserializeResponseFrame$chrome_reactive_kotlin(FrameMapper.kt:39) at pl.wendigo.chrome.protocol.websocket.WebSocketFramesStream$getResponse$2.apply(WebSocketFramesStream.kt:66) at pl.wendigo.chrome.protocol.websocket.WebSocketFramesStream$getResponse$2.apply(WebSocketFramesStream.kt:28)
Maybe https://github.com/wendigo/chrome-reactive-kotlin/blob/master/src/main/kotlin/pl/wendigo/chrome/protocol/websocket/FrameMapper.kt#L39 needs to be relaxed as well?
Running this simple code leads to unlimited grow of heap size. Target should free up memory after closing or at least it should use some bounded amount of memory.
val browser = Browser.builder().withAddress("localhost:9222").withEventsBufferSize(1).build()
while(true) {
browser.target().close()
Thread.sleep(1000)
}
...
One possible solution is to switch from unbounded buffer in WebSocketFramesStream to bounded buffer. Now you use EventsBufferSize just as hint for size of the buffer. It would be better to use this function instead of this one
Hi Mateusz,
in attempt to get a better control over targets, especially in spawning multiple tabs in multiple browser contexts, I'm hitting a blocker in ChromeDebuggerConnection, which is an internal class.
That prevents me from rewriting HeadlessChromeProtocol.create on my own 😞
Is there a reason for that?
It would be great if the library allowed both high-level and low(er)-level approaches
The exception messages:
io.reactivex.exceptions.OnErrorNotImplementedException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: N/A; line: -1, column: -1] (through reference chain: pl.wendigo.chrome.domain.runtime.ExecutionContextCreatedEvent["context"]->pl.wendigo.chrome.domain.runtime.ExecutionContextDescription["auxData"])
at io.reactivex.internal.functions.Functions$OnErrorMissingConsumer.accept(Functions.java:704)
at io.reactivex.internal.functions.Functions$OnErrorMissingConsumer.accept(Functions.java:701)
at io.reactivex.internal.subscribers.LambdaSubscriber.onError(LambdaSubscriber.java:76)
at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onError(BasicFuseableSubscriber.java:101)
at io.reactivex.internal.operators.flowable.FlowableOnBackpressureBuffer$BackpressureBufferSubscriber.checkTerminated(FlowableOnBackpressureBuffer.java:235)
at io.reactivex.internal.operators.flowable.FlowableOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FlowableOnBackpressureBuffer.java:166)
at io.reactivex.internal.operators.flowable.FlowableOnBackpressureBuffer$BackpressureBufferSubscriber.onError(FlowableOnBackpressureBuffer.java:123)
at io.reactivex.internal.operators.flowable.FlowableFromObservable$SubscriberObserver.onError(FlowableFromObservable.java:47)
at io.reactivex.internal.operators.observable.ObservableSubscribeOn$SubscribeOnObserver.onError(ObservableSubscribeOn.java:63)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle$FlatMapSingleObserver.drainLoop(ObservableFlatMapSingle.java:239)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle$FlatMapSingleObserver.drain(ObservableFlatMapSingle.java:210)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle$FlatMapSingleObserver.innerError(ObservableFlatMapSingle.java:202)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle$FlatMapSingleObserver$InnerObserver.onError(ObservableFlatMapSingle.java:289)
at io.reactivex.internal.operators.single.SingleMap$MapSingleObserver.onError(SingleMap.java:69)
at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
at io.reactivex.Single.subscribe(Single.java:2700)
at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
at io.reactivex.Single.subscribe(Single.java:2700)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle$FlatMapSingleObserver.onNext(ObservableFlatMapSingle.java:113)
at io.reactivex.internal.operators.observable.ObservableFilter$FilterObserver.onNext(ObservableFilter.java:52)
at io.reactivex.internal.operators.observable.ObservableFilter$FilterObserver.onNext(ObservableFilter.java:52)
at io.reactivex.subjects.ReplaySubject$UnboundedReplayBuffer.replay(ReplaySubject.java:655)
at io.reactivex.subjects.ReplaySubject.subscribeActual(ReplaySubject.java:243)
at io.reactivex.Observable.subscribe(Observable.java:10903)
at io.reactivex.internal.operators.observable.ObservableFilter.subscribeActual(ObservableFilter.java:30)
at io.reactivex.Observable.subscribe(Observable.java:10903)
at io.reactivex.internal.operators.observable.ObservableFilter.subscribeActual(ObservableFilter.java:30)
at io.reactivex.Observable.subscribe(Observable.java:10903)
at io.reactivex.internal.operators.observable.ObservableFlatMapSingle.subscribeActual(ObservableFlatMapSingle.java:48)
at io.reactivex.Observable.subscribe(Observable.java:10903)
at io.reactivex.internal.operators.observable.ObservableSubscribeOn$SubscribeTask.run(ObservableSubscribeOn.java:96)
at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:452)
at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:61)
at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: N/A; line: -1, column: -1] (through reference chain: pl.wendigo.chrome.domain.runtime.ExecutionContextCreatedEvent["context"]->pl.wendigo.chrome.domain.runtime.ExecutionContextDescription["auxData"])
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1122)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1075)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:60)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:504)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:400)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1198)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:504)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:400)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1198)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3770)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2099)
at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2596)
at pl.wendigo.chrome.FrameMapper$deserializeEvent$1.call(FrameMapper.kt:68)
at pl.wendigo.chrome.FrameMapper$deserializeEvent$1.call(FrameMapper.kt:12)
at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:35)
... 25 more
The code:
val inspector = Inspector.connect("127.0.0.1:9222")
val protocol = ChromeProtocol.openSession(inspector.openedPages().firstOrError().blockingGet())
protocol.Page.enable().blockingGet()
protocol.Runtime.enable().blockingGet()
val (frameId) = protocol.Page.navigate(NavigateRequest("https://v.qq.com/")).blockingGet()
protocol.Runtime.executionContextCreated().forEach {
println("context")
}
protocol.Page.frameStoppedLoading().blockingForEach {
println("frameStoppedLoading=${frameMap[it.frameId]}")
if (frameId != it.frameId) return@blockingForEach
// top frame
val frame = frameMap[frameId]
println("ROOT frame=$frame")
val (doc) = protocol.DOM.getDocument(GetDocumentRequest()).blockingGet()
}like this title, document url happened 404 not fond error.
https://wendigo.github.io/chrome-reactive-kotlin/.
thanks
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.