Comments (20)
This appears to be a QtWebkit bug, see ariya/phantomjs#11390 for more info.
I've been unable to find a solution with dryscrape, and because of instability issues I mentioned in #43, I've switched over to using PhantomJS with selenium, and there I've solved it with:
from selenium.webdriver import PhantomJS
class WebDriver(PhantomJS):
def __init__(self, *a, **kw):
super(WebDriver, self).__init__(*a, **kw)
self._maxRequestCount = 100
self._RequestCount = 0
def get(url):
if self._RequestCount > self._maxRequestCount:
self.__reset()
super(WebDriver, self).get(url)
self._RequestCount += 1
def __reset(self):
try:
self.quit()
except:
print("couldn't quit, do so manually")
self.__dict__ = self.__class__().__dict__
from dryscrape.
Hi @bjmb.
It is actually well possible that there is a memory leak in webkit-server. I don't think many people have used it in a persistent way. If this is possible for your use case, a simple workaround would be to restart webkit-server every once in a while by killing it and creating a new session... Of course a better fix would be to track down the leak, but I'm not sure this is something I will be able to do in the next few weeks or so.
Niklas
from dryscrape.
Thanks for the answer. I will try your advice.
Best
from dryscrape.
Hello,
Another thanks for this awesome package, and unfortunately another report of this condition. You mentioned killing webkit_server and re-starting it as a workaround. I'm new, and can't think of a relatively seamless way to do this. Any suggestions?
Thanks again!
from dryscrape.
Memory leaks are giant! For 70 visited pages almost 15 GB of virtual memory...
from dryscrape.
Hi,
how can I restart the webkit-server? Thanks!
from dryscrape.
I'm with that guy from 4 hours ago. Is there a trick to restarting the server. I could never get my OSError's resolved.
from dryscrape.
Hello guys,
I'm experiencing similar issues here. I'm iterating over approx. 300 urls and after 70-80 urls webkit_server takes up about 3GB of memory. However it is not really the memory that is the problem for me, but it seems that dryscrape/webkit_server is getting slower with each iteration. After the said 70-80 iterations dryscrape is so slow that it raises a timeout error (set timeout = 10 sec) and I need to abort. Restarting the webkit_server (e.g. after every 30 iterations) might help and would empty the memory, however I'm unsure if the 'memory leaks' are really responsible that dryscrape is getting slower and slower.
Does anyone know how to restart the webkit_server so I could test that?
I have not found an acceptable workaround for this issue, however I also don't want to switch to another solution (selenium/phantomjs, ghost.py) as I simply love dryscrape for its simplicity. Dryscrape is working great btw. if one is not iterating over too many urls in one session.
from dryscrape.
It seems like a lot of people could really use the simple workaround of just restarting the server. After my exams for this semester are over I will try to implement this in a way that should be useful to most people with this particular issue.
from dryscrape.
Much appreciated, Niklas!
from dryscrape.
I see this
def reset(self):
""" Resets the current web session. """
self.conn.issue_command("Reset")
In the Client class in webserver.py here. Is this the point of focus if one is attempting a workaround fix for the memory leak issue?
If so I guess I could try to play around with implementing it here
class Driver(webkit_server.Client,
dryscrape.mixins.WaitMixin,
dryscrape.mixins.HtmlParsingMixin):
""" Driver implementation wrapping a ``webkit_server`` driver.
Keyword arguments are passed through to the underlying ``webkit_server.Client``
constructor. By default, `node_factory_class` is set to use the dryscrape
node implementation. """
def __init__(self, **kw):
kw.setdefault('node_factory_class', NodeFactory)
super(Driver, self).__init__(**kw)
from dryscrape.
@trendsetter37 I think that pretty much only triggers https://github.com/niklasb/webkit-server/blob/master/src/NetworkAccessManager.cpp#L50, which clears request headers and username + password for HTTP auth, so no, that doesn't really do anything interesting in that context
from dryscrape.
Also that command is already exposed here: https://github.com/niklasb/webkit-server/blob/master/webkit_server.py#L254 and can be called as Session#reset()
from dryscrape.
Ahh ok I see. I will keep poking around. I know that the Server.cpp catches my eye but I am still unsure if that's going in the right direction. I am willing to take a crack at it...would it be better if I sent an email?
from dryscrape.
If I could just bump this, it takes ~20 minutes to scrape 30 or so links and memory usage climes exponentially, hitting almost 80-90 MB per new link towards the end of those 30. Javascript execution also slows down immensely as the number of links grows.
from dryscrape.
@siddarth so why not start Xvfb once manually and running your python script inside it? What you describe seems to have nothing to do with the rest of this threads
from dryscrape.
@niklasb Thanks and agree
from dryscrape.
I ended up using Phantomjs and approach mentioned by 13steinj .
p.s. There is a bug in his wrapper. get function is missing self. The correct version is get(self, url).
from dryscrape.
Whoops, thanks for the catch. It may be important to note however the only thing that wrapper "solves" is the memory usage (and somethings changed since I wrote that, I believe the driver's .close()
should be called after .quit()
) on a basic level, as in, only one tab, no persistence of data via a browser extension (though this would only apply to firefox/chrome), and doesn't solve selenium's own instability (though it's far rarer than with dryscrape). However due to seleniums structure I ended up being able to write a more in depth wrapper to "solve" those via a reset as well, which I couldn't with dryscrape (I'd share but I'd have to dig up that old project)
from dryscrape.
There is a bug in QT till 5.2.1 when memory increase very fast if auto load images is off https://bugreports.qt.io/browse/QTBUG-34494. See if this helps.
from dryscrape.
Related Issues (20)
- EndOfStreamError when visiting sites (Yelp, google business) remotely HOT 5
- HI contributor question about cookies HOT 1
- Brilliant HOT 1
- Provide instructions for Red Hat / Fedora / CentOS HOT 3
- [request] Session Closed
- dryscrape console.log HOT 3
- Windows 10 Support HOT 1
- Interacting with iframes HOT 2
- Wait javascript page to execute commands HOT 1
- Cannot install HOT 6
- Webkit_server error HOT 6
- {"class":"NodeNotAttachedError","message":"Element at 2 no longer present in the DOM"}
- follow redirects?
- How to make POST request? HOT 2
- image not found error for qt 5.5
- Cannot render page on RedHat, characters all become squares
- Can't get DOM that's generated by javascript HOT 5
- Unable to find QtWebKit on MacOS Darwin HOT 1
- Is it possible to use dryscrape with socks proxy?
- how to override default timeout parameters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dryscrape.