Loop delay with big lists about proxybroker HOT 3 CLOSED

@ksantr,
If I understand you correctly, you use the raw data: broker.find(data=data, limit=1). Right?

Could you insert at the beginning of executable file the following code, run it and send the log file (issue3.log) to constverum [] gmail dot com (you can also attach it here, but be careful, it shows your real ip)?

import logging
logging.basicConfig(
    filename='./issue3.log',
    format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',
    datefmt='[%H:%M:%S]',
    level=logging.DEBUG)

from proxybroker.

@ksantr thank you for logs, they were useful! I understood what was the problem =)

TL;DR
You need to update ProxyBroker. In latest versions this bug is fixed.

Bug: when the limit is reached we closed active tasks, but didn't clear the queue and so the test continued.

# api.py
def _done(self):
    if self._isDone:
        return
    self._isDone = True
    self._to_check.clear() # clear the queue <= added in v0.1.2
    for f in self._on_check:
        if not f.cancelled():
            f.cancel()
    self._tasks.cancel()
    self.push_to_result(None)
    log.debug('Done!')

Tip: use search by key instead of the expectedType property (my fault; not marked as private - this is an experimental):

# bad
if proxy.expectedType == 'H':
    ptype = 'http'
elif proxy.expectedType == 'S':
    ptype = 'socks5'

# good
if 'HTTP' in proxy.types:
    ptype = 'http'
elif 'SOCKS5' in proxy.types:
    ptype = 'socks5'

Just in case some further information:
If we remove the infinite loop (server3.2.py.issue.txt, line 54: while True:), then the execution time will depend on the number of live proxy. In our case, to find 2 live proxies among 22,5 k takes on average 5-15 seconds. Why is such difference? Because proxies are stored in an unordered collection (set)), so the order of proxy check is different every run.

In case we use providers instead of raw data, the execution time takes 2-4 seconds.

from proxybroker.

Thanks for fix.

from proxybroker.