Implement HTTP persistent connections for API requests about pypowerwall HOT 7 CLOSED

Based on your research, connection pooling and persistence should be the default. I am wondering if we could present the option to toggle it off or perhaps adjust pool size. What do you think?

That's a great idea. It would give the user the option to control this based on their use case. So in the class definition I could add an option for the pool size like below:

def __init__(self, host="", password="", email="[email protected]", \
    timezone="America/Los_Angeles", pwcacheexpire=5, timeout=10, poolmaxsize=10):

And if the user wanted to disable this, I could add some conditions to the code so if they had set poolmaxsize=0 it would essentially function as it does now, without persistent connections.

I'll work on adding this functionality before submitting the PR.

Defaulting to using persistent connections makes sense from my testing, and the sweet spot for max pool size certainly seems to be about 10.

If using Powerwall-Dashboard it appears most of the time, about 6-7 connections are established, so it does not reach the default max pool size. This is probably due to e.g. the number of input urls configured in telegraf which is 6. If you use Powerwall-Dashboard and are viewing the Power Flow panel, this increases the connection use while being viewed.

I noticed if you reduce the max pool size too much, this will result in similar behaviour to before, where connections are continually opened/closed, as they get discarded when the pool is full:

[urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 10.255.222.2. Connection pool size: 3

This doesn't stop anything from working per se, it just means a new connection is opened again to serve the request.

from pypowerwall.

Nicely done @mcbirse ! Beautiful addition with an elegant approach. ❤️ I'm running tests including some soak tests tonight. If everything looks good, I'll target a release of v0.6.0 to PyPI tomorrow. Thank you! 🙏

from pypowerwall.

Awesome! You're welcome Jason glad I could contribute to such as great project! 👍

from pypowerwall.

Hi @mcbirse - Impressive work! I love this! Good analysis and testing. Currently pypowerwall caches the calls but you are right, after TTL, it creates a new TCP session each time which is unnecessary. Please do submit a PR and feel free to take us to 0.6.0.

Based on your research, connection pooling and persistence should be the default. I am wondering if we could present the option to toggle it off or perhaps adjust pool size. What do you think?

Again, great work! This is a great improvement.

from pypowerwall.

Please do submit a PR and feel free to take us to 0.6.0.

Done - hope you like! 😊

Some notes on these changes:

• Requests to Gateway will now re-use persistent http connections which reduces load and increases response time
• Default connection poolmaxsize=10 to align with Session object defaults. Note: pool use applies to multi-threaded use of pyPowerwall only, e.g. as with the pyPowerwall Proxy Server
• Added env PW_POOL_MAXSIZE to proxy server to allow this to be controlled (persistent connections disabled if set to zero)
• Added env PW_TIMEOUT to proxy server to also allow timeout on requests to be adjusted
• For release prep, pyPowerwall version changed to v0.6.0 and proxy version to t17 (release notes still required)

from pypowerwall.

Soak test looked good - no memory leaks. New version has been pushed to PyPI and Docker Hub for Powerwall-Dashboard. Thanks again, Michael!

from pypowerwall.

No issue reported. Closing this, thanks to @mcbirse great work. ❤️

from pypowerwall.