Git Product home page Git Product logo

python_portpicker's Introduction

Python portpicker module

PyPI version GH Action Status

This module is useful for finding unused network ports on a host. If you need legacy Python 2 support, use the 1.3.x releases.

This module provides a pure Python pick_unused_port() function. It can also be called via the command line for use in shell scripts.

If your code can accept a bound TCP socket rather than a port number consider using socket.bind(('localhost', 0)) to bind atomically to an available port rather than using this library at all.

There is a race condition between picking a port and your application code binding to it. The use of a port server by all of your test code to avoid that problem is recommended on loaded test hosts running many tests at a time.

Unless you are using a port server, subsequent calls to pick_unused_port() to obtain an additional port are not guaranteed to return a unique port.

What is the optional port server?

A port server is intended to be run as a daemon, for use by all processes running on the host. It coordinates uses of network ports by anything using a portpicker library. If you are using hosts as part of a test automation cluster, each one should run a port server as a daemon. You should set the PORTSERVER_ADDRESS=@unittest-portserver environment variable on all of your test runners so that portpicker makes use of it.

A sample port server is included. This portserver implementation works but has not spent time in production. If you use it with good results please report back so that this statement can be updated to reflect that. :)

A port server listens on a unix socket, reads a pid from a new connection, tests the ports it is managing and replies with a port assignment port for that pid. A port is only reclaimed for potential reassignment to another process after the process it was originally assigned to has died. Processes that need multiple ports can simply issue multiple requests and are guaranteed they will each be unique.

Typical usage:

import portpicker
test_port = portpicker.pick_unused_port()

Or from the command line:

TEST_PORT=`/path/to/portpicker.py $$`

Or, if portpicker is installed as a library on the system Python interpreter:

TEST_PORT=`python3 -m portpicker $$`

DISCLAIMER

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

python_portpicker's People

Contributors

gpshead avatar nehaljwani avatar nicdumz avatar patricevignola avatar peap avatar piperchester avatar rainwoodman avatar tewalds avatar ujjwalsh avatar yhg1s avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python_portpicker's Issues

Why does _pick_unused_port_without_server() try random ports instead of letting the OS choose?

Upgrading to the latest Go AppEngine runtime (which has an embedded copy of this code) put us into a code path that uses port_picker to select an open port number, which is then passed off to the Go subprocess to bind. Since the upgrade, we've seen a large number of both CI failures and local dev failures stemming from the Go process being unable to bind the port that the python code is asking it to bind.

Digging into the portpicker code, I'm curious why it tries random ports in the 15000-25000 range rather than just letting the OS pick a free port? My understanding is that letting the OS assign a free port would interoperate better with other (non-Python) processes random binding port 0, since the OS will try to ensure that successive port 0 binds don't conflict with each other.

Transfering ports while minimizing races from other threads

Currently between portpicker picks a port, and the port number is sent to another thread / process to use, there can be a large time window where neither the portpicker nor the port consumer holds the ownership of the port. As a consequence, some other user of the socket API can grab that port, causing the portuser to fail to bind that port number.

This pattern is likely the cause of several flaking tests in TensorFlow OSS workflows, and negatively affected the productivity of github contributors to the surrounding community (OpenXLA, JAX etc).

I wonder if we can add some facility in portpicker to facilitate the transfer of allocated ports between the picker thread/process to the user thread/process. It seems to be rather doable using current Python multiprocessing primitives.

Thoughts? @gpshead

Some tests fail on Windows

> python src\tests\portpicker_test.py
sE...s.E.E.
======================================================================
ERROR: testIsPortFree (__main__.PickUnusedPortTest)
This might be flaky unless this test is run with a portserver.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "src\tests\portpicker_test.py", line 167, in testIsPortFree
    sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY,
AttributeError: 'module' object has no attribute 'IPPROTO_IPV6'

======================================================================
ERROR: testPidDefaultsToOwnPid (__main__.PickUnusedPortTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "src\tests\portpicker_test.py", line 91, in testPidDefaultsToOwnPid
    port = portpicker.get_port_from_port_server('portserver')
  File "D:\nwani\m2u\envs\scratch\lib\site-packages\portpicker.py", line 191, in get_port_from_port_server
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
AttributeError: 'module' object has no attribute 'AF_UNIX'

======================================================================
ERROR: testSendsPidToPortServer (__main__.PickUnusedPortTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "src\tests\portpicker_test.py", line 82, in testSendsPidToPortServer
    port = portpicker.get_port_from_port_server('portserver', pid=1234)
  File "D:\nwani\m2u\envs\scratch\lib\site-packages\portpicker.py", line 191, in get_port_from_port_server
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
AttributeError: 'module' object has no attribute 'AF_UNIX'

----------------------------------------------------------------------
Ran 11 tests in 1.226s

FAILED (errors=3, skipped=2)

PortserverFunctionsTest's test_main fails on Debian sid with Python 3.8

During Debian packaging, we noticed that the tests recently started to fail on current Debian sid versions:

PYTHONPATH=src python3 src/tests/portserver_test.py
I0322 08:56:23.837 139989833676608 portserver.py:191] Port 22 unexpectedly in use, last owning pid 0.
I0322 08:56:23.837 139989833676608 portserver.py:191] Port 22559 unexpectedly in use, last owning pid 0.
I0322 08:56:23.837 139989833676608 portserver.py:195] All ports in use.
..I0322 08:56:23.839 139989833676608 portserver.py:191] Port 12344 unexpectedly in use, last owning pid 0.
I0322 08:56:23.839 139989833676608 portserver.py:187] Can't read start time for pid 12345.
.I0322 08:56:23.839 139989833676608 portserver.py:191] Port 12345 unexpectedly in use, last owning pid 0.
I0322 08:56:23.840 139989833676608 portserver.py:187] Can't read start time for pid 12345.
..W0322 08:56:23.841 139989833676608 portserver.py:247] Could not parse request: invalid literal for int() with base 10: b''
W0322 08:56:23.841 139989833676608 portserver.py:247] Could not parse request: invalid literal for int() with base 10: b'\n'
W0322 08:56:23.841 139989833676608 portserver.py:247] Could not parse request: invalid literal for int() with base 10: b'99Z\n'
W0322 08:56:23.842 139989833676608 portserver.py:247] Could not parse request: invalid literal for int() with base 10: b'99 8\n'
.I0322 08:56:23.843 139989833676608 portserver.py:250] Request on behalf of pid 6.
I0322 08:56:23.843 139989833676608 portserver.py:251] cmdline: <MagicMock name='_get_process_command_line()' id='139989822568720'>
.I0322 08:56:23.845 139989833676608 portserver.py:250] Request on behalf of pid 5.
I0322 08:56:23.845 139989833676608 portserver.py:251] cmdline: <MagicMock name='_get_process_command_line()' id='139989822673872'>
.I0322 08:56:23.847 139989833676608 portserver.py:250] Request on behalf of pid 8.
I0322 08:56:23.847 139989833676608 portserver.py:251] cmdline: <MagicMock name='_get_process_command_line()' id='139989822230384'>
D0322 08:56:23.848 139989833676608 portserver.py:261] Allocated port 999 to pid 8
.......E0322 08:56:23.865 139989833676608 portserver.py:345] No ports.  Invalid port ranges in --portserver_static_pool?
I0322 08:56:23.867 139989833676608 portserver.py:359] Serving on @TST-27521
I0322 08:56:23.868 139989833676608 portserver.py:363] Stopping due to ^C.
I0322 08:56:23.870 139989833676608 portserver.py:267] Dumping statistics:
I0322 08:56:23.870 139989833676608 portserver.py:277] client-request-errors 0
I0322 08:56:23.870 139989833676608 portserver.py:277] denied-allocations 0
I0322 08:56:23.870 139989833676608 portserver.py:277] num-ports-managed 1
I0322 08:56:23.870 139989833676608 portserver.py:277] num-ports-checked-for-last-request 0
I0322 08:56:23.870 139989833676608 portserver.py:277] total-allocations 0
I0322 08:56:23.870 139989833676608 portserver.py:370] Goodbye.
F.E0322 08:56:23.873 139989833676608 portserver.py:312] Ignoring unparsable port range ''.
E0322 08:56:23.873 139989833676608 portserver.py:312] Ignoring unparsable port range 'not'.
E0322 08:56:23.874 139989833676608 portserver.py:312] Ignoring unparsable port range 'numbers'.
E0322 08:56:23.874 139989833676608 portserver.py:312] Ignoring unparsable port range '8080-8081x'.
E0322 08:56:23.874 139989833676608 portserver.py:315] Ignoring out of bounds port range '0-1138'.
E0322 08:56:23.874 139989833676608 portserver.py:315] Ignoring out of bounds port range '1138-65536'.
.I0322 08:56:23.874 139989833676608 portserver.py:111] Not allocating a port to invalid pid
I0322 08:56:23.874 139989833676608 portserver.py:116] Not allocating a port to init.
I0322 08:56:23.877 139989833676608 portserver.py:121] Not allocating a port to a non-existent process
.
======================================================================
FAIL: test_main (__main__.PortserverFunctionsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1348, in patched
    return func(*newargs, **newkeywargs)
  File "src/tests/portserver_test.py", line 154, in test_main
    mock_event_loop.run_until_complete.assert_any_call(
  File "/usr/lib/python3.8/unittest/mock.py", line 984, in assert_any_call
    raise AssertionError(
AssertionError: run_until_complete(<Mock name='start_unix_server()' id='139989822229952'>) call not found

----------------------------------------------------------------------
Ran 19 tests in 0.042s

FAILED (failures=1)

Beta release?

Just wanting to ask if 1.5.0b1 is a beta release and there will be an actual 1.5.0 eventually. I am the maintainer of portpicker in Debian and would like to know if I will need to ensure proper version order for future packages.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.