Comments (8)
@szabi @stefan6419846 micro-releases (the x in 3.9.x) of old Python versions, i.e. <3.10, do not grow new standard library modules (see https://devguide.python.org/developer-workflow/development-cycle/ and https://devguide.python.org/developer-workflow/stdlib/), ergo, the lists that stdlib_list has for those versions are correct now, and will continue to be correct in the future. The machinery for maintaining those lists will fail though, as soon as the documentation indexes for those versions disappear (that is how the lists are generated). It is therefore more reasonable to vendorize those lists than any other approach to work with older versions of Python.
stdlib_list does not currently support Python>=3.10, and is not likely to do so in the future.
The proposed, but not merged, patch for >=3.10 support generates lists simply from
list(sys.stdlib_module_names | set(sys.builtin_module_names))
which has also been suggested numerous times for pydeps. Per my comment above, this list is missing 656 items that are in the Python 3.9 list for stdlib_list.
All of the new modules in 3.10, 3.11 are underscore-prefixed, except two: nt and tomllib. The nt
module is not picked up by stdlib_list due to the way its list is generated, however it is manually added in pydeps (https://github.com/thebjorn/pydeps/blob/master/pydeps/pystdlib.py#L36)).
It would therefore give worse 3.10, 3.11 support today to use the proposed stdlib_list 3.10/1 patch, and it would give long-time issues to try to support stdlib_list's generation mechanism for older Python versions.
Pydeps will therefore continue to use the current published version of stdlib_list, and if needed add extra modules in https://github.com/thebjorn/pydeps/blob/master/pydeps/pystdlib.py#L36 depending on Python version (i.e. tomllib
should only be added if the Python version is >=3.10). If stdlib_list disappears in the future, pydeps will only copy out (i.e. vendorize) the lists that stdlib_list has generated for older versions (as explained above these will continue to be valid and unchanged for older Python versions.
Pydeps might grow a feature where any module starting with a prefix generated from list(sys.stdlib_module_names | set(sys.builtin_module_names))
to more automatically support Python versions > 3.11 but there is currently no need, and it will require rewriting https://github.com/thebjorn/pydeps/blob/master/pydeps/py2depgraph.py#L253 to be a prefix match.
PR are always very welcome if anyone wants to do this.
from pydeps.
This is actually not a huge problem for pydeps. The current stdlib_list package is (and will probably always be) on PyPI and is quite comprehensive for the python versions it supports. Pydeps will use the lists from the latest supported python version from stdlib_list, and since new stdlib modules are added rather infrequently to python this works well.
The modules that are in 3.11 but not in stdlib_list's 3.9 are
{'_contextvars', '_tkinter', '_socket', '_tokenize', 'tomllib', '_symtable', '_warnings', '_md5', '_posixsubprocess', '_random', '_sre', '_sha256', '_struct', '_locale', '_codecs', '_codecs_hk', '_xxsubinterpreters', '_bisect', '_posixshmem', '_sha1', '_operator', '_sqlite3', '_pickle', '_string', 'pyexpat', '_codecs_cn', '_codecs_jp', '_functools', '_io', '_codecs_tw', '_statistics', '_winapi', '_bz2', '_curses_panel', '_abc', '_dbm', '_stat', '_decimal', '_codecs_kr', '_signal', 'xxsubtype', '_msi', '_multibytecodec', 'nt', '_crypt', '_frozen_importlib', '_overlapped', '_csv', '_sha512', '_ssl', '_tracemalloc', '_queue', '_ast', '_curses', '_lzma', '_opcode', '_blake2', '_scproxy', '_zoneinfo', '_uuid', '_imp', '_typing', '_sha3', '_ctypes', '_heapq', '_gdbm', '_weakref', '_frozen_importlib_external', '_multiprocessing', '_lsprof', '_collections', '_hashlib', '_asyncio', '_elementtree', '_datetime', '_codecs_iso2022', '_json'}
Since the list returned by list(sys.stdlib_module_names | set(sys.builtin_module_names))
only contains top-level modules, there are 656 modules that are listed in stdlib_list's 3.9 modules that are not returned from py3.11. I'm not sure there is an efficient way of populating these.
The dev/py310-native-stdlib-list branch (https://github.com/thebjorn/pydeps/tree/dev/py310-native-stdlib-list) implements the py310+ PR to stdlib_list (pypi/stdlib-list#56).
from pydeps.
Given that stdlib-list was archved with "This repo is now archived. I no longer have the spoons to maintain this in my spare time." (from README.md) I guess, even PRs won't be merged.
You could try contacting @jackmaney and maybe fork stdlib-list and learn how he compiled the complete list (or even take over maintenance).
The actual effort to maintaining it might be well worth for pydeps to exist: new Python versions come around once a year, in October, so the maintenance effort likely is limited.
from pydeps.
I'm afraid you might seriously underestimate the effort it takes to maintain a project like stdlib_list, which needs to always be ahead of new python versions and the challenges that poses with installations of alpha/beta/rc versions both locally and in the CI.
If you want to take it on though, I'm sure everyone would be delighted :-) Documentation for how the list was compiled, as well as code for doing it is in the repo.
For pydeps, I'm much more likely to vendorize the resulting lists than taking over maintenance - but since Python versions < 3.10 are already covered (and unlikely to change, since it goes against python's non-current micro-version release policy) there really is no practical point.
Of course, if you have a real issue, and not just a fear of future viability, I would be happy to look at it.
from pydeps.
I absolutely believe that I am underestimating it. However, I would have assumed that for keeping pydeps productive, as a "MVP" it would suffice to keep up to date with stable versions, which come out once a year with the current schedule.
Vendorizing the list does not solve the problem in so far, that the list still has to be compiled somehow (and stdlib-list seems to be up to the task per se, unless it falls behind and stays unmaintained for too long a time).
I don't quite get the "there really is no practical point", as neither the current nor the last-but-current versions are supported: The last supported version, 3.9 is already beyond it's end of full support date (2022-05-17).
I fully understand that you don't want to take over the maintenance of it, I am certainly not trying to convince or urge you, but then, I guess, one has to think about other ways to procure the list (for vendorizing).
from pydeps.
As already mentioned in #95 (comment), Python since version 3.10 does have in-built support for retrieving such a list - it just has to be evaluated to which extent this actually matches with the stdlib_list
data (3.10.txt 3.11.txt).
from pydeps.
Just for reference, stdlib_list
is no longer archived and unmaintained, but has been transferred to the pypi team: https://github.com/pypi/stdlib-list
from pydeps.
@szabi yes, I'm aware :-)
from pydeps.
Related Issues (20)
- The cycles display mode shows useless information and does not work properly for larger projects. HOT 6
- [Feature Request] Specify module name instead of filename HOT 4
- Resolving relative dependencies? HOT 7
- Support for namespace packages HOT 1
- use of `--externals` results in TypeError: 'Config' object does not support item assignment HOT 1
- --dot-output option is not effective HOT 1
- Support output markdown mermaid HOT 1
- empty result with WARNING: SKIPPING ILLEGAL MODULE_NAME: D:.test.a HOT 2
- Support for specifying more than one `fname`
- 1.12.19: pitest fails in `tests/test_package_names.py::test_find_package_names` unit HOT 2
- `--exclude` option not working HOT 1
- Show only top-level imports HOT 1
- Exclude/include dependencies by filepath HOT 1
- Add type hints HOT 1
- Use of pydeps in CI to reject introduction of new import cycles? HOT 3
- Constrain output geometry / size HOT 2
- pydeps randomly generates compiled pyc files when examine python source file HOT 4
- Support for editable packages / direct_url.json HOT 10
- Different types of imports get disconnected. HOT 3
- Feature Request: Pydeps for pipelines HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pydeps.