wheelodex / wheelodex Goto Github PK

SQLAlchemy documentation: https://docs.sqlalchemy.org/en/20/orm/queryguide/api.html#orm-queryguide-yield-per

Possible places where this could be used:

• Fetching wheels for the dump command (currently paginated via
  Flask-SQLAlchemy)

• Assuming it's safe to use Yield Per while modifying the results:

  • Project query in purge_old_versions()
  • Wheel.to_process()
  • process_orphan_wheels()

Thanks for wheelodex! I find the "reverse dependencies" option really useful, but it would be nice to able to sort the list by some proxy for usage (e.g. download count, or possibly the number of their reverse dependencies).

Organize the output by processing errors vs. wheel-inspect validation errors and their respective types.

Base the command on the following scripts I've been using for ad-hoc error dumping:

from pathlib import Path
import sqlalchemy as sa
from sqlalchemy.orm import Session
from wheelodex.models import Wheel

suberrors = {
    "no unique *.dist-info/ directory in wheel": "no-dist-info",
    "headerparser.errors.": "header-error",
    "Invalid name or filename": "invalid-name",
    "Invalid wheel filename": "invalid-name",
}

DIR = Path(__file__).with_name("errors")
DIR.mkdir(exist_ok=True)
for k, v in suberrors.items():
    d = DIR / v
    d.mkdir(exist_ok=True)
    suberrors[k] = d

engine = sa.create_engine("---URL REDACTED---")
session = Session(engine)

for whl in session.scalars(sa.select(Wheel).filter(Wheel.errors.any())):
    for e in whl.errors:
        for k, d in suberrors.items():
            if k in e.errmsg:
                edir = d
                break
        else:
            edir = DIR
        with (edir / f"whl{whl.id}-{e.id}.txt").open("w") as fp:
            print("Filename:", whl.filename, file=fp)
            print("URL:", whl.url, file=fp)
            print("Uploaded:", whl.uploaded, file=fp)
            print("Wheelodex-Version:", e.wheelodex_version, file=fp)
            print(file=fp)
            print(e.errmsg, file=fp)

from pathlib import Path
import sqlalchemy as sa
from sqlalchemy.orm import sessionmaker
from wheelodex.models import WheelData

DIR = Path(__file__).with_name("invalid")
DIR.mkdir(exist_ok=True)

engine = sa.create_engine("---URL REDACTED---")
session = Session(engine)

for data in session.scalars(sa.select(WheelData).filter(WheelData.valid == False)):
    whl = data.wheel
    vdir = DIR / data.raw_data["validation_error"]["type"]
    vdir.mkdir(exist_ok=True)
    with open(str(vdir / f"whl{whl.id}.txt"), "w") as fp:
        print("Filename:", whl.filename, file=fp)
        print("URL:", whl.url, file=fp)
        print("Uploaded:", whl.uploaded, file=fp)
        print("Processed:", data.processed, file=fp)
        print("Size:", whl.size, file=fp)
        print("Wheel-Inspect-Version:", data.wheel_inspect_version, file=fp)
        print("Error-Type:", data.raw_data["validation_error"]["type"], file=fp)
        print("Message:", data.raw_data["validation_error"]["str"], file=fp)

• Include an option for forcing reanalysis of already-analyzed wheels.

• Include an option for fetching details about the given resources from PyPI if they're not already in the database.

  • Include an option for refetching PyPI data?

See https://docs.github.com/en/actions/deployment/about-deployments/deploying-with-github-actions.

Deployment should be triggered whenever a GitHub release is created (which means I'll have to start creating releases for tags).

Store the Ansible Vault password and a private SSH key for [email protected] in GitHub secrets.

See https://ansible.readthedocs.io/projects/lint/configuring/#pre-commit-setup on using ansible-lint with pre-commit.

Blocked by: ansible/ansible-lint#3846

I'm thinking a blue tire-like wheel viewed from the side and tilted up slightly.

The icon should also be used as the picture for the wheelodex GitHub organization.

• Tests for each command (process-orphan-wheels in particular)
• More thorough tests for views

Currently, the deployment pins the PostgreSQL package to prevent unattended security updates, as updating PostgreSQL causes it to restart, breaking Wheelodex's database connection. This is obviously sub-optimal.

Possible resolution: Configure systemd to restart nginx and/or uwsgi whenever PostgreSQL is restarted. In addition, configure unattended upgrades to not run while Wheelodex jobs are running (Use systemd's Conflicts field?).

• It seems the only way to prevent two systemd timer services from running at the same time without causing one of them to fail is to use the flock command.
  • Problem: unattended-upgrades is run as root, and the wheelodex jobs are run as the wheelodex user, so there will likely be permission errors if they both have the same lockfile.

Look into other possible resolutions, as well.

This would require first fixing Wheel.set_data(); see the comment in its source.

Don't select projects that don't have wheels.

Wheelodex has a list of every entry point group defined by a wheel, each one linking to a list of the entry points defined for it. In order to make things less bland and to give people more information on what they're looking at, individual groups can have summaries displayed next to them in the groups list and descriptions displayed at the tops of their entry points list (example). However, this requires someone to write out summaries & descriptions for the entry point groups first, and that's where I could use some help.

If there's an entry point group you're familiar with that's lacking a description, you can add a description to it by creating a pull request modifying the wheelodex/data/entry_points.ini file. Add a section with the same name as the group (keeping the sections in lexicographic order), and give it summary and description fields whose values are CommonMark Markdown describing the group. The description should include what projects consume the entry point group, a brief idea of what defining an entry point in the group accomplishes, and (if it exists) a link to the consuming project's documentation on using the entry point group. See entry_points.ini for examples.

I am looking to analyse the dependency graph between Python packages. Does Wheelodex make this data available as a single download?

Really like what you built. Amazing idea.
Though it's really hard to be able to help if you don't provide a simple way to start running the service on a local machine / server.

If I may, a docker compose recipe would probably be a good start :)

Give dump an option for including wheel processing error details in the dumped structures, and make load support loading this information.

Naïvely, if a release or asset is yanked, it should be deleted from the database — but what if the latest release (or all its wheels) is yanked, the previous release has already been purged, and the project doesn't make another release for some time? (This is similar to #32.)

Possible setup: See "Sending errors via email" at https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-vii-unit-testing-legacy.

The wheels provided by pydantic-core should be a good source of modern tags.

Incomplete list of new tag elements to sort:

• Platforms:

  • macosx_11_0_{arch}
  • manylinux_2_5_{arch}
  • musllinux*

• Architectures:

  • aarch64
  • arm64
  • ppc64le
  • s390x

For use when repeatedly running the playbook again & again due to some stupid mistake that ansible-lint couldn't find.

Give load-entry-points an option for deleting the summaries & descriptions of entry point groups not listed in the input file

Project dependencies that are only required for extras should be distinguished from non-extra dependencies somehow.

Ideas:

• Make "Most Depended-On Projects" display two leaderboards, one taking extras into account and one not.

• In addition to projects' "Reverse Dependencies" counts & lists, add a "Reverse Dependencies (No Extras)" (Working title) count and list

Wheels have a bunch of data in them, but Wheelodex isn't quite taking advantage of all of it — yet. What information do people want to be able to search, browse, or click on?

• Browsing by keywords and/or licenses, similar to PyDigger?
• Browsing by classifiers? (or does PyPI already cover that well enough?)
• Browsing Platform fields?
• Project-URL labels?
• Statistics on wheel Generator fields?
• Searching by namespace packages?
• Listing which projects have the most reverse dependencies?
• Other metadata?

I'll probably get around to implementing most of these eventually, but I'd like some feedback on what people want first.

It seems that PyPI's JSON API now always fills in the relevant digests, and there are no wheels in the database with missing digests. The nullability thus no longer serves a purpose.

Be sure to change all md5 and sha256 fields & arguments in the code to non-None.

Also remove the display of null digests as "[Unknown]" in the wheel_data template.

The deletion of a version from PyPI may leave Wheelodex with no wheels registered for the project even though there may be lower-versioned versions on PyPI with wheels. Try to keep this from happening.

Should a project's display name be updated whenever it gets a new release or wheel?

100 lines should do it.

Try to speed up file search queries with:

CREATE EXTENSION pg_trgm;
    -- ^^ Must be run inside the database by a superuser
CREATE INDEX files_path_idx ON files USING GIN (path gin_trgm_ops);

Do likewise for other columns queried with LIKE/ILIKE?

Note: The SQLAlchemy equivalent of the CREATE INDEX statement appears to be:

sa.Index(
    "files_path_idx",
    File.path,
    postgresql_using="gin",
    postgresql_opts={"path": "gin_trgm_ops"},
)

Use [Unit]Conflicts=?

And store them somewhere other than the wheelodex.org server.

PostgreSQL is currently configured to log queries that take more than one second, but I'm not paying attention to the logs. The logs should be shipped to an ELK stack, Papertrail.com, Data Dog, or whatever hip sysadmins are using these days.

• In the METADATA displays, make each classifier into a hyperlink to a page listing all projects with that classifier.

• Add a page listing all classifiers (linking to pages listing matching projects), alongside matching project counts.

If the latest version of a project doesn't have any wheels, should scan_pypi() instead register the latest version that does?

Show the current Wheelodex version (and a link to Wheelodex's GitHub repo?) at the bottom of every page.

The "Browse Entry Point Groups" page should gain a search box for filtering down to just groups whose names match a given glob pattern. If only one group matches the given pattern, redirect to that group's page.

See https://medium.com/horrible-hacks/using-systemd-as-a-better-cron-a4023eea996d.

To look into: What would happen if a job reached its resource usage? Would this ever cause it to fail or be killed?

Wheelodex is, at time of launch, severely lacking in the aesthetics department. Some sort of styling by someone with a vague grasp of web design and UX could make quite the difference.

Some specific areas in need of beautification include:

• The list of wheels for a project can fill almost the entire page (example); some sort of collapsible display would be nice.

• Header field names in METADATA and WHEEL files often get wrapped at hyphens; might want to keep that from happening

• File paths in RECORD files often get wrapped at slashes; might want to keep that from happening

• The search boxes on the main page should really be aligned. Not sure how to do that ...

• Putting all the page content in a 500pt-wide box is probably a stupid thing to do

• The "recently analyzed wheels" table gets stretched out too much by wheels with long names (e.g., just about anything for Mac OS X)

Because projects without wheels are excluded from search and the "Browse Projects" list, it can be difficult to reach their pages in order to view their reverse dependencies (which is the only thing their pages offer other than a link to PyPI). At the moment, the only way to get to such a page is via either URL manipulation or by clicking on a link from a project that depends on them or in the reverse dependencies list of a project that they depend on.

Idea: Add a checkbox next to the project search input for also searching projects without wheels.

Currently, when a project creation or version creation event is encountered in the PyPI changelog, a Project or Version is created immediately. Project and Version creation should instead be delayed until a wheel is actually uploaded.

The command should delete any projects that don't have any versions and that aren't dependencies of anything.

The command should be run on a schedule, but it'd be fine if it only ran, say, once a week.

• In the METADATA displays, make each keyword into a hyperlink to a page listing all projects with that keyword.

• Add a "Browse Keywords" page listing keywords and their counts, with a search box for filtering by a given glob pattern.

• See https://stackoverflow.com/q/18228994/744178 for how to keep a table of keywords and their counts up to date.

• Should the database normalize all keywords to lower case?

• Monitor whether the site is up and not returning errors.

• Monitor page load times.

• Possible resources:

  • Sentry.io
  • Statuscake
  • https://github.com/lebinh/ngxtop
  • https://github.com/muatik/flask-profiler
  • https://news.ycombinator.com/item?id=18293434