quansight-labs / conda-metadata-app Goto Github PK

Currently, the app will (correctly IMO!) try to show the latest version and latest build. However, there's some mismatch between what the UI sees as the newest available version, and what's actually available in terms of artefact metadata. This often manifests in the following kind of errors (example is from just entering boost-cpp):

Using the dropdown to navigate to an older build usually works, but it's not a great user experience (especially for those who don't know this quirk and wrongly conclude that app is broken). So I think we should probably catch that error and fall back to the last version for which we have metadata.

The second issue is that there seems to be some source of substantial delay with adding entirely new packages to the UI dropdown. For example, we added libboost builds 4 days ago, and they're still not visible in the UI. In many ways, this is the opposite extreme of the above problem. It would be good to understand (for me at least) where this delay comes from, and if we can somehow get it down to under one day.

It should be possible to fetch .conda metadata (.tar.bz2 can't be streamed) with JS only. This works on the browser:

import { RemoteZipPointer } from "@basisai/remote-zip";

(async () => {
    const url = new URL("https://conda.anaconda.org/conda-forge/linux-64/des-fitvd-0.9.19-py39hf3d152e_2.conda");
    const remoteZip = await new RemoteZipPointer({ url }).populate();
    const fileListing = remoteZip.files(); // RemoteZipFile[]
    console.log(fileListing);
})();

Resulting in:

[
    {
        "filename": "metadata.json",
        "size": 31,
        "modified": "2024-02-06T18:29:10",
        "attributes": 25165824
    },
    {
        "filename": "pkg-des-fitvd-0.9.19-py39hf3d152e_2.tar.zst",
        "size": 93411,
        "modified": "1980-01-01T00:00:00",
        "attributes": 25165824
    },
    {
        "filename": "info-des-fitvd-0.9.19-py39hf3d152e_2.tar.zst",
        "size": 8325,
        "modified": "1980-01-01T00:00:00",
        "attributes": 25165824
    }
]

Paired with a zst library, we should be able to fetch the info/ bits like conda-package-streaming does. This could be implemented in the conda-forge.org website and only point here for the .tar.bz2 bits (which are provided by the OCI backend).

Example: https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-64%2Fjinja2-2.10-py36_0.tar.bz2

xref regro/conda-forge-metadata#14

I think the current caching might be incurring in too much RAM usage (tricky to know on streamlit.cloud), because the app has required a reboot a couple times now just to load.

Maybe using something like diskcache would help here.

Hello, I noticed that I can't bookmark another channel other than conda-forge. For example, opening https://conda-metadata-app.streamlit.app/?q=pkgs%2Fmain will instead redirect to https://conda-metadata-app.streamlit.app/?q=conda-forge.