flatpak / flatpak-builder-tools Goto Github PK

Various helper tools for flatpak-builder

Python 94.22% Ruby 0.64% Perl 1.58% Rust 0.34% Shell 1.15% JavaScript 0.11% Swift 1.96%

flatpak-builder-tools's Introduction

Flatpak is a system for building, distributing, and running sandboxed desktop applications on Linux.

See https://flatpak.org/ for more information.

Flatpak is available in the package repositories of most Linux distributions and can be installed from there. See https://flatpak.org/setup/ for quick setup instructions for many distributions.

Community discussion happens in #flatpak:matrix.org, on the mailing list, and on the Flathub Discourse.

Read documentation for Flatpak here.

Contributing

Flatpak welcomes contributions from anyone! Here are some ways you can help:

Fix one of the issues and submit a PR
Update flatpak's translations and submit a PR
Improve flatpak's documentation, hosted at http://docs.flatpak.org and developed over in flatpak-docs
Find a bug and submit a detailed report including your OS, flatpak version, and the steps to reproduce
Add your favorite application to Flathub by writing a flatpak-builder manifest and submitting it
Improve the Flatpak support in your favorite Linux distribution

Hacking

See CONTRIBUTING.md

Related Projects

Here are some notable projects in the Flatpak ecosystem:

Flatseal: An app for managing permissions of Flatpak apps without using the CLI
Flat-manager: A tool for managing Flatpak repositories

flatpak-builder-tools's People

Contributors

Stargazers

Watchers

Forkers

ptomato amtlib-dot-dll jonathon-love ghisvail mdeguzis yuntan g-ray lazka rshuler solarliner adds68 cagatay-y wjt scx refi64 danyeaw gasinvein jbbe bonomali muelli gdesmott tallero nanonyme operutka phw gjanssens patchedsoul cloudnull a6gibkm strugee louib 5l1v3r1 msgpo alatiera saur2000 outergod roypen ws-rush aleb catsout proletarius101 ransome1 joaquimrocha marksvc okias nickavem razzeee hqh2010 lunarequest dusansimic chookity-pokk tpikonen hfiguiere micahflee davidmhewitt ataffanel dreua olof-nord johannesjh tfmoraes mailaender tinywrkb sebastian-de mrmyhuang dylanmccall fabianhauser subhadeepjasu superrnovae tobtobxx freeseacher real-yfprojects abbe98 luciusp thejaskiranps be-ing andersblomdell jakobdev danielsvindseth dennwc projectsynchro developers81828182 picsel2 gabmus da2ce7 trevyn endrain furzmund bbhtt kbdharun sanjacob inukshuk ian2020 parnassius kentoseth felinira tarkah newbytee benoitf zecakeh pieterdd

flatpak-builder-tools's Issues

[cargo] git+https urls seem to be unsupported

This seems to be similar to #66

I want to run the tool on this Cargo.lock file. The interesting bit is this: https://github.com/deltachat/deltachat-core-rust/blob/e251c7b1c825929f69125b6436fe51ed8fe56dc2/Cargo.lock#L499

The generated-sources.json does not contain the module and building the package fails:

    Updating git repository `https://github.com/jonhoo/rust-imap`
warning: spurious network error (2 tries remaining): curl error: Could not resolve host: github.com
; class=Net (12)
warning: spurious network error (1 tries remaining): curl error: Could not resolve host: github.com
; class=Net (12)
error: failed to load source for a dependency on `imap`

It would be good if the generator dealt with git urls.

[pip] possibility to chose checksum algorithm

flatpak-builder-tools adds modules with sha256 checksums in flatpak manifests.

flatpak-builder supports many others: md5, sha1, sha256, and sha512.

Some users may have a preference and appreciates the possibility to chose a specific one, for instance with a --checksum_type=CHECKSUM_TYPE option.

flatpak-node-generator cannot resolve correctly some Node dependencies

I don't know if it is possible to correct this on your side, but I've been trying to Flatpak Standard Notes using the flatpak-node-generator. Previously I had an issue with a similar dependency (spellchecker...gone for good!) and now it's standard-notes-web: https://github.com/standardnotes/desktop/blob/v3.4.0-beta1/app/package.json#L15

So I was wondering if there is a way for flatpak-node-generator to correctly resolve those type of dependencies, or if there is a workaround available; currently the generated-sources.json output file has something like this:

    {
        "type": "file",
        "url": "https://codeload.github.com/standardnotes/web/tar.gz/b075d1b7fb66d0f6235a72f4faa41cb2f2b3b5c6",
        "sha256": "397e93f6dc4978f9d67b9421a833943bbe4e9474fa5e1c65294ab3d0dbe4665a",
        "dest-filename": "standard-notes-web-3.3.5",
        "dest": "flatpak-node/yarn-mirror"
    },
    {
        "type": "file",
        "url": "https://registry.yarnpkg.com/@marshallofsound/enjoi-browser/-/enjoi-browser-1.0.5.tgz#00507c6fb3ee6a032aa45bb4da8d78624b6fec16",
        "sha512": "940678aba0cfaa84af0a6f1fcfe031ba7645848b7918baa78395c874bbd33557be22e4ec17772685aae96db4b5880049f5c8989d82dfe77f620ba2173e50102d",
        "dest-filename": "@marshallofsound-enjoi-browser-1.0.5.tgz",
        "dest": "flatpak-node/yarn-mirror"
    },
   ...

[node] Malformed git+https package url

When running flatpak-node-generator on this package-lock.json file, the URL of the corresponding module is malformed with 4 slashes after https instead of 2. Running flatpak-builder then fails with the following error message: fatal: unable to access 'https:////github.com/duosecurity/duo_web_sdk.git/': URL using bad/illegal format or missing URL.

In package-lock.json:

    "duo_web_sdk": {
      "version": "git+https://github.com/duosecurity/duo_web_sdk.git#410a9186cc34663c4913b17d6528067cd3331f1d",
      "from": "git+https://github.com/duosecurity/duo_web_sdk.git"
    },

In generated-sources.json:

    {
        "type": "git",
        "url": "https:////github.com/duosecurity/duo_web_sdk.git",
        "commit": "410a9186cc34663c4913b17d6528067cd3331f1d",
        "dest": "flatpak-node/git-packages/duo_web_sdk-410a9186cc34663c4913b17d6528067cd3331f1d"
    },

[npm] KeyError on npm-shrinkwrap.json from resin-io/etcher

I am trying to generate the sources for io.resin.etcher. The project only publishes a npm-shrinkwrap.json file as the manifest, and calling flatpak-npm-generator fails with the following error:

$ ./flatpak-npm-generator.py ../../etcher/npm-shrinkwrap.json -o ../../flathub/generated-sources.json
Scanning "../../etcher/npm-shrinkwrap.json" 
Traceback (most recent call last):
  File "./flatpak-npm-generator.py", line 228, in <module>
    main()
  File "./flatpak-npm-generator.py", line 208, in main
    s = getModuleSources(root, None, seen, include_devel=include_devel, npm3=npm3)
  File "./flatpak-npm-generator.py", line 164, in getModuleSources
    child_sources = getModuleSources(deps[dep], seen, include_devel=include_devel)
  File "./flatpak-npm-generator.py", line 96, in getModuleSources
    integrity = module["integrity"]
KeyError: 'integrity'

some package.json provide github commits or tags

As I said, some of npm's package.json have git urls as dependancies. These urls are not converted to sources. Here is a example. After parsing this file, the generated json doesn't have this (there are more such git urls in above json, they are also not generated).

flatpak-pip-generator fails for python-dateutil package

I generated a manifest for the 'python-dateutil' package. flatpak-builder failed to build it, with the following error:

Looking in links: file:///run/build/python3-dateutil
Collecting python-dateutil
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-d9lhe25f --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/python3-dateutil -- "setuptools; python_version != '3.3'" "setuptools<40.0; python_version == '3.3'" wheel setuptools_scm:
  Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
  Looking in links: file:///run/build/python3-dateutil
  Collecting setuptools
    Could not find a version that satisfies the requirement setuptools (from versions: )
  No matching distribution found for setuptools

I'm building against the GNOME 3.32 platform, which does have setuptools already installed. However, notice that pip is being passed --ignore-installed which may be causing it to be ignored.

I worked around this issue by adding 'wheel' and 'setuptools_scm' to the list of packages needed, and by commenting out the line in flatpak-pip-generator that skips the 'setuptools' package. Including those 3 modules in the list of sources for python-dateutil led to a successful build.

pypi-dependencies.json not recognized by flatpak-builder 1.0.8

I used flatpak-pip-generator to generate a pypi-dependencies.json file containing all the pip dependencies but when I try to add it to the manifest, like indicated in the readme, i get this error:

(flatpak-builder:30833): flatpak-builder-ERROR **: 16:24:40.089: Failed to load included manifest (/home/leinardi/Workspace/gitlab/gwe/build/flatpak/pypi-dependencies.json): Expecting a JSON object, but the root node is of type “JsonArray”

This is how my module section looks like:

  "modules": [
    "shared-modules/libappindicator/libappindicator-gtk3-introspection-12.10.json",
    "libdazzle.json",
    "python3-injector.json",
    "python3-setuptools.json",
    "python3-python-dateutil.json",
    "pypi-dependencies.json",
    {
      "name": "gwe",
      "buildsystem": "meson",
      "config-opts": [
        "--buildtype=release"
      ],
      "sources": [
        {
          "type": "git",
          "url": "https://gitlab.com/leinardi/gwe.git",
          "tag": "0.12.3",
          "commit": "0d8ccc6a1e73d8e36ffca08e42c0754d93b9f1e6"
        }
      ]
    }
  ]

This is the content of the pypi-dependencies.json:

[
    {
        "name": "python3-injector",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} injector==0.17.0"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/2a/d1/0dfdf9cbc160883d408d1f610ac75e0b10f0af5da74e4f9d5d9d4b2c38f7/injector-0.17.0.tar.gz",
                "sha256": "882e91f8b4d693cfe29aa4eead3055480d7e2a6fb80a6a2c3b795650db720702"
            }
        ]
    },
    {
        "name": "python3-matplotlib",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} matplotlib==3.1.1"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/12/d1/7b12cd79c791348cb0c78ce6e7d16bd72992f13c9f1e8e43d2725a6d8adf/matplotlib-3.1.1.tar.gz",
                "sha256": "1febd22afe1489b13c6749ea059d392c03261b2950d1d45c17e3aed812080c93"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/c2/4b/137dea450d6e1e3d474e1d873cd1d4f7d3beed7e0dc973b06e8e10d32488/cycler-0.10.0.tar.gz",
                "sha256": "cd7b2d1018258d7247a71425e9f26463dfb444d411c39569972f4ce586b0c9d8"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/16/e7/df58eb8868d183223692d2a62529a594f6414964a3ae93548467b146a24d/kiwisolver-1.1.0.tar.gz",
                "sha256": "53eaed412477c836e1b9522c19858a8557d6e595077830146182225613b11a75"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz",
                "sha256": "d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/ad/99/5b2e99737edeb28c71bcbec5b5dda19d0d9ef3ca3e92e3e925e7c0bb364c/python-dateutil-2.8.0.tar.gz",
                "sha256": "c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/d3/4b/f9f4b96c0b1ba43d28a5bdc4b64f0b9d3fbcf31313a51bc766942866a7c7/numpy-1.16.4.zip",
                "sha256": "7242be12a58fec245ee9734e625964b97cf7e3f2f7d016603f9e56660ce479c7"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/5d/3a/24d275393f493004aeb15a1beae2b4a3043526e8b692b65b4a9341450ebe/pyparsing-2.4.0.tar.gz",
                "sha256": "1873c03321fc118f4e9746baf201ff990ceb915f433f23b395f5580d1840cb2a"
            }
        ]
    },
    {
        "name": "python3-peewee",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} peewee==3.9.6"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/5e/66/5902f91abec78be8ef41eff4e547fe595c9e7b6c20f4be3d8cfb8e079061/peewee-3.9.6.tar.gz",
                "sha256": "bc879e1ffb7f7684f90d9dd236c77b5042118d079f94764f35e3806948829ac8"
            }
        ]
    },
    {
        "name": "python3-py3nvml",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} py3nvml==0.2.3"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/ee/63/59bf7e15b81c038b30931057cec2f3ce1d038a43be7b5b36809daa77229b/py3nvml-0.2.3.tar.gz",
                "sha256": "0a7b32e01c4d94142e9d44d826cd3b9637f86aa4d4e0c18e26464aa1cb9949bf"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/58/40/0d783e14112e064127063fbf5d1fe1351723e5dfe9d6daad346a305f6c49/xmltodict-0.12.0.tar.gz",
                "sha256": "50d8c638ed7ecb88d90561beedbf720c9b4e851a9fa6c47ebd64e99d166d8a21"
            }
        ]
    },
    {
        "name": "python3-pygobject",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} pygobject"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/40/ee/8daf797cda4ffe7724b9d47cbb5e2b44ca6235764622d163c586bd9bd7c5/PyGObject-3.32.2.tar.gz",
                "sha256": "fa82746df059b9e0dcf473e0a9ff0693aa5e064f1ca1e2c46b5ac815bceda1ad"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/48/20/5e83af98eb897935bf7dc39455e892ba866feebb9b7c3b392982866f9958/pycairo-1.18.1.tar.gz",
                "sha256": "70172e58b6bad7572a3518c26729b074acdde15e6fee6cbab6d3528ad552b786"
            }
        ]
    },
    {
        "name": "python3-pyxdg",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} pyxdg==0.26"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/47/6e/311d5f22e2b76381719b5d0c6e9dc39cd33999adae67db71d7279a6d70f4/pyxdg-0.26.tar.gz",
                "sha256": "fe2928d3f532ed32b39c32a482b54136fe766d19936afc96c8f00645f9da1a06"
            }
        ]
    },
    {
        "name": "python3-requests",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} requests"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz",
                "sha256": "c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz",
                "sha256": "11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
                "sha256": "84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/c5/67/5d0548226bcc34468e23a0333978f0e23d28d0b3f0c71a151aef9c3f7680/certifi-2019.6.16.tar.gz",
                "sha256": "945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695"
            },
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/4c/13/2386233f7ee40aa8444b47f7463338f3cbdf00c316627558784e3f542f07/urllib3-1.25.3.tar.gz",
                "sha256": "dbe59173209418ae49d485b87d1681aefa36252ee85884c31346debd19463232"
            }
        ]
    },
    {
        "name": "python3-Rx",
        "buildsystem": "simple",
        "build-commands": [
            "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} Rx==3.0.1"
        ],
        "sources": [
            {
                "type": "file",
                "url": "https://files.pythonhosted.org/packages/9b/27/87ab7fc16a00141e0570ef8fc38c84f09e0840c9ce38ea7f1a79a9b32bcb/Rx-3.0.1.tar.gz",
                "sha256": "c7d168618e3cec35fda9c9c8b5d7f966739df4b1a99b315eae1fc2d7d69e2512"
            }
        ]
    }
]

flatpak-cargo-generator doesn't support the new cargo lock format

I tried to use the flatpak-cargo-generator script for a new rust project and realized it doesn't support the new rust 1.41 cargo.lock format:

https://blog.rust-lang.org/2020/01/30/Rust-1.41.0.html#whats-in-1.41.0-stable

a more git-friendly Cargo.lock

https://blog.rust-lang.org/2020/01/30/Rust-1.41.0.html#less-conflict-prone-cargolock-format

This new format will be used for all new lockfiles, while existing lockfiles will still rely on the previous format. You can learn about the choices leading to the new format in the PR adding it.

The error I got using the new script with a new Cargo.lock file was:

$ python3 flatpak-cargo-generator.py Cargo.lock -o cargo-sources.json                                                                                                   
Traceback (most recent call last):
  File "flatpak-cargo-generator.py", line 216, in <module>
    main()
  File "flatpak-cargo-generator.py", line 210, in main
    generated_sources = generate_sources(load_cargo_lock(args.cargo_lock))
  File "flatpak-cargo-generator.py", line 68, in generate_sources
    metadata = cargo_lock['metadata']
KeyError: 'metadata'

No network access with pip

In my case, during the build, there was no network access. Therefore, I couldn't get pip3 to work. I added this to the generated file from the pip tool, which made it work. It looks like it needs to be added by default:

"build-options": {
    "build-args": [
        "--share=network"
    ]
}

pip-generator tries to build lxml instead of just downloading the package

When I try to generate a manifest for lxml, I get this:

$ ./flatpak-pip-generator lxml
Collecting lxml
  Using cached lxml-4.1.1.tar.gz
  Saved /tmp/pip-generator-lxml-h78x83nm/lxml-4.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    Building lxml version 4.1.1.
    Building without Cython.
    ERROR: b'ERROR: /usr/bin/xslt-config should not be used, use an alternative such as pkg-config\n'
    ** make sure the development packages of libxml2 and libxslt are installed **
    
    Using build configuration of libxslt --should-not-have-used-/usr/bin/xslt-config
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-_4tafq7b/lxml/setup.py", line 240, in <module>
        **setup_extra_options()
      File "/tmp/pip-build-_4tafq7b/lxml/setup.py", line 150, in setup_extra_options
        STATIC_CFLAGS, STATIC_BINARIES)
      File "/tmp/pip-build-_4tafq7b/lxml/setupinfo.py", line 115, in ext_modules
        versions_ok |= check_min_version(lib_versions[1], (1, 1, 23), 'libxslt')
      File "/tmp/pip-build-_4tafq7b/lxml/setupinfo.py", line 378, in check_min_version
        version = tuple(map(int, version.split('.')[:3]))
    ValueError: invalid literal for int() with base 10: '--should-not-have-used-/usr/bin/xslt-config'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_4tafq7b/lxml/
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
  File "./flatpak-pip-generator", line 52, in <module>
    ] + opts.packages, check=True)
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip3', 'download', '--dest', '/tmp/pip-generator-lxml-h78x83nm', '--no-binary', ':all:', 'lxml']' returned non-zero exit status 1

Looks like there's something special about the lxml package that causes it to try to build the package instead of just downloading it.

No licence for NPM code

There isn't a licence on the NPM code. It'd be good if we had one (pip is GPLv3+ but there's no requirement to be in sync) and now, whilst we only have two contributors would be the best time.

@alexlarsson @adityashah1212 what licence do you want?

manifest from flatpak-pip-generator fails to build python3-pdftotext

I generated the manifest for pdftotext and copied it to the manifest for an application that I am developing with Gnome Builder. Building the project with Gnome Builder fails with the error

pdftotext.cpp:3:10: fatal error: poppler/cpp/poppler-document.h: No such file or directory #include <poppler/cpp/poppler-document.h> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated.

How would I get it to build and install poppler-cpp files?

[node] cache downloads

The node generator downloads loads of data. It would be nice if it could cache some of the content, so that it doesn't need to go and fetch each and every package again. The cache would not need to live forever, maybe a day would be sufficient for casual packagers like me who need to fiddle with npm, yarn, the lock file, and the generator until the flathub bots are happy enough to accept the generated files.

[node] yarn --offline - node ./download-chromedriver.js - Error: getaddrinfo EAI_AGAIN github.com

About
Hi! I am trying to package BloomRPC https://github.com/uw-labs/bloomrpc.
Here is my PR on flathub flathub/flathub#1406
I have looked for existing issues like this, but I did not find any.

Build options

  - name: bloomrpc
    sources:
      - type: git
        url: http://github.com/uw-labs/bloomrpc
        commit: '1.4.1'
      - generated-sources.json
      - type: script
        dest-filename: run.sh
        commands:
          - export TMPDIR="$XDG_RUNTIME_DIR/app/$FLATPAK_ID"
          - /app/BloomRPC/bloomrpc
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node12/bin:/app/yarn/bin:$FLATPAK_BUILDER_BUILDDIR/flatpak-node/chromedriver
      env:
        ELECTRON_CACHE: /run/build/bloomrpc/flatpak-node/electron-cache
        CHROMEDRIVER_SKIP_DOWNLOAD: 'true'
        npm_config_nodedir: /usr/lib/sdk/node12
    build-commands:
      - HOME=$PWD yarn config --offline set yarn-offline-mirror $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn-mirror
      - yarn --offline
      - sh flatpak-node/electron-builder-arch-args.sh
      - yarn run --offline package -- $ELECTRON_BUILDER_ARCH_ARGS
      # Max icon size for Flatpaks are 512x512. Having bigger icons results in build error
      - rm -rf build/files/share/icons/hicolor/1024x1024
      - cp -r build/files/* $FLATPAK_DEST
      - install -Dm 755 run.sh $FLATPAK_DEST/bin/run

Output

➜  flathub git:(com.github.uwlabs.BloomRPC) ✗  flatpak-builder --repo=.localrepo --force-clean build com.github.uwlabs.BloomRPC.yaml
Emptying app dir 'build'
Downloading sources
warning: redirecting to https://github.com/uw-labs/bloomrpc/
Fetching git repo http://github.com/uw-labs/bloomrpc, ref refs/tags/1.4.1
warning: redirecting to https://github.com/uw-labs/bloomrpc/
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
Starting build of com.github.uwlabs.BloomRPC
Cache hit for yarn, skipping build
Cache miss, checking out last cache hit
========================================================================
Building module bloomrpc in /home/jonas/git/github/Arxcis/flathub/.flatpak-builder/build/bloomrpc-11
========================================================================
Note: checking out '1.4.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at afe9e60 v1.4.1 (#168)
Running: HOME=$PWD yarn config --offline set yarn-offline-mirror $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn-mirror
yarn config v1.22.4
success Set "yarn-offline-mirror" to "/run/build/bloomrpc/flatpak-node/yarn-mirror".
Done in 0.04s.
Running: yarn --offline
yarn install v1.22.4
[1/4] Resolving packages...
[2/4] Fetching packages...
warning [email protected]: Invalid bin field for "mini-css-extract-plugin".
warning [email protected]: Invalid bin field for "url-loader".
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "prop-types@^15.5.7".
warning " > [email protected]" has incorrect peer dependency "webpack@^3.1.0".
warning "jest > jest-cli > @jest/core > [email protected]" has unmet peer dependency "jest-resolve@^24.1.0".
warning "jest > jest-cli > jest-config > [email protected]" has unmet peer dependency "jest-haste-map@^24.0.0".
[4/4] Building fresh packages...
[-/5] ⢀ waiting...
[2/5] ⢀ electron
[-/5] ⢀ waiting...
[4/5] ⢀ node-sass
error /run/build/bloomrpc/node_modules/electron-chromedriver: Command failed.
Exit code: 1
Command: node ./download-chromedriver.js
Arguments: 
Directory: /run/build/bloomrpc/node_modules/electron-chromedriver
Output:
/run/build/bloomrpc/node_modules/electron-chromedriver/download-chromedriver.js:19
  if (err != null) throw err
                   ^

Error: getaddrinfo EAI_AGAIN github.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26) {
  errno: 'EAI_AGAIN',
  code: 'EAI_AGAIN',




^[[A^Cmake: *** [Makefile:11: build] Interrupt

Thoughts

Clearly yarn --offline tries to install electron-chromedriver which in turn tries to run node ./download-chromedriver.js and this fails because it's offline, right?
I thought setting CHROMEDRIVER_SKIP_DOWNLOAD='true' would fix this, but there is no difference.
Here is the line in electron-chromedriver that throws in the output above - https://github.com/electron/chromedriver/blob/master/download-chromedriver.js#L19
It catches an error from a callback function which is returned from another package, electron-download. Here is the npm package https://www.npmjs.com/package/electron-download, but the github-link redirects me to electron/get which has different source code. Maybe electron-download does not exist on github anymore.

pip-generator should not include all previously downloaded packages in the manifest

If I run flatpak-pip-generator pep8 and then flatpak-pip-generator python-dbusmock, for example, then the generated manifest for python-dbusmock will include the pep8 package in its sources.

pip generator: should it use ${FLATPAK_DEST}/bin/pip3 instead of pip3

hi,

i'm using the pip generator, and it generates pip install commands like this:

pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} name

however, if you're using pip, you would presumably already be including a python module? by using just pip3, the build uses the system python version, which may not match the python version used in the package.

in my case, i have to change all the pip3's to:

${FLATPAK_DEST}/bin/pip3

should the pip generator do this too? or should i be doing this a different way?

with thanks

[pip] fails with feedparser package (regression)

Tried updating the submodule today, which led to this:

Collecting feedparser
  Using cached https://files.pythonhosted.org/packages/91/d8/7d37fec71ff7c9dbcdd80d2b48bcdd86d6af502156fc93846fb0102cb2c4/feedparser-5.2.1.tar.bz2
  Saved /tmp/pip-generator-python3-feedparser-l82ahg_k/feedparser-5.2.1.tar.bz2
Successfully downloaded feedparser
Traceback (most recent call last):
  File "flatpak-builder-tools/pip/flatpak-pip-generator", line 132, in <module>
    name = get_package_name(filename)
  File "flatpak-builder-tools/pip/flatpak-pip-generator", line 58, in get_package_name
    'Downloaded filename: {} does not end with gz, zip, or whl'.format(filename)
Exception: Downloaded filename: feedparser-5.2.1.tar.bz2 does not end with gz, zip, or whl

Looking at distutils code (https://github.com/python/cpython/blob/c5d5dfdb223efb0e668e3f317d31b8b70ae96aa6/Lib/distutils/archive_util.py), xz is also possible.

flatpak-pip-generator: Use existing parser

Use existing parsers like setuptools or requirements-parser. The current parser seems to work only with exact package versions and not VCS specifiers, greater-than specifiers etc.

[pip-generator/Archlinux] Unknow proprety xxxx for type BuilderManifest

Hello there,

I got some strange error and I don't know if it is an excepted behavior. I hope that somebody may help me understand what I'm doing wrong.

Thanks in advance.

`flatpak{-builder}`

I installed flatpak{-builder} from archlinux/extra (pacman -S flatpak flatpak-builder)

`pip-generator`

I got pip-generator with:

$ wget https://raw.githubusercontent.com/flatpak/flatpak-builder-tools/master/pip/flatpak-pip-generator
$ chmod +x flatpak-pip-generator

Creation of the manifest

I created the manifest with (I added the script to my ${PATH}:

$ flatpak-pip-generator --cleanup all pyfunceble-dev

and I got the following manifest

{
    "name": "python3-pyfunceble-dev",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} pyfunceble-dev"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/2a/fb/aefe5d5dbc3f4fe1e815bcdb05cbaab19744d201bbc9b59cfa06ec7fc789/wheel-0.31.1.tar.gz",
            "sha256": "0a2e54558a0628f2145d2fc822137e322412115173e8a2ddbe1c9024338ae83c"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/69/81/52b68d0a4de760a2f1979b0931ba7889202f302072cc7a0d614211bc7579/pip-18.0.tar.gz",
            "sha256": "a0e11645ee37c90b40c46d607070c4fd583e2cd46231b1c06e389c5e814eed76"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/53/0d/d1d13a63563cc50a27b310f5612645bef06d29a5022a7e79ac659dd0fc50/certifi-2018.8.13.tar.gz",
            "sha256": "4c1d68a1408dd090d2f3a869aa94c3947cc1d967821d1ed303208c9f41f0f2f4"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/65/c4/80f97e9c9628f3cac9b98bfca0402ede54e0563b56482e3e6e45c43c4935/idna-2.7.tar.gz",
            "sha256": "684a38a6f903c1d71d6d5fac066b58d7768af4de2b832e426ec79c30daa94a16"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
            "sha256": "84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz",
            "sha256": "3ef3092145e9b70e3ddd2c7ad59bdd0252a94dfe3949721633e41344de00a6bf"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/71/81/00184643e5a10a456b4118fc12c96780823adb8ed974eb2289f29703b29b/pypandoc-1.4.tar.gz",
            "sha256": "e914e6d5f84a76764887e4d909b09d63308725f0cbb5293872c2c92f07c11a5b"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/3c/d2/dc5471622bd200db1cd9319e02e71bc655e9ea27b8e0ce65fc69de0dac15/urllib3-1.23.tar.gz",
            "sha256": "a68ac5e15e76e7e5dd2b8f94007233e01effe3e50e8daddf69acfd81cb686baf"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/e6/76/257b53926889e2835355d74fec73d82662100135293e17d382e2b74d1669/colorama-0.3.9.tar.gz",
            "sha256": "48eb22f4f8461b1df5734a074b57042430fb06e1d61bd1e11b078c0fe6d7a1f1"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz",
            "sha256": "ec22d826a36ed72a7358ff3fe56cbd4ba69dd7a6718ffd450ff0e9df7a47ce6a"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/42/e9/00b10d8b5eef9946e3c9fa5c1739164365765c32187d261e6d4557b12bec/PyFunceble-dev-0.94.1.tar.gz",
            "sha256": "03129c782b283122e837abfd5bf17fd17ecadff331240b3efb99b705570acd71"
        }
    ],
    "cleanup": [
        "*"
    ]
}

Trying to build

I did

$ mkdir build-flat
$ flatpak-builder build-flat/ python3-pyfunceble-dev.json

and I got

(flatpak-builder:24769): flatpak-builder-WARNING **: 20:39:05.320: Unknown property name for type BuilderManifest

(flatpak-builder:24769): flatpak-builder-WARNING **: 20:39:05.320: Unknown property buildsystem for type BuilderManifest

(flatpak-builder:24769): flatpak-builder-WARNING **: 20:39:05.320: Unknown property build-commands for type BuilderManifest

(flatpak-builder:24769): flatpak-builder-WARNING **: 20:39:05.320: Unknown property sources for type BuilderManifest
Failed to init: sdk not specified

Versions

$ flatpak --version
Flatpak 0.99.3

[cargo] provide an example

I didn't manage to make an offline build using the cargo generator. Would be good to provide an example as the one we have for node.

[yarn] git dependency generates a type file for source

taken from cozy-drive, one dependency is declared as github url

[email protected]:
  version "0.6.2"
  resolved "https://registry.yarnpkg.com/cozy-client-js/-/cozy-client-js-0.6.2.tgz#5aac0cdb497db3ff06c6da911bb051dd80ab143e"
  dependencies:
    pouchdb "6.1.1"
    pouchdb-adapter-cordova-sqlite "https://github.com/SnceGroup/pouchdb-adapter-cordova-sqlite.git#f3ee23009b70209c611435d57491aa77fb88802a"
    pouchdb-find "0.10.4"

Once generated it gives

…
    {
        "type": "file",
        "url": "https://github.com/SnceGroup/pouchdb-adapter-cordova-sqlite.git",
        "sha1": "f3ee23009b70209c611435d57491aa77fb88802a",
        "dest": "yarn-mirror",
        "dest-filename": "pouchdb-adapter-cordova-sqlite@https:--github.com-SnceGroup-pouchdb-adapter-cordova-sqlite.git#f3ee23009b70209c611435d57491aa77fb88802a\":-2.0.2.tgz"
    },
…

but type should be "git" and "dest-filename" shouldn't exist from flatpak-builder

flatpak-pip-generator: YAML support

Allow outputting the generated pip module in YAML format.

flatpak-pip-generator not working for matplotlib

If I generate the json module for matplotlib using flatpak-pip-generator and than I try to use it it fails with a numpy error:

Running: pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} matplotlib
Looking in links: file:///run/build/python3-matplotlib
Collecting matplotlib
Complete output from command python setup.py egg_info:
Download error on https://pypi.python.org/simple/numpy/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
Couldn't find index page for 'numpy' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
No local packages or working download links found for numpy>=1.10.0

If you manually add the command to install numpy if fails with python-dateutil not finding setuptools.

I managed to workaround this issue with this module configuration (not sure if the sed command is necessary):

{
  "name": "python-setuptools_scm",
  "buildsystem": "simple",
  "build-commands": [
    "python3 setup.py install --prefix=/app --root=/"
  ],
  "sources": [
    {
      "type": "archive",
      "url": "https://files.pythonhosted.org/packages/09/b4/d148a70543b42ff3d81d57381f33104f32b91f970ad7873f463e75bf7453/setuptools_scm-3.1.0.tar.gz",
      "sha256": "1191f2a136b5e86f7ca8ab00a97ef7aef997131f1f6d4971be69a1ef387d8b40"
    },
    {
      "type": "shell",
      "commands": [
        "sed -i 's/if not platform_/if not dirs/' setup.py"
      ]
    }
  ]
}

{
  "name": "python3-python-dateutil",
  "buildsystem": "simple",
  "build-commands": [
    "python3 setup.py install --prefix=/app --root=/"
  ],
  "sources": [
    {
      "type": "archive",
      "url": "https://files.pythonhosted.org/packages/0e/01/68747933e8d12263d41ce08119620d9a7e5eb72c876a3442257f74490da0/python-dateutil-2.7.5.tar.gz",
      "sha256": "88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02"
    },
    {
      "type": "shell",
      "commands": [
        "sed -i 's/if not platform_/if not dirs/' setup.py"
      ]
    }
  ]
}

{
    "name": "python3-matplotlib",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} numpy",
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} matplotlib"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/c2/4b/137dea450d6e1e3d474e1d873cd1d4f7d3beed7e0dc973b06e8e10d32488/cycler-0.10.0.tar.gz",
            "sha256": "cd7b2d1018258d7247a71425e9f26463dfb444d411c39569972f4ce586b0c9d8"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/31/60/494fcce70d60a598c32ee00e71542e52e27c978e5f8219fae0d4ac6e2864/kiwisolver-1.0.1.tar.gz",
            "sha256": "ce3be5d520b4d2c3e5eeb4cd2ef62b9b9ab8ac6b6fedbaa0e39cdb6f50644278"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz",
            "sha256": "d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/0e/01/68747933e8d12263d41ce08119620d9a7e5eb72c876a3442257f74490da0/python-dateutil-2.7.5.tar.gz",
            "sha256": "88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/89/0c/653aec68e9cfb775c4fbae8f71011206e5e7fe4d60fcf01ea1a9d3bc957f/matplotlib-3.0.2.tar.gz",
            "sha256": "c94b792af431f6adb6859eb218137acd9a35f4f7442cea57e4a59c54751c36af"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/2d/80/1809de155bad674b494248bcfca0e49eb4c5d8bee58f26fe7a0dd45029e2/numpy-1.15.4.zip",
            "sha256": "3d734559db35aa3697dadcea492a423118c5c55d176da2f3be9c98d4803fc2a7"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/d0/09/3e6a5eeb6e04467b737d55f8bba15247ac0876f98fae659e58cd744430c6/pyparsing-2.3.0.tar.gz",
            "sha256": "f353aab21fd474459d97b709e527b5571314ee5f067441dc9f88e33eecd96592"
        }
    ]
}

[node] Example of using git packages in the flatpak manifest

The current examples assumes all dependencies are fetched from a single cached location under $FLATPAK_BUILDER_BUILDDIR/flatpak-node/npm-cache.

Some applications use a mixture of npm and git packages, the latter being cached under a different location at $FLATPAK_BUILDER_BUILDDIR/flatpak-node/git-packages. It would be nice to have an example or piece of documentation explaining how to use them with npm and yarn. So far, I have failed to tell npm install to use both cache locations to resolve offline dependencies.

npm: Electron v3.0.11-arm fails

I'm getting this exception

flatpak-npm-generator.py --recursive package-lock.json 
Scanning "package-lock.json" 
Traceback (most recent call last):
  File "flatpak-npm-generator.py", line 259, in <module>
    main()
  File "flatpak-npm-generator.py", line 239, in main
    s = getModuleSources(root, None, seen, include_devel=include_devel, npm3=npm3)
  File "flatpak-npm-generator.py", line 195, in getModuleSources
    child_sources = getModuleSources(deps[dep], seen, include_devel=include_devel)
  File "flatpak-npm-generator.py", line 182, in getModuleSources
    "sha256": shasums[basename],
KeyError: 'electron-v3.0.11-linux-arm.zip'

with this package-lock.json file:
package-lock.json.log

Rust (Cargo) support

It would be nice to have Rust (Cargo) support! this would allow an easier build for Fractal/Podcasts in gnome-apps-nightly.

Yarn support for the npm tools

Redirecting the issue from the previous repo: alexlarsson/flatpak-npm-generator#1

[yarn] Potentially malformed generated source

Same context as in #20.

The corresponding generated-sources.json contains the following entry which looks malformed:

{
        "type": "file",
        "url": "data:4497dd6b2f742ce8de3d0fedd5b109dff8abfb9f0c004e2e81ae7f0bf00cf1b0%20%2Aelectron-v1.8.3-linux-ia32.zip%0A0bd8ba049ee63282f00ff25208d77bfb6500fba88181bb919219373557f79515%20%2Aelectron-v1.8.3-linux-x64.zip%0A3e282d3caa1fdff3b97dfa1718e2a9fa4e5658e91b9bdfbbc55a8815c79087ae%20%2Aelectron-v1.8.3-linux-arm.zip%0A",
        "dest": "electron-cache",
        "dest-filename": "SHASUMS256.txt-1.8.3"
}

Am I correct?

flatpak-npm-generator.py: Electron v1.6.2 fails for arm64

I'm getting this exception:

flatpak-npm-generator.py --recursive package-lock.json
Scanning "package-lock.json" 
Traceback (most recent call last):
  File "flatpak-npm-generator.py", line 259, in <module>
    main()
  File "flatpak-npm-generator.py", line 239, in main
    s = getModuleSources(root, None, seen, include_devel=include_devel, npm3=npm3)
  File "flatpak-npm-generator.py", line 195, in getModuleSources
    child_sources = getModuleSources(deps[dep], seen, include_devel=include_devel)
  File "flatpak-npm-generator.py", line 182, in getModuleSources
    "sha256": shasums[basename],
KeyError: 'electron-v1.6.2-linux-arm64.zip'

This is because there are no arm64 builds for older versions of electron.
PR on the way.

flatpak-pip-generator not working for wxPython

I used flatpak-pip-generator to generate the necessary JSON for wxPython, but using the JSON, the build process stopped while trying to install the numpy dependency of wxPython:

========================================================================
Building module python3-wxPython in /home/jm/Projekte/org.kicad_pcb.KiCad/.flatpak-builder/build/python3-wxPython-7
========================================================================
Running: pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} wxPython
Looking in links: file:///run/build/python3-wxPython
Collecting wxPython
Collecting pillow (from wxPython)
Requirement already satisfied: six in /usr/lib/python3.7/site-packages (from wxPython) (1.12.0)
Collecting numpy (from wxPython)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-y_okrg_o/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/python3-wxPython -- setuptools wheel 'Cython>=0.29.14'
       cwd: None
  Complete output (4 lines):
  Looking in links: file:///run/build/python3-wxPython
  Collecting setuptools
    ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
  ERROR: No matching distribution found for setuptools
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-y_okrg_o/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/python3-wxPython -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.
Error: module python3-wxPython: Der Kindprozess wurde mit Status 1 beendet

This is the JSON that was generated after invoking flatpak-pip-generator wxPython:

{
    "name": "python3-wxPython",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} wxPython"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/40/de/0ea5092b8bfd2e3aa6fdbb2e499a9f9adf810992884d414defc1573dca3f/numpy-1.18.1.zip",
            "sha256": "b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/21/9f/b251f7f8a76dec1d6651be194dfba8fb8d7781d10ab3987190de8391d08e/six-1.14.0.tar.gz",
            "sha256": "236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/39/47/f28067b187dd664d205f75b07dcc6e0e95703e134008a14814827eebcaab/Pillow-7.0.0.tar.gz",
            "sha256": "4d9ed9a64095e031435af120d3c910148067087541131e82b3e8db302f4c8946"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl",
            "sha256": "8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/b9/8b/31267dd6d026a082faed35ec8d97522c0236f2e083bf15aff64d982215e1/wxPython-4.0.7.post2.tar.gz",
            "sha256": "5a229e695b64f9864d30a5315e0c1e4ff5e02effede0a07f16e8d856737a0c4e"
        }
    ]
}

I searched for clues in the other pip issues that looked similar, and tried a few things I found, like adding --no-build-isolation, or adding setuptools to flatpak-pip-generator, but none of these helped.

Manifest for requests lib does not work with gnome builder

I ran the flatpak-pip-generator requests and got the following output:

{
    "name": "python3-requests",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} requests"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz",
            "sha256": "de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/55/54/3ce77783acba5979ce16674fc98b1920d00b01d337cfaaf5db22543505ed/certifi-2018.11.29.tar.gz",
            "sha256": "47f9c83ef4c0c621eaef743f133f09fa8a74a9b75f037e8624f83bd1b6626cb7"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
            "sha256": "84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz",
            "sha256": "502a824f31acdacb3a35b6690b5fbf0bc41d63a24a45c4004352b0242707598e"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz",
            "sha256": "c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407"
        }
    ]
}

When I copy this into my manifest gnome builder fails building.

flatpak-pip-generator: fails to install poetry

Poetry is a more modern way to manage package dependencies with Python. It currently doesn't work with flatpak-pip-generator.

(Gaphor) dan@dan-Librem-13-v2:~/Projects/gaphor$ python flatpak/pip-generator.py poetry
Collecting poetry
  Using cached https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gu40j27d/poetry-0.12.11.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pkginfo<2.0,>=1.4 (from poetry)
  Using cached https://files.pythonhosted.org/packages/6c/04/fd6683d24581894be8b25bc8c68ac7a0a73bf0c4d74b888ac5fe9a28e77f/pkginfo-1.5.0.1.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gu40j27d/pkginfo-1.5.0.1.tar.gz
Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry)
  Using cached https://files.pythonhosted.org/packages/86/f9/e80fa23edca6c554f1994040064760c12b51daff54b55f9e379e899cd3d4/requests-toolbelt-0.8.0.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gu40j27d/requests-toolbelt-0.8.0.tar.gz
Collecting tomlkit<0.6.0,>=0.5.1 (from poetry)
  Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gu40j27d/tomlkit-0.5.3.tar.gz
  Installing build dependencies ... error
  Complete output from command /home/dan/Projects/gaphor/.venv/bin/python3 /home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bwjf36dl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3:
  Collecting poetry>=0.12a3
    Using cached https://files.pythonhosted.org/packages/93/8a/270aa699a07405df9b260f9192eec043cf486b4b49fafd7ebcee09cb2cc5/poetry-1.0.0a2.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting requests<3.0,>=2.18 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz
  Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/1f/7f/a020327823b9c405ee6f85ab3053ff171e10801b19cfe55c78bb0b3810e7/jsonschema-3.0.1.tar.gz
  Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/8c/46/4e93ab8a379d7efe93f20a0fb8a27bdfe88942cc954ab0210c3164e783e0/pyrsistent-0.14.11.tar.gz
  Collecting html5lib<2.0,>=1.0 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/85/3e/cf449cf1b5004e87510b9368e7a5f1acd8831c2d6691edd3c62a0823f98f/html5lib-1.0.1.tar.gz
  Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/5e/f0/2c193ed1f17c97ae539da7e1c2d48b80d8cccb1917163b26a91ca4355aa6/CacheControl-0.12.5.tar.gz
  Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/b9/b8/6b32b3e84014148dcd60dd05795e35c2e7f4b72f918616c61fdce83d27fc/pyparsing-2.3.1.tar.gz
  Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/86/f9/e80fa23edca6c554f1994040064760c12b51daff54b55f9e379e899cd3d4/requests-toolbelt-0.8.0.tar.gz
  Collecting shellingham<2.0,>=1.1 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/04/47/c3f0b5fce44827d011f7f4b8831dd20984a228d7a7f46d4d45f1aaa6fd90/shellingham-1.2.8.tar.gz
  Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12a3)
    Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
  Exception:
  Traceback (most recent call last):
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 179, in main
      status = self.run(options, args)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 315, in run
      resolver.resolve(requirement_set)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
      self._resolve_one(requirement_set, req)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
      self.require_hashes
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 348, in prepare_linked_requirement
      with self.req_tracker.track(req):
    File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/contextlib.py", line 112, in __enter__
      return next(self.gen)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 86, in track
      self.add(req)
    File "/home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 55, in add
      % (link, fp.read()))
  LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry)
  
  ----------------------------------------
Command "/home/dan/Projects/gaphor/.venv/bin/python3 /home/dan/Projects/gaphor/.venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bwjf36dl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3" failed with error code 2 in None
Traceback (most recent call last):
  File "flatpak/pip-generator.py", line 75, in <module>
    ] + opts.packages, check=True)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/subprocess.py", line 481, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip3', 'download', '--dest', '/tmp/pip-generator-python3-poetry-gu40j27d', '--no-binary', ':all:', 'poetry']' returned non-zero exit status 1.

I can get around some of these errors by remove the --no-binary :all: option from pip-generator, but then I get 404 errors for some of the packages.

(Gaphor) dan@dan-Librem-13-v2:~/Projects/gaphor$ python flatpak/pip-generator.py poetry
Collecting poetry
  Using cached https://files.pythonhosted.org/packages/64/2d/8221b951085667039cc45b4a9013f7cd4efd34f13dfcca04a90ed4046c72/poetry-0.12.11-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/poetry-0.12.11-py2.py3-none-any.whl
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry)
  Using cached https://files.pythonhosted.org/packages/5e/f0/2c193ed1f17c97ae539da7e1c2d48b80d8cccb1917163b26a91ca4355aa6/CacheControl-0.12.5.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/CacheControl-0.12.5.tar.gz
Collecting cachy<0.3,>=0.2 (from poetry)
  Using cached https://files.pythonhosted.org/packages/26/37/8ce3e7b330078b6797a34e79a80a8ad6935e404a3b903765417182c9ce19/cachy-0.2.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/cachy-0.2.0-py2.py3-none-any.whl
Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry)
  Using cached https://files.pythonhosted.org/packages/97/8a/d710f792d6f6ecc089c5e55b66e66c3f2f35516a1ede5a8f54c13350ffb0/requests_toolbelt-0.8.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/requests_toolbelt-0.8.0-py2.py3-none-any.whl
Collecting shellingham<2.0,>=1.1 (from poetry)
  Using cached https://files.pythonhosted.org/packages/74/b7/36cdb13e9ecf1e9584cd78a47a1853831cf8817da3438aeec550cd83619c/shellingham-1.2.8-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/shellingham-1.2.8-py2.py3-none-any.whl
Collecting pyparsing<3.0,>=2.2 (from poetry)
  Using cached https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/pyparsing-2.3.1-py2.py3-none-any.whl
Collecting html5lib<2.0,>=1.0 (from poetry)
  Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/html5lib-1.0.1-py2.py3-none-any.whl
Collecting cleo<0.7.0,>=0.6.7 (from poetry)
  Using cached https://files.pythonhosted.org/packages/a7/b9/270301a3a87587f09bc3985973f2e362ffa45fa5fcd5128501516b2f5e31/cleo-0.6.8-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/cleo-0.6.8-py2.py3-none-any.whl
Collecting pkginfo<2.0,>=1.4 (from poetry)
  Using cached https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/pkginfo-1.5.0.1-py2.py3-none-any.whl
Collecting requests<3.0,>=2.18 (from poetry)
  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/requests-2.21.0-py2.py3-none-any.whl
Collecting jsonschema<4.0,>=3.0a3 (from poetry)
  Using cached https://files.pythonhosted.org/packages/aa/69/df679dfbdd051568b53c38ec8152a3ab6bc533434fc7ed11ab034bf5e82f/jsonschema-3.0.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/jsonschema-3.0.1-py2.py3-none-any.whl
Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry)
  Using cached https://files.pythonhosted.org/packages/8c/46/4e93ab8a379d7efe93f20a0fb8a27bdfe88942cc954ab0210c3164e783e0/pyrsistent-0.14.11.tar.gz
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/pyrsistent-0.14.11.tar.gz
Collecting tomlkit<0.6.0,>=0.5.1 (from poetry)
  Using cached https://files.pythonhosted.org/packages/71/c6/06c014b92cc48270765d6a9418d82239b158d8a9b69e031b0e2c6598740b/tomlkit-0.5.3-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/tomlkit-0.5.3-py2.py3-none-any.whl
Collecting msgpack (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
  Using cached https://files.pythonhosted.org/packages/a8/7b/630049fc4af9e68a625738612edc264ce7cb586c5001a2d4d2209a4f61c1/msgpack-0.6.1-cp37-cp37m-manylinux1_x86_64.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/msgpack-0.6.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting lockfile>=0.9 (from cachecontrol[filecache]<0.13.0,>=0.12.4->poetry)
  Using cached https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/lockfile-0.12.2-py2.py3-none-any.whl
Collecting webencodings (from html5lib<2.0,>=1.0->poetry)
  Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/webencodings-0.5.1-py2.py3-none-any.whl
Collecting six>=1.9 (from html5lib<2.0,>=1.0->poetry)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/six-1.12.0-py2.py3-none-any.whl
Collecting pastel<0.2.0,>=0.1.0 (from cleo<0.7.0,>=0.6.7->poetry)
  Using cached https://files.pythonhosted.org/packages/9b/7e/7d701686013c0d7dae62e0977467232a6adc2e562c23878eb3cd4f97d02e/pastel-0.1.0-py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/pastel-0.1.0-py3-none-any.whl
Collecting pylev<2.0,>=1.3 (from cleo<0.7.0,>=0.6.7->poetry)
  Using cached https://files.pythonhosted.org/packages/40/1c/7dff1d242bf1e19f9c6202f0ba4e6fd18cc7ecb8bc85b17b2d16c806e228/pylev-1.3.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/pylev-1.3.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0,>=2.18->poetry)
  Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/certifi-2018.11.29-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests<3.0,>=2.18->poetry)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests<3.0,>=2.18->poetry)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/urllib3-1.24.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0,>=2.18->poetry)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/chardet-3.0.4-py2.py3-none-any.whl
Collecting setuptools (from jsonschema<4.0,>=3.0a3->poetry)
  Using cached https://files.pythonhosted.org/packages/d1/6a/4b2fcefd2ea0868810e92d519dacac1ddc64a2e53ba9e3422c3b62b378a6/setuptools-40.8.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/setuptools-40.8.0-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from jsonschema<4.0,>=3.0a3->poetry)
  Using cached https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
  Saved /tmp/pip-generator-python3-poetry-gx0pjjs3/attrs-19.1.0-py2.py3-none-any.whl
Successfully downloaded poetry cachecontrol cachy requests-toolbelt shellingham pyparsing html5lib cleo pkginfo requests jsonschema pyrsistent tomlkit msgpack lockfile webencodings six pastel pylev certifi idna urllib3 chardet setuptools attrs
Generating hash for /tmp/pip-generator-python3-poetry-gx0pjjs3/idna-2.8-py2.py3-none-any.whl
Extracting download url for idna-2.8-py2.py3-none
Traceback (most recent call last):
  File "flatpak/pip-generator.py", line 103, in <module>
    url = get_pypi_url(name, filename)
  File "flatpak/pip-generator.py", line 32, in get_pypi_url
    with urllib.request.urlopen(url) as response:
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/dan/.pyenv/versions/3.7.1/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

One option might be to generate the json file from the poetry.lock file, I understand that is more like how the yarn builder-tool works.

flatpak-pip-generator not working for sqlalchemy

If I generate the json file for SQLAlchemy with ./flatpak-pip-generator SQLAlchemy I get the following file:

{
    "name": "python3-SQLAlchemy",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} SQLAlchemy"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/8c/30/4134e726dd5ed13728ff814fa91fc01c447ad8700504653fe99d91fdd34b/SQLAlchemy-1.3.15.tar.gz",
            "sha256": "c4cca4aed606297afbe90d4306b49ad3a4cd36feb3f87e4bfd655c57fd9ef445"
        }
    ]
}

However if I include this file in my manifest, I get the following error when building:

Downloading sources
Fetching full git repo file:///home/felix/workspace/projects/pyexpenses
Starting build of de.felixweilbach.pyexpenses
Cache miss, checking out last cache hit
========================================================================
Building module python3-SQLAlchemy in /home/felix/workspace/projects/pyexpenses/.flatpak-builder/build/python3-SQLAlchemy-1
========================================================================
Running: pip3 install --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} SQLAlchemy
Looking in links: file:///run/build/python3-SQLAlchemy
Collecting SQLAlchemy
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-d9x3xl0s/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/python3-SQLAlchemy -- setuptools>=40.8.0 wheel:
  Looking in links: file:///run/build/python3-SQLAlchemy
  Collecting setuptools>=40.8.0
    Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: )
  No matching distribution found for setuptools>=40.8.0
  
  ----------------------------------------
Command "/usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-d9x3xl0s/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/python3-SQLAlchemy -- setuptools>=40.8.0 wheel" failed with error code 1 in None
Error: module python3-SQLAlchemy: Child process exited with code 1

I use org.gnome.Platform with version 3.32 as runtime and if I inspect the runtime with flatpak run org.gnome.platform//3.32, I can see that setuptools 40.8.0 are already installed.

cargo: crash with checksum KeyError

Tried to run the generator with Cargo.lock from https://github.com/gdesmott/karapulse

$ ./flatpak-cargo-generator.py ~/dev/rust/karapulse/Cargo.lock
Traceback (most recent call last):
  File "./flatpak-cargo-generator.py", line 81, in <module>
    main()
  File "./flatpak-cargo-generator.py", line 76, in main
    generated_sources = generate_sources(load_cargo_lock(args.cargo_lock))
  File "./flatpak-cargo-generator.py", line 38, in generate_sources
    checksum = metadata[f'checksum {name} {version} ({source})']
KeyError: 'checksum gst-plugin-cdg 0.5.0 (git+https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs?branch=0.5#43aeb840e59b6b019789f80714459f9bff5f954d)'

flatpak-pip-generator fail on jupyter

$> python3 flatpak-pip-generator jupyter
Collecting jupyter
  Using cached https://files.pythonhosted.org/packages/c9/a9/371d0b8fe37dd231cf4b2cff0a9f0f25e98f3a73c3771742444be27f2944/jupyter-1.0.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/jupyter-1.0.0.tar.gz
Collecting notebook (from jupyter)
  Using cached https://files.pythonhosted.org/packages/cb/f9/e7b0df5cc36286ccafb7a70e23a9dcf9a1c2ee961dcd18509db3757639bf/notebook-5.6.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/notebook-5.6.0.tar.gz
Collecting qtconsole (from jupyter)
  Using cached https://files.pythonhosted.org/packages/e4/6d/5af06751ece05ff0892dd02ce043062bc2d8f426f91894876f669a70ef1b/qtconsole-4.4.1.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/qtconsole-4.4.1.tar.gz
Collecting jupyter-console (from jupyter)
  Using cached https://files.pythonhosted.org/packages/cd/bb/fa8901715c009a00ea4c42897da8585189d4781c64f4617a8bd65ca01a3a/jupyter_console-5.2.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/jupyter_console-5.2.0.tar.gz
Collecting nbconvert (from jupyter)
  Using cached https://files.pythonhosted.org/packages/88/68/0417164f344ad702245424d8f998b5c99f1c6147ad1ad645a465ae932134/nbconvert-5.4.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/nbconvert-5.4.0.tar.gz
Collecting ipykernel (from jupyter)
  Using cached https://files.pythonhosted.org/packages/83/2d/1e9860d470725ef22a2c06c15d082ca06d425e91eba1a1292bb0165af128/ipykernel-4.9.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/ipykernel-4.9.0.tar.gz
Collecting ipywidgets (from jupyter)
  Using cached https://files.pythonhosted.org/packages/88/8d/c5d3ac1bdcd74c7327c733c9d02354ff12d7d72744f262ed1e26560962b2/ipywidgets-7.4.2.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/ipywidgets-7.4.2.tar.gz
Collecting jinja2 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/56/e6/332789f295cf22308386cf5bbd1f4e00ed11484299c5d7383378cf48ba47/Jinja2-2.10.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/Jinja2-2.10.tar.gz
Collecting tornado>=4 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/e6/78/6e7b5af12c12bdf38ca9bfe863fcaf53dc10430a312d0324e76c1e5ca426/tornado-5.1.1.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/tornado-5.1.1.tar.gz
Collecting pyzmq>=17 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/b9/6a/bc9277b78f5c3236e36b8c16f4d2701a7fd4fa2eb697159d3e0a3a991573/pyzmq-17.1.2.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/pyzmq-17.1.2.tar.gz
Collecting ipython_genutils (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/e8/69/fbeffffc05236398ebfcfb512b6d2511c622871dca1746361006da310399/ipython_genutils-0.2.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/ipython_genutils-0.2.0.tar.gz
Collecting traitlets>=4.2.1 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/a5/98/7f5ef2fe9e9e071813aaf9cb91d1a732e0a68b6c44a32b38cb8e14c3f069/traitlets-4.3.2.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/traitlets-4.3.2.tar.gz
Collecting jupyter_core>=4.4.0 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/b6/2d/2804f4de3a95583f65e5dcb4d7c8c7183124882323758996e867f47e72af/jupyter_core-4.4.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/jupyter_core-4.4.0.tar.gz
Collecting jupyter_client>=5.2.0 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/4c/df/1e8df7f4de63cc667a7a9aa234539c0419513bf94ac57d36d73b3b434786/jupyter_client-5.2.3.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/jupyter_client-5.2.3.tar.gz
Collecting nbformat (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/6e/0e/160754f7ae3e984863f585a3743b0ed1702043a81245907c8fae2d537155/nbformat-4.4.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/nbformat-4.4.0.tar.gz
Collecting Send2Trash (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/13/2e/ea40de0304bb1dc4eb309de90aeec39871b9b7c4bd30f1a3cdcb3496f5c0/Send2Trash-1.5.0.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/Send2Trash-1.5.0.tar.gz
Collecting terminado>=0.8.1 (from notebook->jupyter)
  Using cached https://files.pythonhosted.org/packages/67/84/ce0ebd0f60e1cbe040f8e065eef7063855d59d9cf5e6438b3f8439fc7e15/terminado-0.8.1.tar.gz
  Saved /tmp/pip-generator-python3-jupyter-oe44xfx0/terminado-0.8.1.tar.gz
  Missing build requirements in pyproject.toml for terminado>=0.8.1 from https://files.pythonhosted.org/packages/67/84/ce0ebd0f60e1cbe040f8e065eef7063855d59d9cf5e6438b3f8439fc7e15/terminado-0.8.1.tar.gz#sha256=55abf9ade563b8f9be1f34e4233c7b7bde726059947a593322e8a553cc4c067a (from notebook->jupyter).
  This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
  Installing build dependencies ... error
  Complete output from command /usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-o62aba1e --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit:
  Collecting flit
    Using cached https://files.pythonhosted.org/packages/72/9d/b8f00445bb458a15c670308cf30e08900dc749690f299b7539ab70273ac2/flit-1.1.tar.gz
    Missing build requirements in pyproject.toml for flit from https://files.pythonhosted.org/packages/72/9d/b8f00445bb458a15c670308cf30e08900dc749690f299b7539ab70273ac2/flit-1.1.tar.gz#sha256=da823d4acae9bda42dcc0c7ab1d9be475a8a47aae5fd6dde63841d9f430ccb2f.
    This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
      Complete output from command python setup.py egg_info:
      error in flit setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
  
      ----------------------------------------
  Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-lrou1uq5/flit/
  
  ----------------------------------------
Command "/usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-o62aba1e --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit" failed with error code 1 in None
Traceback (most recent call last):
  File "flatpak-pip-generator", line 74, in <module>
    ] + opts.packages, check=True)
  File "/usr/lib/python3.7/subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip3', 'download', '--dest', '/tmp/pip-generator-python3-jupyter-oe44xfx0', '--no-binary', ':all:', 'jupyter']' returned non-zero exit status 1.

Using

pip 18.0 from /usr/lib/python3.7/site-packages/pip (python 3.7)

Rectify/Update Electron to Flatpak arches in Yarn generator

[yarn] TypeError: unhashable type: 'dict'

I tried to package drawio-desktop from source using the build tool for yarn. I have not been successful as reported here.

The main issue is the following error thrown by the tool at line 114:

TypeError: unhashable type: 'dict'

Commenting the code allows the tool to complete, but the corresponding piece of logic is lost (removal of potential duplicate dependencies as far as I can tell).

[pip] Error on PyPi packages stored in zip file

flatpik-pip-generator throws an error when pulling a dependency that is stored in a zip file.

Traceback (most recent call last):
  File "./flatpak-builder-tools/pip/flatpak-pip-generator", line 139, in <module>
    name = get_package_name(filename)
  File "./flatpak-builder-tools/pip/flatpak-pip-generator", line 58, in get_package_name
    'Downloaded filename: {} does not end with gz or whl'.format(filename)
Exception: Downloaded filename: setuptools-41.2.0.zip does not end with gz or whl

Current solution is to run a

sed -i "s/filename.endswith('gz')/filename.endswith('gz') or filename.endswith('zip')/g" flatpak-builder-tools/pip/flatpak-pip-generator

on the script before running it. In this case, the setuptools dependency is never even built as it is provided by the org.gnome.Platform environment.

Thanks

npm packages to download build tarballs

There is a issue that concerns me the most. It seems there are npm packages that are just downloaders. What I mean by that is these packages are only meant to run during build time to download a release of another package. I came across this over the weekend. To give an example there is this chromedriver. It seems to be downloading the tarball based on some configuration at build time. I have yet to figure out how to pre-download this. But that is not the real issue here.
If I were to list the issues, I would say there are two

What should we do in case the package doesn't support pre-downloading of the release?
And how many such packages should we try to keep a track of? There could be tons out there.

[go] generator assumes github structure (domain, user, repo) for URLs, breaks for gopkg.in

syncthing seems to depend on libraries not hosted on github.

Trying to follow the README, I have produced this manifest:

---
app-id: me.kozec.syncthingtk

runtime: org.gnome.Platform
runtime-version: '3.34'
sdk: org.gnome.Sdk
sdk-extensions:
  - org.freedesktop.Sdk.Extension.golang

command: syncthing-gtk
rename-desktop-file: syncthing-gtk.desktop
rename-icon: syncthing-gtk

finish-args:
  # Wayland
  - --socket=wayland
  # Fallback X11 + XShm access
  - --socket=fallback-x11
  - --share=ipc
  # Folders for syncing can be anywhere
  - --filesystem=host
  # Network access for sync
  - --share=network
  # System tray icon
  - --talk-name=org.kde.StatusNotifierWatcher

# Cleanup after the python modules
cleanup:
  - /include
  - /lib/pkgconfig

modules:
  - name: syncthing
    buildsystem: simple
    build-options:
        env:
            GOBIN: /app/bin
        build-args:
            - --share=network

    build-commands:
      - |
        . /usr/lib/sdk/golang/enable.sh
        export GOPATH=$PWD/go
        go get github.com/syncthing/syncthing/cmd/syncthing

after running, i.e. flatpak-builder --user --install --ccache --keep-build-dirs --force-clean -v --repo=/var/tmp/fb.repo fpbuilder me.kozec.syncthingtk2.yaml, I execute the generator:

$ ~/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py .flatpak-builder/build/syncthing/go/
Traceback (most recent call last):
  File "/home/muelli/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 84, in <module>
    main()
  File "/home/muelli/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 74, in main
    source_list = sources(args.build_dir)
  File "/home/muelli/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 60, in sources
    return list(map(repo_source, repo_paths(build_dir)))
  File "/home/muelli/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 56, in repo_source
    source_object = {'type': 'git', 'url': remote_url(repo_path), 'commit': current_commit(repo_path), 'dest': dest_path}
  File "/home/muelli/vcs/flatpak-builder-tools/go-get/flatpak-go-get-generator.py", line 51, in remote_url
    cwd=repo_path).decode('ascii').strip()
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: PosixPath('.flatpak-builder/build/syncthing/go/src/gopkg.in/ldap.v2/.gitignore')

The code seems to assume that there is a certain number of levels, i.e.

flatpak-builder-tools/go-get/flatpak-go-get-generator.py

Lines 34 to 38 in 1198988

 for domain in domains: 

 domain_users = domain.iterdir() 

 for user in domain_users: 

 user_repos = user.iterdir() 

 repo_paths += list(user_repos)

Maven/Gradle support

This could be used to package Bisq and jadx.

[pip] add yaml support

It is also possible to write flatpak manifests in yaml language.

It can be useful to add an option to flatpak-pip-generator such as --yaml in order to produce manifests in this format.

No licence for Yarn code

We need to licence the Yarn code @adityashah1212 what licence do you want?

[node] node-headers from electron

Some node native deps require rebuilding them via node-gyp, which requires node-headers. Electron provides their own, e.g. node-v4.2.12-headers.tar.gz for electron 4.2.12. Without it, native deps still could be rebuilt (with node headers installed in SDK extension), but may result in broken binaries, giving errors like this at run time:

 Error: The module '/app/main/resources/app/node_modules/native-keymap/build/Release/keymapping.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 64. This version of Node.js requires
NODE_MODULE_VERSION 69. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).

To my understanding, this happens with electron ≥ 4.0.3 with node 10, but other broken combinations possible.
This can be solved by adding node-headers source matching electron version manually and pointing node-gyp to it via npm_config_nodedir env var.
@refi64 Can it be automated in flatpak-node-generator.py with e.g. option like --add-electron-node-headers?

pip-generator fails for pyqt5

When running against the mu-editor package

Collecting pyqt5==5.10.1 (from mu-editor)
  Could not find a version that satisfies the requirement pyqt5==5.10.1 (from mu-editor) (from versions: )
No matching distribution found for pyqt5==5.10.1 (from mu-editor)

Or against pyqt5 directly

Collecting pyqt5
  Could not find a version that satisfies the requirement pyqt5 (from versions: )
No matching distribution found for pyqt5
Traceback (most recent call last):
  File "./flatpak-builder-tools/pip/flatpak-pip-generator", line 64, in <module>
    ] + opts.packages, check=True)
  File "/usr/lib64/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip3', 'download', '--dest', '/tmp/pip-generator-python3-pyqt5-kjvf0c4u', '--no-binary', ':all:', 'pyqt5']' returned non-zero exit status 1.

[node] KeyError: 'from' in flatpak-node-generator.py", line 461, in process_dependencies

I am trying to run the the node generator on this lockfile, but it fails:

$ flatpak-node-generator.py --recursive npm package-lock.json 
WARNING: aiohttp is not found, performance will suffer.
  (Pass --no-aiohttp to silence this warning.)
Found 6 lockfiles.
Reading packages from lockfiles...
Traceback (most recent call last):
  File "/home/muelli/vcs/flatpak-builder-tools/node/flatpak-node-generator.py", line 939, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/home/muelli/vcs/flatpak-builder-tools/node/flatpak-node-generator.py", line 908, in main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "/home/muelli/vcs/flatpak-builder-tools/node/flatpak-node-generator.py", line 476, in process_lockfile
    yield from self.process_dependencies(lockfile, data['dependencies'])
  File "/home/muelli/vcs/flatpak-builder-tools/node/flatpak-node-generator.py", line 468, in process_dependencies
    yield from self.process_dependencies(lockfile, info['dependencies'])
  File "/home/muelli/vcs/flatpak-builder-tools/node/flatpak-node-generator.py", line 461, in process_dependencies
    git_source = self.parse_git_source(version, info['from'])
KeyError: 'from'
$

I am generously CCing @refi64 as the only author of that file.

I've found out that it is complaining about the abbrev package. It looks a bit like RocketChat/Rocket.Chat#15408 is describing a very similar problem.
I've monkey patched my way around, but now detect-libc is entirely missing from the package-lock.json. Weird.

pip-generator fails for ipykernel

After labouring for a while it fails with the following output. This is a blocker regards releasing my app as a flatpak, so it would be nice if it could be fixed.

    Using cached https://files.pythonhosted.org/packages/3c/83/7ecbe9b762829f589fa9734026e0ccb63cb128fe5615ae1698f65df72bfe/pyzmq-18.1.1.tar.gz
  Collecting tornado>=4.1 (from jupyter_client)
    Using cached https://files.pythonhosted.org/packages/30/78/2d2823598496127b21423baffaa186b668f73cd91887fcef78b6eade136b/tornado-6.0.3.tar.gz
  Collecting parso>=0.5.2 (from jedi>=0.10->ipython>=5)
    Using cached https://files.pythonhosted.org/packages/e0/a2/3786c568fc8e9f64b9f7143e4c3904e61a8c5cb17260f22a6a3019d80c02/parso-0.5.2.tar.gz
  Collecting ptyprocess>=0.5 (from pexpect->ipython>=5)
    Using cached https://files.pythonhosted.org/packages/7d/2d/e4b8733cf79b7309d84c9081a4ab558c89d8c89da5961bf4ddb050ca1ce0/ptyprocess-0.6.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pu0sb232 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit:
    Collecting flit
      Using cached https://files.pythonhosted.org/packages/35/87/a92625dc5c81cf1608553b27b5c216bdc1e678ea6738a3a8a69ee2e5fb51/flit-2.2.0.tar.gz
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pqaygza0 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- "flit_core >=2.2.0,<3":
      Collecting flit_core<3,>=2.2.0
        Using cached https://files.pythonhosted.org/packages/77/72/5dda5dc417a4e702e0d7e4a77e9802792a0e4a2daec2aeed915ead7db477/flit_core-2.2.0.tar.gz
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
          Complete output from command python setup.py egg_info:
          Traceback (most recent call last):
            File "<string>", line 1, in <module>
            File "/usr/lib/python3.7/tokenize.py", line 447, in open
              buffer = _builtin_open(filename, 'rb')
          FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-ommvg2hw/flit-core/setup.py'
  
          ----------------------------------------
      Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-ommvg2hw/flit-core/
  
      ----------------------------------------
    Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pqaygza0 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- "flit_core >=2.2.0,<3"" failed with error code 1 in None
  
    ----------------------------------------
  Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pu0sb232 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit" failed with error code 1 in None
  
  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9mci66_9 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel ipython>=5 jupyter_core>=4.2 jupyter_client" failed with error code 1 in None
Failed to download ipykernel
Please fix the module manually in the generated file

[node] yarn config - Error: Cannot find module '/app/bin/../lib/cli'

Hey! :)

I am currently working on a PR to Flathub -> flathub/flathub#1406
Here is the app I am trying to build https://github.com/uw-labs/bloomrpc

I got a review that I should use the flatpak-builder-tools for building from source with yarn and node. I have written almost copy paste from the example node/webpack-quick-start.

I understand that I am missing /app/lib/cli, but I don't understand why.
Any help would be greatly appreciated. Thanks :)

Manifest

app-id: com.github.uwlabs.BloomRPC
runtime: org.freedesktop.Platform
runtime-version: '19.08'
sdk: org.freedesktop.Sdk
base: org.electronjs.Electron2.BaseApp
base-version: '19.08' 
sdk-extensions:
  - org.freedesktop.Sdk.Extension.node12
command: run
separate-locales: false
finish-args:
  - --share=ipc
  - --socket=x11
  - --share=network
  - --filesystem=home  
modules:
  - name: yarn
    sources:
      - type: archive
        url: https://github.com/yarnpkg/yarn/archive/v1.22.4.tar.gz
        sha256: f9e09bf375388f489a62a08d6ab0a455ab47d62b32512b35e4315d12d12db4e5
    buildsystem: simple
    build-commands:
      - 'cp -a * /app'
    cleanup:
      - '*'
  
  - name: bloomrpc
    sources:
      - type: git
        url: http://github.com/uw-labs/bloomrpc
        commit: afe9e60be6248be62c511d71d93fb25518d2e374
      - generated-sources.json
      - type: script
        dest-filename: run.sh
        commands:
          - export TMPDIR="$XDG_RUNTIME_DIR/app/$FLATPAK_ID"
          - /app/BloomRPC/bloomrpc
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node12/bin:/app/yarn/bin:/run/build/bloomrpc/flatpak-node/chromedriver'
      env:
        ELECTRON_CACHE: /run/build/bloomrpc/flatpak-node/electron-cache
        CHROMEDRIVER_SKIP_DOWNLOAD: 'true'
        npm_config_nodedir: /usr/lib/sdk/node12
    build-commands:
      - HOME=$PWD yarn config --offline set yarn-offline-mirror $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn-mirror
      - yarn --offline
      - sh flatpak-node/electron-builder-arch-args.sh
      - yarn run --offline package -- $ELECTRON_BUILDER_ARCH_ARGS
      # Max icon size for Flatpaks are 512x512. Having bigger icons results in build error
      - rm -rf build/files/share/icons/hicolor/1024x1024 
      - cp -r build/files/* $FLATPAK_DEST
      - install -Dm 755 run.sh $FLATPAK_DEST/bin/run

Output

$ flatpak-builder --repo=.localrepo --force-clean build com.github.uwlabs.BloomRPC.yaml
Emptying app dir 'build'
Downloading sources
warning: redirecting to https://github.com/uw-labs/bloomrpc/
Fetching git repo http://github.com/uw-labs/bloomrpc, ref refs/tags/1.4.1
warning: redirecting to https://github.com/uw-labs/bloomrpc/
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
Starting build of com.github.uwlabs.BloomRPC
Cache miss, checking out last cache hit
========================================================================
Building module yarn in /home/jonas/git/github/Arxcis/flathub/.flatpak-builder/build/yarn-1
========================================================================
Running: cp -a * /app
fusermount: failed to unmount /home/jonas/git/github/Arxcis/flathub/.flatpak-builder/rofiles/rofiles-eGmV6n: Device or resource busy
Committing stage build-yarn to cache
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
========================================================================
Building module bloomrpc in /home/jonas/git/github/Arxcis/flathub/.flatpak-builder/build/bloomrpc-4
========================================================================
Note: checking out 'afe9e60be6248be62c511d71d93fb25518d2e374'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at afe9e60 v1.4.1 (#168)
Running: HOME=$PWD yarn config --offline set yarn-offline-mirror $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn-mirror
internal/modules/cjs/loader.js:984
  throw err;
  ^

Error: Cannot find module '/app/bin/../lib/cli'
Require stack:
- /app/bin/yarn.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:981:15)
    at Function.Module._load (internal/modules/cjs/loader.js:863:27)
    at Module.require (internal/modules/cjs/loader.js:1043:19)
    at require (internal/modules/cjs/helpers.js:77:18)
    at Object.<anonymous> (/app/bin/yarn.js:24:13)
    at Module._compile (internal/modules/cjs/loader.js:1157:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1177:10)
    at Module.load (internal/modules/cjs/loader.js:1001:32)
    at Function.Module._load (internal/modules/cjs/loader.js:900:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/bin/yarn.js' ]
}
Error: module bloomrpc: Child process exited with code 1
make: *** [Makefile:2: build] Error 1

pip-generator: Issues with setup-tools

When generating from

pyaes>=0.1a1
ecdsa>=0.13
qrcode
protobuf
dnspython
jsonrpclib-pelix
qdarkstyle<2.7
aiorpcx>=0.18,<0.19
aiohttp>=3.3.0
aiohttp_socks
certifi

I get the following error:

Traceback (most recent call last):
  File "/var/home/msandoval/.local/bin/flatpak-pip-generator", line 139, in <module>
    name = get_package_name(filename)
  File "/var/home/msandoval/.local/bin/flatpak-pip-generator", line 58, in get_package_name
    'Downloaded filename: {} does not end with gz or whl'.format(filename)
Exception: Downloaded filename: setuptools-41.2.0.zip does not end with gz or whl

	for domain in domains:
	domain_users = domain.iterdir()
	for user in domain_users:
	user_repos = user.iterdir()
	repo_paths += list(user_repos)