Implement simple packager about dust HOT 9 OPEN

Yeah, that would be really nice!

We can basically just pack up pixie-vm + dust + the library, but we also need to specify what program to run after unpacking. (Or we read it from a :main key in project.edn.)

On Linux, we could unpack to ~/.cache/<some-unique-id> and use that if present. However, we'd really have to use some unique key, maybe an MD5 hash of the whole package.

from dust.

Kinda, I was thinking not to bundle dust with it, try to make it slim and use the project.edn to generate a thin startup script with things like the entry point derived from project.edn and proper load path, and have all the dependencies prefetched on the dev machine and bundled in the archive. Then we don't need to bundle dust, worry about curl presence or whatnot, it makes deploys to other OSes even ok'ish in the long run.

For the unpacking yes, something of that sort this could be as simple as ~/.cache/<package-name>-<version> even (it's easier to get to it if you want to clean things up), but an md5 would be fine.

from dust.

Ah yes, baking-in the load path makes sense. To make things start up more quickly we can (and should) byte-compile everything as well.

from dust.

Yes that what I mean with "trigger compilation (pxic)", sorry I am not super clear, just throwing ideas :]
The goal is clojure like uberjar + the runtime + startup script all in one

from dust.

I have written a tarfile reader in pixie so it would be possible to make a tar.gz of a byte compiled library with all its dependencies and read them as though they were extracted.

from dust.

@thomasmulvaney the problem is that pixie would have to be installed first in that case.
I think it could be better to have all of it (runtime, deps, compiled (or not) pxi files) in a single self extracting archive. This way we just have a startup penality the first time it's run and no need for prerequisites on a fresh machine. makeself is a good example to follow and get inspiration from imho.

from dust.

Unfortunately my bash-foo is very weak, I could probably get something working but this will be poorly written. Feel free to take over this task.

from dust.

I've got an example up and running. It's not automated, but it works™.

By dark magic, you'll have to assemble the pixie stdlib, your dependencies and your own code into a directory. It will look something like the following:

$ tree packaged-base
packaged-base
├── deps
│   └── heyLu
│       └── hiccup.pxi
│           ├── LICENSE.html
│           ├── project.edn
│           ├── README.md
│           ├── src
│           │   └── hiccup
│           │       ├── compiler.pxi
│           │       ├── compiler.pxic
│           │       ├── core.pxi
│           │       ├── core.pxic
│           │       ├── def.pxic
│           │       ├── element.pxic
│           │       ├── form.pxic
│           │       ├── middleware.pxi
│           │       ├── middleware.pxic
│           │       ├── page.pxic
│           │       ├── util.pxi
│           │       └── util.pxic
│           └── test
│               └── hiccup
│                   └── test
│                       ├── def.clj
│                       ├── element.clj
│                       ├── form.clj
│                       ├── middleware.clj
│                       ├── page.clj
│                       ├── test-core.pxi
│                       ├── test-core.pxic
│                       └── util.clj
├── pixie
│   ├── async.pxi
│   ├── async.pxic
│   ├── buffers.pxi
│   ├── buffers.pxic
│   ├── channels.pxi
│   ├── channels.pxic
│   ├── csp.pxi
│   ├── csp.pxic
│   ├── ffi-infer.pxi
│   ├── ffi-infer.pxic
│   ├── fs.pxi
│   ├── fs.pxic
│   ├── io
│   │   ├── common.pxi
│   │   ├── common.pxic
│   │   ├── tcp.pxi
│   │   ├── tcp.pxic
│   │   ├── tty.pxi
│   │   ├── tty.pxic
│   │   ├── uv-common.pxi
│   │   └── uv-common.pxic
│   ├── io-blocking.pxi
│   ├── io-blocking.pxic
│   ├── io.pxi
│   ├── io.pxic
│   ├── math.pxi
│   ├── math.pxic
│   ├── parser
│   │   ├── json.pxi
│   │   └── json.pxic
│   ├── parser.pxi
│   ├── parser.pxic
│   ├── PixieChecker.hpp
│   ├── repl.pxi
│   ├── repl.pxic
│   ├── set.pxi
│   ├── set.pxic
│   ├── stacklets.pxi
│   ├── stacklets.pxic
│   ├── stdlib.pxi
│   ├── stdlib.pxic
│   ├── streams
│   │   ├── utf8.pxi
│   │   ├── utf8.pxic
│   │   ├── zlib
│   │   │   ├── ffi.pxi
│   │   │   └── ffi.pxic
│   │   ├── zlib.pxi
│   │   └── zlib.pxic
│   ├── streams.pxi
│   ├── streams.pxic
│   ├── string.pxi
│   ├── string.pxic
│   ├── system.pxi
│   ├── system.pxic
│   ├── test.pxi
│   ├── test.pxic
│   ├── time.pxi
│   ├── time.pxic
│   ├── uv.pxi
│   └── uv.pxic
├── pixie-vm
├── run.pxi
├── run.sh
└── src
    └── packaged
        ├── core.pxi
        └── core.pxic

15 directories, 85 files

In addition, I added a top-level run.pxi, which simply calls a function called main in a namespace:

$ cat packaged-base/run.pxi
(require packaged.core)

(apply packaged.core/main program-arguments)

Now you can use makeself to generate a self-extracting archive, that runs that script:

$ makeself packaged-base packaged.run "An example packaged pixie app" ./pixie-vm `cat .load-path` run.pxi
...
$ ./packaged.run 'hello' 'world?!'
...
<h1>HELLO WORLD?!</h1>

And then you have to automate this... Maybe I'll try that tomorrow, but if anyone wants to take over, feel free to do so!

from dust.

Cool stuff.
We don't need to include the .pxi files in the package right? just the pxic should be enough (given we compiled everything first).
There are other "improvements" we'll have to think about later, like how to handle "resources" (prolly via project.edn).

from dust.