How to output the compiled exe file and the library file libso to different directories? about nuitka HOT 20 CLOSED

@iskylite I'm working on a rust written wrapper, maybe that's what you want, you can check it over
here

from nuitka.

It sounds like you want —onefile which will bundle everything into a single exe including any needed dependencies, you can then use that binary anywhere you want.

from nuitka.

@KevinRodriguez777, no, not --onefile. onefile will extrace libraries into /tmp/onefile-xxxxx directory. i just want use binary file with some libraries in others dir.

from nuitka.

That's not supported. The DLL layout is as dirty as it is. you can create a wrapper directory on the outside and make it launch the dist folder if you want that yourself. At this time, Nuitka doesn't create those for you and requires DLLs to live alongside the binary.

from nuitka.

Actually yes, good point @shenjackyuanjie but of course, we will want to not need rust or even a binary for that on Linux, that would be bad, symlinks probably can do the trick there.

from nuitka.

I must admit I would have stumbled across this same feature request in the near future, too. pyinstaller supports --contents-directory (https://pyinstaller.org/en/v6.3.0/usage.html#cmdoption-contents-directory), which lets you separate the my-app.exe from the rest of the files (in .my-app, for instance).

I like that because it allows users to extract the built app into their C:\Tools folder, without having to use wrappers or adapt system paths (assuming C:\Tools is already in there), while allowing easy removal (just delete two items).

from nuitka.

yeah，thx everybody。i will close the issue.

from nuitka.

@shenjackyuanjie 你这个https://github.com/shenjackyuanjie/wrapper-rs 我看代码好像是win平台的，我需要用Linu的，不过谢谢了，目前只能暂时用软连接折腾下了

from nuitka.

你这个https://github.com/shenjackyuanjie/wrapper-rs 我看代码好像是win平台的，我需要用Linu的，不过谢谢了，目前只能暂时用软连接折腾下了

Linux下我感觉你要不然直接写个bash脚本，整个类似我这玩意的操作就行
估计连软连接都不用

from nuitka.

有道理，我试试看

from nuitka.

@kayhayen I know you don't usually read closed issues, so please excuse my pinging you here. I feel there is a number of reasons why it would be great if nuitka supported putting all supporting files in a separate folder next to the binary.

1. The Python code of my great_app project resides in a folder named great_app, from where I include a few files in the standalone app. I would like the binary to be called great_app, too. Boom. FileExistsError: [Errno 17] File exists: 'nuitka.dist/great_app'. Yes, I could add a .bin suffix to the binary, but this seems odd on Linux. No other binary I use has that. With a directory, I would have a great_app binary and .great_app/*support files, including .great_app/greap_app with files from my Python code directory.
2. "create a wrapper directory on the outside and make it launch the dist folder if you want that yourself" it not easy on Windows. That would either be another thing to compile, or a .bat file that opens a visible terminal window. Not great, especially with a terminal-less GUI application.
3. "create a wrapper directory on the outside and make it launch the dist folder if you want that yourself" it not easy on Linux, either. In fact, compiling on GitLab CI means that artifacts get zipped, and symlinks get lost while zipping the artifacts, see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4241. Yes, I could zip myself, but GitLab zips again, see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1955.
4. If you want to zip all the files (binary + dozens of DLLs and folders) without including top-level folder information, in GitLab, you need to move these files to the working directory and zip them from there, see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1057. That is usually not a problem, but knowing the names of all these files next to the binary in advance is hard.

In summary, I would find it very beneficial if I could instruct nuitka do create a great_app binary and everything else in a .great_app folder. That would solve all problems above. The binary has the name I want without any chance of collisions; I do not need symlink or batch-file wrappers; and I can simply move great_app and .great_app to the root and zip them in GitLab. Hope that makes sense to you, any maybe you consider it feasible at one point.

from nuitka.

@bersbersbers about No.2 I think you can try my rust project, which is a wrapper for windows( it only support windows for now, because I'm not familiar with Linux desktop) https://github.com/shenjackyuanjie/wrapper-rs. It will can meet your requirement

from nuitka.

@shenjackyuanjie thanks, but that is not the solution I am looking for. I do not use rust, I do not want to keep an extra config file around, do not want to duplicate efforts including metadata and icons into the wrapper, etc. etc.

from nuitka.

Questions are closed when I feel they are answered, they are still up for discussion @bersbersbers of course, unlike issues that have had solutions, and symptoms reappearing (maybe).

The thing is, making the real result work like that, is not going to reliably work, not even for Linux, and it's a lot of effort too.

The next best is an approach similar to what @shenjackyuanjie is trying to do, namely a bootstrap binary on the outside, that delegates the work. In my book, onefile can do that. You can compile onefile with a relative temporary path, and use the cached mode, where it's not deleting that afterward. With the onefile caching of the SFX files in the best way as described on the roadmap, trying to use NTFS extended attributes to recognise changes quickly, might be really fast to launch. But that is currently subject to file size limitations of 2-3GB on Windows, which makes it non-general anyway.

In a sense you then have something like that then. On Windows, that is the one way to launch a program that doesn't need the Python DLL to be loaded from the DLL search path, and most of the reason why onefile is like it is.

I cherish @shenjackyuanjie for creating a solution, however it being in rust means I cannot merge it to Nuitka. The solution would have to be portable C code for integration in Nuitka and take into account all of Nuitka's features. Needs to pass command line arguments, needs to somehow do something to sys.argv[0] and should e.g. handle console/no console properly. While at it, a splash screen would be nice. BTW: We now believe that it might be possible to create programs that do not actually require a console window to open, but use existing ones, but that is a side issue. He did a bunch of interesting work there.

The next best approach is symlinks. Nuitka has supported symlinks for a while now, including on Windows. Deploying those with a zip file or anything but an installer is probably extremely hard on Windows. For Linux and macOS, they are probably fair solutions, although I think on macOS, the .app folders are so common, and considered cleaned up I guess even though they are a mess, so macOS is maybe out of the question.

I think CMD files are kind of symlinks too. Nuitka already creates some for accelerated mode with some success. However, CMD are having a definite idea about terminals, and LNK files are also not portable, but that would be great solutions too.

What I won't do is to "solve this issue" myself. I will focus on making onefile temporary file and cached mode as good as possible, and that will eat up plenty of my time. I will definitely not delay that in order to solve this.

But if somebody were to take onefile to be something that doesn't contain payload, but instead expects files to be already there and unpacked, well, yeah, that might be something. However I feel that it needs a hand that is not mine. I seem to not have the time to continue the onefile improvements I planned to make. So go and find somebody who will do it

from nuitka.

use the cached mode

Wow, that is a neat idea. I understand that is actually onefile and onedir mode combined. I have asked pyinstaller people for this, and while they seem to receive this request regularly, they don't like implementing it, mainly for reasons of stability (regarding incomplete extraction, simultaneous extractions, etc). You sound as nuitka handles those things, do I understand that correctly? I will definitely try it - after all, it looks like it's little more than adding --onefile-tempdir-spec=.great_app to start with.

from nuitka.

I don't know or follow what pyinstaller does to be honest. They have an entirely different problem I believe. Problems and solutions rarely apply and vice versa.

The cache performance of onefile is not (yet) as good as I would like it, and decompression is still done unless you use --onefile-as-archive which I am bound to remove once I find the time to merge the best of both worlds, but I believe it's the best there is at this point. And then of course to implement the other stuff from the roadmap, to avoid reading the files to see if they changed.

I would recommend to include {VERSION} such that different versions of your program do not conflict with each other, maybe like .great_app/{VERSION} or such even, then I don't see how that could go wrong. If Nuitka doesn't handle something properly, like concurrent first execution, well, then we will need to fix that. So far I never got a report of that though.

from nuitka.

Great. As I said, that's looks like the ideal combination, and I'll try it out.

Regarding --onefile-as-archive, I don't understand what it does exactly. The help says

use an archive format that can be unpacked with "nuitka-onefile-unpack" rather than a stream that only the onefile program itself unpacks

What it nuitka-onefile-unpack? I looked at the code, and

looks like both are doing very different things (but maybe there's more differences under the hood).

Anyway, do I understand correctly that for maximum startup performance, I would use --onefile --onefile-tempdir-spec .great_app/{VERSION} --onefile-as-archive --onefile-no-compression

from nuitka.

The --onefile-as-archive was intended to be the future of the format, but I had another idea in the mean time, namely that there is simple a header part in the compressed data, and that then the file data follows unskippable, requiring a decompression still, but in the case of a second start, it will be detected as unnecessary.

I am not sure the combination you gave with no compression even works, but it's surely not needed, since skipping file data is quick with no compression already.

The decompressor is something I want to add, such that AVs can stop discriminating against onefile, as it is possible to look into it, without running it. Not sure if that hope works out. Since I am going to implement that in Python, I want to have the final format for that though. The new format is going to combine the best result for caching with best compression. The archive format compresses per file, and then allows skipping without decompression, but I now feel I realized, that's not at all necessary. Ultimately it just skips all the files the second time around.

Disabling compression should be wrong for that approach and not make a performance difference except the first time around.

from nuitka.

I am not sure the combination you gave with no compression even works

fyi: it does.

but it's surely not needed, since skipping file data is quick with no compression already

That is true. And speeding up first unpacking by <10% (~1s) is not worth the 5x file size for me :)

from nuitka.

I am not even sure decompression+reading is always slower than reading more data from disk. The used compression is supposed to have very little overhead.

from nuitka.