is it technically possible to make an AppImage that is also a valid zip file?

Reason:
aforensics/HiddenVM#10 (comment)

Wouldn't it be ideal if we could have a zip file that would contain an AppImage, but that could also be used as an AppImage unzipped? Best of both worlds!

As mentioned in #6 we need a better way to store additional information in an AppImage, outside of its main compressed filesystem image.

Currently we are using ELF sections to store various metadata, but this is not elegant. (While I am writing this I am asking myself: Why is this not elegant? Why do we need something else?)

So we are looking for a way to store additional metadata in AppImages without having to touch the main compressed filesystem image.

In other words, we are looking for a way to store key-values in executables. Here are the existing ones I know of:

• ELFIcon by @glensc, despite its name, is a full-fledged resource system not limited to icons ("resource files should be used in order to store images in a standardized way, this will also allow applications to store other data"). Spec: https://wiki.ubuntu.com/ELFIconSpec, implementation: https://github.com/pld-linux/elficon
• Haiku OS file resources: https://www.haiku-os.org/docs/api/classBResources.html. I have written about them at https://medium.com/@probonopd/my-sixth-day-with-haiku-under-the-hood-of-resources-icons-and-packages-abec8d0e4ec6. There is even a GUI tool to edit those resources: https://github.com/HaikuArchives/Resourcer. If we would use this system, of course we would need to define our own resources, e.g., UPDI (for update information), etc. Maybe @waddlesplash or @pulkomandy have an opinion on how suitable Haiku OS file resources would be in the Linux world?
• FatELF by @rcgordon is a file format that embeds multiple ELF binaries for different architectures into one file: http://icculus.org/fatelf/ Possibly it could be extended to hold generic resources in addition to ELF binaries?

What would be the downsides of using one of those, rather than defining something AppImage specific? By pushing something that is not AppImage specific, chances would be much better to get support into e.g., file managers (so that they can show proper icons without the need for desktop files or thumbnailers)

Some applications have multiple modes of operation that share most of their data.
One such example is DCSS.
It has a "tiles" mode (GUI) and a "terminal" mode (TUI), that share most of their libraries and dependencies (except for the GUI-specific ones).
It would be nice if the integration allowed one to launch either variant via their own .desktop file.

Combining this with #4 would suggest (to me) that the optimal way forward is to have an entry in the header - specifying a list of .desktop files that should be integrated (or perhaps keeping the .desktop files in the header, but that can be discussed separately).
This way the entry can be validly empty, and use-cases like the above become more convenient.

Hi @TheAssassin. In order to get a serious discussion started around the ideas and concepts in this repository, we should clarify the use of some terms first, so that everyone (including the two of us) understands what is meant. Reading in this repository, I think some terms are mixed and matched in ways that make them less clear to understand.

So here is my proposal:

• Image format. Describes the file format of a spec-compliant AppImage. (Originally I was tempted to call this the "AppImage version", but that term could mean too many things, so I went with "image format".) The Image format can be type 0, type 1, or type 2. _Are you proposing to introduce a type 3 image format? Then let's draft a section for it that would go into https://github.com/AppImage/AppImageSpec/blob/master/draft.md#image-format _. I understand that one of the killer features of a type 3 image format would be that the magic bytes would at another offset. Which offset are you proposing? The image format is part of the AppImageSpec which is under the MIT License
• Runtime. An AppImage runtime is the executable code that is placed at the beginning of an AppImage file. Each runtime usually implements the mounting and running of AppImages that are in a certain image format. So if we define a type 3 image format, then we also need a runtime that can deal with this image format. Nothing forbids a runtime to support more than one image format, by the way. So if the only change between type 2 and type 3 is the location of the magic bytes, then the executable code of the runtime might otherwise well be the same (so that the same runtime code could run both). The current runtime is under the MIT License
• Resources. The term "AppImage header" you are using is confusing me. As you know, at the beginning ("head") of each AppImage file there is an ELF. The ELF has a "header" that is defined in the ELF standard. Are you looking for a better way to store additional information in an AppImage, outside of its main compressed filesystem image? Currently we are using ELF sections to store various metadata, but I'd agree that this approach is less than elegant. What would be needed is a more flexible way to store additional metadata in AppImages without having to touch the main compressed filesystem image. Instead of inventing yet-another AppImage-specific clutch, however, I would hope that we can add/find a way to achieve this with ELF. I would propose to call this "(ELF) resources" rather than "header" by the way. Let's think of resources as key-value pairs that can be added to ELFs (and hence, AppImages). Haiku OS seems to have a way of adding metadata to ELF files. Why can't we use their mechanism, and specify our own metadata keys?

This is going to sound ridiculous at first, but this is my rationale:

• Max portability. Using shell scripting and an internal squashfuse binary (or a custom build with fuseless-extraction as a fallback?) allows not only for use on any Linux system of the same architecture, but any Linux system of any supported architecture. For small AppImages and those that use an interpreter or JIT runtime, supporting multiple architectures in one file may not significantly increase the size of the application. I did a little test building Pypy into SquashFS bundles, combining the x86_64 and AARCH64 installations increased the total size of the runtime by 60% in both ZSTD (30MB to 48MB) and LZ4 (42MB to 68MB). This change is not insignificant, but depending on the size of the project itself, it may change the total size very little.

• Rapid development. Obviously once it gets to a mature stage, the language doesn't really matter as long as it works, but during early development, this would allow for much faster bug fixes.

• Smaller codebase. I've created a PoC implementation of this that I'm calling shImg (shappimage), and the current runtime (comments included) is under 400 lines. After further optimizations and cleaning up have been done, I really can't see it getting much above 600 LoC.

• Easy resource access. It's incredibly easy to embed a desktop files, SVGs and binary resources in a shell script using cat<<EOF and other tricks. These could also be safely extracted using a couple lines of code for desktop integration software.

The runtime has a speed penalty, my current solution is caching the extracted squashfuse in $XDG_RUNTIME_DIR. Using faster compression algorithms like LZ4_HC and ZSTD also significantly improve performance of the app, which easily make up for the slower runtime. These obviously have nothing to do with the runtime, but are both huge improvements over ZLIB. A build of another project of mine, aisap, which is just over 2MB uncompressed actually runs faster using shImg than the current standard AppImage with DEFLATE, and this difference very quickly scales with larger applications.

Once I clean up the code a bit, I'll further work on optimizing it, probably by adding a Perl section to do the parsing. Thanks in advance and let me know what you think

From design goals:

• Support different image formats (squashfs remaining the default for now)

I don't quite understand what advantages this brings. From my POV this only seems to introduce unnecessary maintenance overhead. I might be missing something here but:

• Either the AppImage ships a runtime that can load it's payload and you don't have to care about it
• Or you want to have different tools accessing the payload in which case you'd probably want to mandate exactly one format to keep things simple and compatible.

Sandboxing is a more and more popular technique that shall prevent some attacks from untrusted applications, or applications which operate on data from non-trustworthy sources (read: things downloaded randomly from the Internet).

In AppImage, sandboxing has never been implemented in a good way. There is a third-party solution called Firejail that uses a very generous default profile, and apparently has a long record of vulnerabilities. An official solution has never been developed, mostly because of opposition to distribute tooling that might give people a false sense of safety.

The biggest issue with sandboxing AppImages is the distribution of profiles. Following the principle of least privilege, a profile should allow only the smallest set of actions that is required for an application to function properly. This means that an Internet browser should be granted access to the Internet as well as the users' ~/Downloads for instance, but there is no reason for them to write to any other directory on the system. A music browser on the other hand may not need access to the Internet but to the many directories in the filesystem (caching metadata, accessing USB disks, ...).

From an IT security point of view, we need some trustworthy instance to approve such profiles. In case of the classic package repositories, this instance is the group of maintainers who sign the packages. In systems like Flatpak, it's the app store that is in control. Of course, you have to trust these to do their job properly, but the big distros have proven in the past to be trustworthy.

In the case of AppImages, we do not have a central authority that can provide profiles for sandboxing. We don't have maintainers in between application developers and users. To put it bluntly, we cannot outsource this task. Thus, we have to have users create and approve profiles for sandboxing. The only two alternatives are adding a store/repository (which we really don't want to) and using a relatively generous profile (a.k.a. "the Firejail approach", which is not guaranteed to grant the right freedoms to applications, and also violates the principle of least privilege).

Of course, we can help users with creating those profiles. We have discussed implementing Android-style dialogs before, e.g., in AppImageLauncher. There are two key elements to such an interface: reducing the amount of options to just a few ones which are also easy to understand (e.g., "grant Internet access", "grant access to home directory", ...), and allowing AppImages to suggest permissions. The latter can be split into "essential" and "nice to have" permissions for an AppImage. The former trigger a warning that the AppImage will likely not work at all, while the latter ones create a hint that some functionality may not be available. These suggested permissions could also be a lot more fine-grained, e.g., instead of letting an application write to the entire home directory, it could suggest a list of directories (e.g., ~/.cache/browser, ~/Downloads, ~/.config/browser).

What clearly does not work is simplifying things too much (like Firejail has done, by default you can either use the way too open default profile or no sandboxing at all, with the former potentially breaking your app already), and patronizing users. I believe educating users is far more successful, because humans are still the weakest link. Some things just can't be solved with technology (otherwise, phone scam would not be a thing any more...).

Note that we will not really be able to enforce sandboxing, it will remain a best-effort. Even system-wide-installed tools like AppImageLauncher have to make use of APIs like binfmt to inject themselves as wrappers for executing AppImages, but these are not meant to be part of any security-related tooling. But sandboxing is just a "last resort" kind of technology anyway, IMO. The rule "if you don't trust it, don't open it" will continue to play a big role in the world of AppImage.

Let's first brainstorming about the design goals of a type-3 image format, then spec a future type-3 format, and then start writing an implementation.

Please be aware that the type-2 spec leaves quite much flexibility to implementors, e.g., (iirc) it doesn't even mandate the payload to be in squashfs format (need to double check this one).

Brainstorming about about the design goals of a future ype-3 image format has been started here:
https://github.com/AppImage/AppImageKit/wiki/Brainstorming

Let's not introduce new image formats frequently. Our tools (e.g., go-appimage, AppImageLauncher, etc.) will need to support all of them to be backward compatible... maybe it is possible to achieve some (or even many) of the goals with merely a new type-2 runtime?

The new AppImage type should be compatible to FUSE 2 (like the existing type 2) as well as FUSE 3 (resp. libfuse 3). To accomplish that, an abstraction layer should be implemented to hide the complexity behind a simple-to-use interface.

Stolen from here

type 3 should specify a way to extract desktop integration data without having to read the payload format as a fallback.

Absolutely. Rationale:

• FileManagers may want to generate thumbnails. Requiring a squashfs driver to pull a tiny PNG out of a file seems unreasonable.
• Other software may also have good reasons to access metadata such as update URLs, icons and .desktop files to be able to easily integrate AppImages into the desktop and keep them up to date.
• Should the imaging format from ever change a way from squashfs, it may be easier to do so in a backward compatible way as most external tools will never need to care about the payload content.

One design goal of this new type is to provide reliable and verifiable versioning. There's several suggestions how to handle this. Basically, we have to decide how to put version numbers on the actual components of an AppImage:

• the runtime (ELF executable header)
• the new AppImage header (metadata format); I've called its versioning "revision" so far
• the AppImage type (e.g., 3 for the upcoming one, 2 would be the direct predecessor)

We can version them each separately, requiring 3 different versioning approaches, or we can use combined version numbers for some or all of them.

Here's a short summary of what I think was suggested so far in #1, where this has been discussed before by @5pacetoast and me primarily.

We've concluded that the runtime ELF header is independent from the actual AppImage format. Any binary which can read the AppImage header should be used interchangeably (i.e., I can use either runtime, provided that runtime supports at least the version of the AppImage header).

The question is if we have to separate AppImage type and AppImage header revision. I personally think we do not have to overly complicate this. Whenever we make really breaking changes, we can bump the AppImage type number. This is IMO rather unlike, though. Therefore I propose to strongly bind the AppImage header to the AppImage type.

There might be situations in which we have to e.g., replace attributes due to security issues, where you can either remove the old attributes and introduce new ones. This breaks the backwards compatibility, however, so my approach would always be to just abandon the attributes (but keep them in the header), and append the new ones. This way, older software can still use the new AppImages. Since the old, broken fields are empty, the features based on them should just not be used by the old software.

If we decided to go for the "breaking" approach, we could have situations in which AppImages of the same type cannot be used by software claiming to support a certain type. That's an inacceptable UX to me. It's a "sub-typing" of AppImage, where we'd have a 3.1.x.y kind of format and a 3.2.x.y (the former uses header format 1.x.y, the latter header format 2.x.y) instead of one type 3 that will be easy to support forever. And if we really have to make a breaking change, we need to signalize that clearly to developers and users of applications which deal with AppImages.

@5paceToast do you have any idea how to maintain the "type 3 means type 3" and avoid a "different header major versions create subtypes" UX?

That said, the header revision does not have to be a scalar number. I don't see the major benefits to using e.g., a minor and a patch version as of now, but adding a second scalar number to the version is an easy task.

.desktop files only make sense for interactive (readline-like, tui and gui) applications.
This means that non-interactive applications can't be easily packaged in an appimage, which I think is a shame.
For example, consider the go runtime: even with Terminal=true, it doesn't make much sense.
There's no icon for go(1) or gofmt(1).
There's no point in launching them from a menu.

However, you cannot distribute them as standalone executables, because they fundamentally depend on GOROOT(_FINAL) to function, since that's where the stdlib etc is kept, even though they have no external dependencies besides, well, themselves and GOROOT.

So to package a golang runtime, you have to procure an unofficial application icon from somewhere and make a nonsense .desktop file just to comply to the specification.
Wouldn't it instead, perhaps, be preferable to make .desktop files optional, and simply not perform any integration work if there isn't one specified?

A while ago @probonopd suggested to implement support for something I think we can call "multi-layer AppImages". They work similar to Docker images, using some sort of overlay filesystem which stacks layers on top of each other, and provides a transparent union view to the user and system.

We could implement such a feature in the new runtime. But the question is rather, do we really need that? Please feel free to leave your opinion in this issue. We don't have to discuss how it's done technically and what to be careful with*.

Please note that using multiple layers does not mean we need to ship them separately. They all need to be contained in the AppImage.

* For example, upper layers might remove files which have been added in lower layers. Just unioning all files doesn't provide any support for this.