theassassin / type3-runtime Goto Github PK
View Code? Open in Web Editor NEWAppImage type 3 runtime draft
License: zlib License
AppImage type 3 runtime draft
License: zlib License
From design goals:
I don't quite understand what advantages this brings. From my POV this only seems to introduce unnecessary maintenance overhead. I might be missing something here but:
Hi @TheAssassin. In order to get a serious discussion started around the ideas and concepts in this repository, we should clarify the use of some terms first, so that everyone (including the two of us) understands what is meant. Reading in this repository, I think some terms are mixed and matched in ways that make them less clear to understand.
So here is my proposal:
One design goal of this new type is to provide reliable and verifiable versioning. There's several suggestions how to handle this. Basically, we have to decide how to put version numbers on the actual components of an AppImage:
We can version them each separately, requiring 3 different versioning approaches, or we can use combined version numbers for some or all of them.
Here's a short summary of what I think was suggested so far in #1, where this has been discussed before by @5pacetoast and me primarily.
We've concluded that the runtime ELF header is independent from the actual AppImage format. Any binary which can read the AppImage header should be used interchangeably (i.e., I can use either runtime, provided that runtime supports at least the version of the AppImage header).
The question is if we have to separate AppImage type and AppImage header revision. I personally think we do not have to overly complicate this. Whenever we make really breaking changes, we can bump the AppImage type number. This is IMO rather unlike, though. Therefore I propose to strongly bind the AppImage header to the AppImage type.
There might be situations in which we have to e.g., replace attributes due to security issues, where you can either remove the old attributes and introduce new ones. This breaks the backwards compatibility, however, so my approach would always be to just abandon the attributes (but keep them in the header), and append the new ones. This way, older software can still use the new AppImages. Since the old, broken fields are empty, the features based on them should just not be used by the old software.
If we decided to go for the "breaking" approach, we could have situations in which AppImages of the same type cannot be used by software claiming to support a certain type. That's an inacceptable UX to me. It's a "sub-typing" of AppImage, where we'd have a 3.1.x.y
kind of format and a 3.2.x.y
(the former uses header format 1.x.y
, the latter header format 2.x.y
) instead of one type 3 that will be easy to support forever. And if we really have to make a breaking change, we need to signalize that clearly to developers and users of applications which deal with AppImages.
@5paceToast do you have any idea how to maintain the "type 3 means type 3" and avoid a "different header major versions create subtypes" UX?
That said, the header revision does not have to be a scalar number. I don't see the major benefits to using e.g., a minor and a patch version as of now, but adding a second scalar number to the version is an easy task.
Sandboxing is a more and more popular technique that shall prevent some attacks from untrusted applications, or applications which operate on data from non-trustworthy sources (read: things downloaded randomly from the Internet).
In AppImage, sandboxing has never been implemented in a good way. There is a third-party solution called Firejail that uses a very generous default profile, and apparently has a long record of vulnerabilities. An official solution has never been developed, mostly because of opposition to distribute tooling that might give people a false sense of safety.
The biggest issue with sandboxing AppImages is the distribution of profiles. Following the principle of least privilege, a profile should allow only the smallest set of actions that is required for an application to function properly. This means that an Internet browser should be granted access to the Internet as well as the users' ~/Downloads
for instance, but there is no reason for them to write to any other directory on the system. A music browser on the other hand may not need access to the Internet but to the many directories in the filesystem (caching metadata, accessing USB disks, ...).
From an IT security point of view, we need some trustworthy instance to approve such profiles. In case of the classic package repositories, this instance is the group of maintainers who sign the packages. In systems like Flatpak, it's the app store that is in control. Of course, you have to trust these to do their job properly, but the big distros have proven in the past to be trustworthy.
In the case of AppImages, we do not have a central authority that can provide profiles for sandboxing. We don't have maintainers in between application developers and users. To put it bluntly, we cannot outsource this task. Thus, we have to have users create and approve profiles for sandboxing. The only two alternatives are adding a store/repository (which we really don't want to) and using a relatively generous profile (a.k.a. "the Firejail approach", which is not guaranteed to grant the right freedoms to applications, and also violates the principle of least privilege).
Of course, we can help users with creating those profiles. We have discussed implementing Android-style dialogs before, e.g., in AppImageLauncher. There are two key elements to such an interface: reducing the amount of options to just a few ones which are also easy to understand (e.g., "grant Internet access", "grant access to home directory", ...), and allowing AppImages to suggest permissions. The latter can be split into "essential" and "nice to have" permissions for an AppImage. The former trigger a warning that the AppImage will likely not work at all, while the latter ones create a hint that some functionality may not be available. These suggested permissions could also be a lot more fine-grained, e.g., instead of letting an application write to the entire home directory, it could suggest a list of directories (e.g., ~/.cache/browser
, ~/Downloads
, ~/.config/browser
).
What clearly does not work is simplifying things too much (like Firejail has done, by default you can either use the way too open default profile or no sandboxing at all, with the former potentially breaking your app already), and patronizing users. I believe educating users is far more successful, because humans are still the weakest link. Some things just can't be solved with technology (otherwise, phone scam would not be a thing any more...).
Note that we will not really be able to enforce sandboxing, it will remain a best-effort. Even system-wide-installed tools like AppImageLauncher have to make use of APIs like binfmt to inject themselves as wrappers for executing AppImages, but these are not meant to be part of any security-related tooling. But sandboxing is just a "last resort" kind of technology anyway, IMO. The rule "if you don't trust it, don't open it" will continue to play a big role in the world of AppImage.
is it technically possible to make an AppImage that is also a valid zip file?
Reason:
aforensics/HiddenVM#10 (comment)
Wouldn't it be ideal if we could have a zip file that would contain an AppImage, but that could also be used as an AppImage unzipped? Best of both worlds!
The new AppImage type should be compatible to FUSE 2 (like the existing type 2) as well as FUSE 3 (resp. libfuse 3). To accomplish that, an abstraction layer should be implemented to hide the complexity behind a simple-to-use interface.
A while ago @probonopd suggested to implement support for something I think we can call "multi-layer AppImages". They work similar to Docker images, using some sort of overlay filesystem which stacks layers on top of each other, and provides a transparent union view to the user and system.
We could implement such a feature in the new runtime. But the question is rather, do we really need that? Please feel free to leave your opinion in this issue. We don't have to discuss how it's done technically and what to be careful with*
.
Please note that using multiple layers does not mean we need to ship them separately. They all need to be contained in the AppImage.
*
For example, upper layers might remove files which have been added in lower layers. Just unioning all files doesn't provide any support for this.
Some applications have multiple modes of operation that share most of their data.
One such example is DCSS.
It has a "tiles" mode (GUI) and a "terminal" mode (TUI), that share most of their libraries and dependencies (except for the GUI-specific ones).
It would be nice if the integration allowed one to launch either variant via their own .desktop file.
Combining this with #4 would suggest (to me) that the optimal way forward is to have an entry in the header - specifying a list of .desktop files that should be integrated (or perhaps keeping the .desktop files in the header, but that can be discussed separately).
This way the entry can be validly empty, and use-cases like the above become more convenient.
As mentioned in #6 we need a better way to store additional information in an AppImage, outside of its main compressed filesystem image.
Currently we are using ELF sections to store various metadata, but this is not elegant. (While I am writing this I am asking myself: Why is this not elegant? Why do we need something else?)
So we are looking for a way to store additional metadata in AppImages without having to touch the main compressed filesystem image.
In other words, we are looking for a way to store key-values in executables. Here are the existing ones I know of:
UPDI
(for update information), etc. Maybe @waddlesplash or @pulkomandy have an opinion on how suitable Haiku OS file resources would be in the Linux world?What would be the downsides of using one of those, rather than defining something AppImage specific? By pushing something that is not AppImage specific, chances would be much better to get support into e.g., file managers (so that they can show proper icons without the need for desktop files or thumbnailers)
.desktop
files only make sense for interactive (readline-like, tui and gui) applications.
This means that non-interactive applications can't be easily packaged in an appimage, which I think is a shame.
For example, consider the go runtime: even with Terminal=true, it doesn't make much sense.
There's no icon for go(1) or gofmt(1).
There's no point in launching them from a menu.
However, you cannot distribute them as standalone executables, because they fundamentally depend on GOROOT(_FINAL) to function, since that's where the stdlib etc is kept, even though they have no external dependencies besides, well, themselves and GOROOT.
So to package a golang runtime, you have to procure an unofficial application icon from somewhere and make a nonsense .desktop
file just to comply to the specification.
Wouldn't it instead, perhaps, be preferable to make .desktop files optional, and simply not perform any integration work if there isn't one specified?
This is going to sound ridiculous at first, but this is my rationale:
Max portability. Using shell scripting and an internal squashfuse binary (or a custom build with fuseless-extraction as a fallback?) allows not only for use on any Linux system of the same architecture, but any Linux system of any supported architecture. For small AppImages and those that use an interpreter or JIT runtime, supporting multiple architectures in one file may not significantly increase the size of the application. I did a little test building Pypy into SquashFS bundles, combining the x86_64 and AARCH64 installations increased the total size of the runtime by 60% in both ZSTD (30MB to 48MB) and LZ4 (42MB to 68MB). This change is not insignificant, but depending on the size of the project itself, it may change the total size very little.
Rapid development. Obviously once it gets to a mature stage, the language doesn't really matter as long as it works, but during early development, this would allow for much faster bug fixes.
Smaller codebase. I've created a PoC implementation of this that I'm calling shImg (shappimage), and the current runtime (comments included) is under 400 lines. After further optimizations and cleaning up have been done, I really can't see it getting much above 600 LoC.
Easy resource access. It's incredibly easy to embed a desktop files, SVGs and binary resources in a shell script using cat<<EOF
and other tricks. These could also be safely extracted using a couple lines of code for desktop integration software.
The runtime has a speed penalty, my current solution is caching the extracted squashfuse in $XDG_RUNTIME_DIR
. Using faster compression algorithms like LZ4_HC and ZSTD also significantly improve performance of the app, which easily make up for the slower runtime. These obviously have nothing to do with the runtime, but are both huge improvements over ZLIB. A build of another project of mine, aisap, which is just over 2MB uncompressed actually runs faster using shImg than the current standard AppImage with DEFLATE, and this difference very quickly scales with larger applications.
Once I clean up the code a bit, I'll further work on optimizing it, probably by adding a Perl section to do the parsing. Thanks in advance and let me know what you think
Let's first brainstorming about the design goals of a type-3 image format, then spec a future type-3 format, and then start writing an implementation.
Please be aware that the type-2 spec leaves quite much flexibility to implementors, e.g., (iirc) it doesn't even mandate the payload to be in squashfs format (need to double check this one).
Brainstorming about about the design goals of a future ype-3 image format has been started here:
https://github.com/AppImage/AppImageKit/wiki/Brainstorming
Let's not introduce new image formats frequently. Our tools (e.g., go-appimage, AppImageLauncher, etc.) will need to support all of them to be backward compatible... maybe it is possible to achieve some (or even many) of the goals with merely a new type-2 runtime?
Stolen from here
type 3 should specify a way to extract desktop integration data without having to read the payload format as a fallback.
Absolutely. Rationale:
.desktop
files to be able to easily integrate AppImages into the desktop and keep them up to date.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.