TBD details

android/ndk#1307

Workaround: you should specify 21.2.* or earlier (as long as it supports android-29) instead.

Until it gets too big to maintain here, we would list up portable existing audio plugins here.

There is some annoyance around CMake/Android/GoogleTest.

• cmake googletest is not available in cmake 3.6. It is possible to specify cmake 3.10 in externalNativeBuild section in build.gradle, but it also means build server (cerbero setup) requires cmake 3.10 setup, which is kind of annoying.
• there is no googletest setup in Ubuntu. There is googletest package, but it only packages sources. Maybe that's how it is used, but it also means googletest cannot be resolved via pkg-config. I assume it exists but no one uses it.
• Adding subtree is likely a way to go. It feels awkward though. Also not sure if it works fine with CMake include(GoogleTest) statement.

There is really no reason to keep this repository closed. I am yet keeping it so for couple of excuses:

• This kind of project should be strictly ABI stable and it's quite not ready for that : we keep it not announced much and loudly claim that everything is subject to change.
• Concepts are not proved workable or usable : it can be "to be proved or can result in failure".
• CI builds don't pass : they don't, but it is either GitHub Actions not supporting CMake 3.10 or bitrise Ubuntu server being too old (16.04) and custom docker image costs. If users can build it then it should be acceptable.
• The C++ code quality is awkward : but everyone writes horrible code in the beginning. I should be honest and expose anything that may result in fair evaluation on any coding skills.
• Various README parts are outdated - they are always likely.

Those excuses are mostly invalid. Yet not making any announcement makes sense.

There are still parts that lack basic considerations, and one of the biggest one is native PluginHost API. It does not manage instantiated plugins, and therefore nothing manages instances in the existing codebase. Obviously, PluginHost is supposed to manage them, and automatically release those instantiated plugins once it is being disposed.

The biggest lack of design happens in the AIDL. The service instantiates a plugin, prepares processing (connect ports), activates, process buffers, deactivates, and lastly destroys. Everything expects only one plugin instance i.e. there is nothing that indicates which instantiated plugin is being processed.

Since there is always at most one service connection between plugin host (client) and the plugin service, whereas there will be more than one instances for a binder connection (multiple tracks x multiple instances of plugins), there should be an instance ID for each plugin instance. The AudioPluginInterface AIDL needs to be rewritten to take it into consideration.

There is a bunch of points there on what we should avoid in realtime processing on this talk: https://www.youtube.com/watch?v=Q0vrQFyAdWI

This is in general difficult and continuous task. We'll close it once a review iteration is done.

tracktion_engine builds on Android, and it is most likely that it can handle audio plugin related bits in JUCE.

Then, if we implement juce_audio_processor for AAP it is likely to work. augene project can be used as a reference player. It is very simple auduo plugin based music player based on tracktion_engine.

Problem: even if we get this working, there is no "instrumental" sample yet.

Currently AAPHostSample contains a bunch of hacks as well as the actual library implementation that should be split from sample app. There has been ongoing efforts to make them separate, but not quite done.

Ideally they should become like:

• libandroidaudioplugin.so contains AAP hosting implementation (plugin package needs this, as Java AndroidAudioPluginService uses this)
• androidaudioplugin.aar contains AAP implementation, for both client and service.
  • it is going to bundle libandroidaudioplugin.so
  • it should be published as a maven package.
• libandroidaudioplugin-lv2.so contains AAP-LV2 specific hosting implementation (AAP-LV2 plugin package needs this)
• androidaudioplugin-lv2.aar contains AAP-LV2 implementation, for service (local hosting).
  • depends on androidaudioplugin.aar.
  • it is going to bundle libandroidaudioplugin-lv2.so
  • it should be published as a maven package too.
• there will be libandroidaudioplugin-vst3.so and androidaudioplugin-vst3.aar for the same purpose.
  • it is important to split them because vst3sdk is licensed under GPLv3.

On a related note, Android native dependencies (lv2 bits, vst3sdk in the future) should be replaced with prebuilt binaries. They can be published as binary git repo which can be submoduled by this repo.

Due to lack of API Level 30 platforms actions/runner-images#1030 the build has been failing.

Right now AAP targets Android-29 due to NdkBinder limitation. But it is not realistic. It should be implemented like Oboe AAudio support where everything is in libdl-based glue API.

We should do the same. For Android < 29, the NdkBinder implementation should be just calls to Java API through JNI (if possible).

Since the NdkBinder API is somewhat big, we had better use clang to generate stub API.

One of the kind of "unique" characteristics of LV2 and inherently AAP is that the plugin ports are exposed as part of manifest. By having them in the manifest, it is possible for hosts to retrieve parameters without instantiating the plugin. On the other hand, each plugin developer has to define parameters in both code and manifest.

It might be a good idea, but there are some use cases that a plugin cannot have a static lists of the ports. For example, a dynamic plugin wrapper doesn't statically know how many ports it will have when it is instantiated. Dexed exposes no parameter from the sound bank "Cart" until it is loaded. This was irrelevant.

LV2 has a (patchy) solution called Dynamic Manifest so that the plugin can dynamically expand the list of ports. AAP does not have that yet, but at this state I wonder if we should have such an "extension" too - this means, ports cannot be determined until a host instantiates the plugin anyways. If we simply remove ports from manifest, things become a lot simpler. JUCE does not expose parameters without instantiation either.

Another (elegant?) solution is to keep current ports in manifests but also have a flag for a plugin that indicates whether it might have additional dynamic ports or not.

Maybe as a solution, we can remove ports from manifest, and if static ports make sense then we provide static information on ports later, which will not become a breaking change.

Now we have various combination of testable audio plugins (LV2-based plugins e.g. mda, and JUCE-based plugins e.g. andes and SARAH) as well as hosts (aaphostsample, JuceAAPAudioPluginHost), there should be some automated connection tests.

Although unlike normal Android apps it is not very straightforward, as we have to install more than just one app, and create a test matrix for all combinations. Also, those hosts need to become testable without UI (or we write UI testing).

To make it harder, there is still no way to automate JUCE-based projects on various CI servers (none successfully built them).

So far, here is what I usually test manually (not always everything):

• aaphostsample can apply mda effects (mda-Delay or mda-Detune)
• aaphostsample can play mda synths (mda-EPiano)
• aaphostsample can play JUCE synths (andes)
• JuceAAPAudioPluginHost can play mda synths (mda-EPiano)
• JuceAAPAudioPluginHost can play JUCE synths (andes)
• JuceAAPAudioPluginHost can play mda synths (mda-EPiano and then apply mda effects (mda-Delay or mda-Detune)

The basic ideas are on README.md.

It cannot be done without some experiment, namely:

• check if loading *.so files is doable under subdirectories e.g. lib/x86/lv2/*/*.so -> NO. Change directory layout at packaging time.
• check if NdkBinder performs well. -> it wouldn't matter as long as we're doing at best.
• check if apps can create realtime audio thread (i.e. whether AAudio can provide ones). -> It is most likely NO, but audio callbacks will be in low latency thread from AAudio.

• Plugin "identifier" is ambiguous so far.

A pluginId (or pluginID? We should use unified naming) is a unique ID across installations and versions.

A true "identifier" is a combination of manufacturer+identifier+version so far, to almost-precisely identify a plugin installed locally (it should probably eliminate manufacturer, just extraneous). We want this to distinguish locally-installed stable product version and ongoing development version, but it might bring further confusion.

Whenever "identifier" is expected, we should probably use "strict match, then lax match" strategy. But it needs some implementation.

As of v0.4 (where we have JUCE processor integration) the API became somewhat inconsistent and nasty. Especially, the plugin API now exposes shared memory FDs which should be internals. There should be some API cleanup for better code quality and hackability.

There are two blockers that seems to prevent things:

• quite unsure if ASharedMemory FDs passed from the client to the plugin are valid. ASharedMemory_getSize() returns 0. And writing results in SEGV_ACCERR or SEGV_MAPPER.
• AIBinder_Class created at plugin (at AAPClientContext) side is problematic. Basically it is dummy now and there is no way to retrieve AIBinder_Class for them from scratch.

They are left unimplemented only because the sizes are different for each. https://github.com/atsushieno/android-audio-plugin-framework/blob/f285b1de64e62e2042d37a4bca0d2e596069ab0a/native/androidaudioplugin-lv2/src/android-audio-plugin-lv2-bridge.cpp#L228

Not only they have installed NDK r21 they have removed NDK r20, which is not the recommended way by NDK team. It makes me think that now their NDK image is not realiably usable...

https://google.github.io/styleguide/cppguide.html

Candidates:

• github actions : no go, due to actions/runner-images#217
• bitrise : stuck at building desktop lv2. Can't find any way to make it public.
• circleci, travis, drone, gitlab : haven't tried yet

We need some AAP example that works as an Instrument.

Now we have LV2 Guitarix port in aap-lv2 atsushieno/aap-lv2#2 but aaphostsample crashes when it tries to use it, because aaphostsample always expects stereo audio while Guitarix only has monoral I/O channels. There should be some adjustment for audio buses to fix this kind of issue.

In terms of desktop support, we'd rename Android-specific serviceIdentifier to containerIdentifier. It also helps JUCE format implementation.

It would be nicer if we can show the plugin icons on the plugin list. I believe VST and AU support it.

Some additional code is needed on aap-metadata parsing, generating, and anywhere else.

For host application, JuceAAPAudioPluginHost and tracktion_engine can be seen as useful proof of concept apps. There should be some useful plugin that works well with the hosts.

An example can be fluidsynth-based plugin like juicysfplugin, but Android build needs non-trivial changes. And since we already have fluidsynth itself working on Android, there is not a lot of advantage over simple MIDI player. Especially, it makes little sense to have to instantiate ones for each track.

What I'm seeing good right now is to have SFZ working. sfizz supports LV2, and it actually involves not a few LV2 extension features. It also requires some complicated parameter bindings through Atom port as well as sfz loader through worker threads, so it would be a good dogfooding for LV2 wrapper.

sfizz needs UI support for file picker as well as other parameters. Therefore it could also be a good dogfooding for UI bindings. sfizz has no UI yet, and there is no UI binding generator, so it would also be required (or at least useful) to create UI binding automator in UI-framework-agnostic way.

Actually it does not have to be LV2 (it is rather difficult) so in practice it would be easy to use the library feature directly.

Current sample app is too hacky and not appropriate at all in terms of output channels. It takes raw wav buffer and pass it to each audio output, where (at least LV2) handles L and R respectively (whereas the app only has "audio output" without mixing two channels) and therefore inconsistent.

It should use MediaCodec and extract audio data for each channel.

Since LV2 backend implementation is stuck on Android and we have very little ideas on how to diagnose and fix the issues, it would be almost time to spare time for desktop backend implementation.

Here is what we don't have locally:

• services and client that depend on AIDL.
• sample app.
• sample LV2 plugin wrapper.

It is somewhat annoying, but time code in LV2 has some kinds while AAP is based on (pseudo) SMF format. To calculate appropriate wait time we need some time format specification and/or calculation.

https://github.com/atsushieno/android-audio-plugin-framework/blob/f285b1de64e62e2042d37a4bca0d2e596069ab0a/native/androidaudioplugin-lv2/src/android-audio-plugin-lv2-bridge.cpp#L221

• Should we still depend on cerbero? -> most likely yes, because lv2 (particularly eg-sampler) depends on libsndfile.

cd serd/
PKG_CONFIG_PATH=/sources/LV2-Android/dest/lib/pkgconfig CC=$CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang CXX=CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang++ LD=$ANDROID_TOOLCHAINS/arm-linux-androideabi-ld CFLAGS="--sysroot=/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot -I/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot/usr/include/ -DANDROID" LDFLAGS="-landroid -Wl,-soname,libserd-0.so" ./waf -d --no-utils --prefix=../dest configure build install

cd ../sord/
PKG_CONFIG_PATH=/sources/LV2-Android/dest/lib/pkgconfig CC=$CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang CXX=CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang++ LD=$ANDROID_TOOLCHAINS/arm-linux-androideabi-ld CFLAGS="--sysroot=/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot -I/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot/usr/include/ -DANDROID" LDFLAGS="-landroid -Wl,-soname,libsord-0.so" ./waf -d --no-utils --prefix=../dest configure build install

lv2: still needs libsndfile

PKG_CONFIG_PATH=/sources/LV2-Android/dest/lib/pkgconfig CC=$CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang CXX=CLANG_TOOLCHAINS/armv7a-linux-androideabi29-clang++ LD=$ANDROID_TOOLCHAINS/arm-linux-androideabi-ld CFLAGS="--sysroot=/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot -I/home/atsushi/Android/Sdk/ndk/20.0.5594570/sysroot/usr/include/ -DANDROID" LDFLAGS="-landroid" ./waf -d --no-coverage --prefix=../dest configure build install

We need plugin UI support. LV2 cannot be a reference model here because we are not living on the desktops that LV2 UI extension specification mentions.

Also, unlike other audio plugin frameworks, AAP exposes an interesting issue here. In general, audio plugin UIs are implemented with the plugin application, and when a user manipulates a plugin UI, it has to be launched from within the plugin process. Then the host goes background. I guess we want to avoid that in general, as it would lose access to foreground AAP services. Also we might want to control plugins while we also want to make some edits on the host. (It might be hardly doable even on desktop nowadays, as they usually show up as a modal dialog.)

Therefore, in short, I'm exploring if we can implement UI foundation that can be loaded within the host process. And here are my thoughts.

There can be in-plugin-process editor and in-host-process editor. Both edit service parameters. It is locally done for in-plugin-process editor, and remotely done for in-host-process editor. For JuceAAPAudioPluginHost, remote access is already achieved through ports.

In-plugin-process editor can be anything (for example we already have juce_gui_basics). To give access to ports we will have to prepare supplemental plugin "parameters" or "properties", which don't exist right now. We will have to define them, as getter for port definitely doesn't exist. (TBD: check how LV2 UI extension handles them.)

Considering desktop integration, my current idea for UI development is to use Flutter (with some necessary dart:ffi backed manipulators), but since it's all up to developer it's just matter of taste.

In-host-process editor cannot be a barely runnable program. What we can do instead is to bring in UI controller container which has access to parameters/properties. The simplest solution here is Web UI components. They only have to be browser-loadable component (can be even hosted on the web, if we give permission).

There were some former attempts to implement audio UI controls for the Web and separate concerns from audio processing:

• https://github.com/g200kg/webaudio-controls/
• https://github.com/hoch/WAAX/ (not maintained)

Also we have juce_emscripten that can even bring JUCE UI for in-host-process editor.

I wouldn't prefer having desktop UI on mobules, but simple UI can be portable with less awkwardness.

What we need here is to define a data transmit method between service and client in our AIDL (internal, not exposed to plugin developers).

Unlike aaphostsample which simply retrieve plugin parameters from imported AAP metadata, the JUCE binding lacks parameter definition imports. They should be done through corresponding juce::AudioProcessor API (addParameter(), addParameterGroup() etc.).

This reminds us that we don't have the concept of parameter groups in AAP itself. There should be some re-designing parameter grouping. Actually grouping might be just too classic. Shorthand property settings might also be helpful. (And considering MIDI 2.0 ages, property "getting" might be also demanded.)

Currently we have to resort to Java API side of Service connection (IBinder instance) to be passed to AAP host (there is only sample code and it is passed directly as jobject so far).

It's better to have native connector without depending on Java API based service queries. The thing here is that there is no corresponding way to query Services in Android NDK API. There have been some example code that shows how to achieve that in NDK, but they are old and depend on non-public API. We likely have to wait for newer Android NDK versions that brings in this feature.

• v0.1
  • host can load and play audio stream, applying in-process plugins that are chained.
  • in-process LV2 plugins can be bridged.
• v0.2
  • host can connect to service-based plugin, operate audio processing, and get results.
  • service-based LV2 plugins can be bridged.
• v0.3
  • host can connect to out-process plugin, operate audio processing, and get results.
  • out-process LV2 plugins can be bridged.
• v0.4
  • create AAP plugin that works as a synthesizer with MIDI support -> AAP-LV2 MIDI synth works.
  • JUCE host works with AAP format support.
• v0.4.5
  • JUCE plugin ports to AAP.
  • review the entire API structure and reduce hacky parts.
  • have some solid client app. -> JUCEAAPAudioPluginHost is good enough.
• v0.5
  • design and implement Kotlin hosting API.
  • remote audio processor runs from within AAudio low latency (realtime-ish) audio thread.
• v0.6
  • shared containers.
  • program/bank names.
  • tail length (VST3 seems to support it, related to "release rate" / "sustain level" ?)
• v0.6
  • plugin editor GUI (we already have JuceAAPAudioPluginHost that can make changes to plugin parameters, so this issue is beyond that)

JuceAAPAudioPluginHost is often? always? horrible with low quality behaviors. It crashes very often only by changing audio settings. The app should consult plugins so that which sampleRate is appropriate (currently it blindly applies the device default sample rate). Each AAP should provide which audio device profile it can support through some property.

But in the end, JuceAAPAudioPluginHost (which is almost unmodified JUCE AudioPluginHost) is too low quality as a showcase. It is most likely problem in JUCE itself, but we need something more appropriate anyways.

Building JUCE bits as part of this repository is getting complicated as we hope to get more samples as well as exploring ways to automate builds. Therefore we're planning to split this module and move JUCE related stuff to another repo, and submodules this repo there.

Current plugin information lacks package name parts (because it was not necessary in desktop, or LV2) but it is required in Android land because it is used to bind services.

It's easy to just add those identifiers, but actually they are 3 parts: packageName, className and plugin name. Maybe packageName and className should be unified into single string like urn:{packageName}/{className}. Android package name can contain only a-zA-Z0-9_.

JuceAAPAudioPluginHost now works for a simple MIDI synth plugin, but if I put another effect plugin after it then the effect part is ignored.

SOUL patches can act as plugins. So just like we import LV2 using lilv, we could import SOUL patches using soul_patch_loader.

There is most likely some intensive operation that prevents realtime processing. Needs some investigation.

• Audio buffers are not precisely mapped. Current implementation is: 0 is left, 1 is right. There is no audio bus consideration. No respect on LV2 channels either.
• control parameters are not precisely assigned. Any changes to JUCE parameters don't get reflected on AAP instance.

Ports should have some properties.

For example, it is not predictable for hosts if a port is supposed to receive a lot of values continuously (e.g. filter gain, pan, pitchbend) or rarely changed (e.g. switching legato, glissando, etc.).

LV2 seems to have such parameter kinds e.g. expensive so we might want to have something similar for better host implementation performance.

While this project has started with first-class LV2 support, this project itself is independent of LV2. And for repository management point of view, it is better to split LV2 bits from AAP framework itself.

(We don't follow monorepo here, we are not fluent in CI resources and need realistically maintainable build setup with rational minutes as well as near-future package management.)

The new repo will be quite like aap-juce, with more attempts to import quality plugins that already exists on Linux desktop world.

The corresponding API exists, but it is not implemented.

JUCE AudioPluginInstance::processBlock() tells us that it is up to the underlying audio plugin framework that interprets and processes MidiBuffer part. JUCE VST3 processor implements it in MidiEventList::toMidiBuffer() which populates Steinberg::Vst::Event instances. For AU it is MusicDeviceMIDIEvent.

AAP needs something similar, but there is a problem here - there should be backend feature that converts those abstract MIDI messages to the backend-specifics. For LV2 backend, it is LV2 Atom.

We need AudioPluginHost implementation (not to be confused with JUCE AudioPluginHost) that host app developers can easily use without native hassle. Also useful for proof of concept implementation.

It does not have to provide fully-featured hosting implementation, and can be done partial migration from AAPHostSample step by step.

Design notes:

• The host works as a client to AudioPluginServices. Services are apps, so one service handles multiple binder connections from multiple clients (hosts). But connections are (should be) made every time client requests. It is possible that many many channels can be connected between the same pair of a service and a client.
• Right now there is no native API that can start and manage service connection as there is no way to query available AudioPluginServices. It will be handled via Kotlin AudioPluginHostHelper API.
• The entire process of connecting to a service is based on asynchronous callback (Activity.onServiceConnected()). While the plugin framework API is synchronous, the actual implementation likely have to be asynchronous. Or maybe the API needs to transform into aynchronous style. The trends are towards asynchronous (e.g. Web MIDI API, Android MIDI API).

(The plan used to be that we were going to migrate to a completely new host app sample based on Jetpack Compose. Since it's not going to happen anytime soon, we should rather make existing aaphostsample more reliable for daily development basis.)

There are bunch of issues to resolve in current aaphostsample:

• plugin list view is inconsistent; its view recycling is broken. There must be something that holds state inappropriately.
• The switch UI is awful; they should be replaced as a single "apply" button as there is currently no chaining.
  • The button should appear only in the parameter details, not the list. Showing switches on the list causes inconsistent state where the parameters are configured with the "selected" plugin while the actually applied plugin is different. That causes plugin crash because there are unexpected parameter value settings.

From issue #24:

we don't have the concept of parameter groups in AAP itself. There should be some re-designing parameter grouping. Actually grouping might be just too classic. Shorthand property settings might also be helpful. (And considering MIDI 2.0 ages, property "getting" might be also demanded.)

clang/llvm on Bitrise server seems quite old and does not support libc++fs, resulting in build failures on missing header for #include <filesystem>. Or it might be just missing packages, but we have libc++-dev and libc++abi-dev installed.

Anyways as a result bitrise builds fail and does not function as a CI server. It is primarily bitrise problem which is stuck at Ubuntu 16.04, but we need some fixes in either way. It might be good to find other CI services.

Considering those CI services should frequently run and those native module dependencies tend to block that, we should probably divide the repository and remove those Android native dependency builds from here.

cerbero build is there only to build various libsndfile dependencies (libogg etc.) and it may be even better idea to isolate them from the rest of LV2 world.