Running SynthMark from the commend line is often not an option.
So we need to control the APK by sending an Intent.

What is the best way to do this?

• Markup .md files?
• Wiki?
• or GitHub Pages?

After a glitch happens in LatencyMark, the experiment restarts, but the experiment is not taken back to a clean status.
HostCpuManagerBase, for example, keeps the last workUnits and CustomCpuManager keeps some time measurements that may be screwed.

Does it make sense to restore the system to an initial state when the experiment is restarted?

The new code that uses strtol() is causing good command line arguments to fail.

REPRODUCE
On S9 running Android OC_MR1, enter:
adb shell synthmark -tv -p90

It will reject the "90" as invalid!
This works fine on Android Pie..

Here is the commit for the code that is failing.
a93ca11

We should test on Pixel running OC_MR1 to see if it is specific to Samsung.

When launched from an Intent, it did a linear search instead of the modified binary search.
So it is taking longer to run the test.

Please add the following, or something similar, taken from ag/7201451/2..3

---- cut here ---

Explanation of benchmarks:

The voicemark.90 is a measure of how much computation the device is
capable of when running at 90% of its maximum CPU load.

A “voice” is a unit of computation based on a two oscillator
subtractive synthesizer that is common in plugins and sound formats
such as Downloadable Sounds (DLS) and Sound Font (SF2). It is roughly
equivalent to a “Mini Moog” synthesizer.

The number of voices was chosen because that is enough polyphony for
a decent keyboard app with voices that release slowly. It allows for
poly-timbral apps that combines multiple types of sound, such as drums
and pads. Note that many phones can play > 200 voices so this number
should be easily achieved.

The latencymark.fixed.little value gives us the minimum latency of the
top level buffer closest to the app for a constant workload. The DSP
and other lower level components might add 4-6 msec of latency when
using EXCLUSIVE MMAP mode in AAudio. This would give us a total output
latency of around 20 msec.

The latencymark.dynamic.little measures latency for a changing workload,
which is common in interactive applications. Higher latency is required
because the scheduler may not increase the clock rate rapidly enough to
meet the changing demand.

(Note that the latency values are for the top level of a virtual audio
device with 2 msec buffers. So SynthMark can achieve lower latency values
than the actual audio framework with multiple levels and a DSP.)

LatencyMark: #8, 10 seconds with bursts = 128 -----------
LatencyMark: detected glitch at 2.97 seconds
ERROR - at maximum buffer size and still glitching
Got LatencyMark = 0.0 msec

CDD_SUMMARY_BEGIN -------
voicemark.90 = 67.6868
latencymark.fixed.little = 2
latencymark.dynamic.little = 0
CDD_SUMMARY_END --------

It measures "latency.light.big = 96", which is confusing.

The 0 result for "latencymark.dynamic.little = 0" is because it hit the maximum latency.

Originally posted by @philburk in #86 (comment)

It should report some high maximum value like 512 msec.

Google is disabling Travis today.

The GUI app allows us to run SynthMark on unrooted phones.

Maybe -od for optimize using debug
-on for no optimization (default)

-xd for experimental Deadline

Steps to repro:

• Open the android subdirectory in Android Studio (tried both 3.4 canary 4 and 3.2.1)

Expected result:

• Project builds successfully

Actual result:

• Project fails to build with "Error configuring" under the "External Native Build issues:" section.

Model it after the way that Rick ORang Loopback works.
Python code to drive Rick ORang is in google3 "android/media/performance/audio_loopback_usb_test.py"

cat /dev/stune/rt/schedtune.boost

On a 4 core device, run SynthMark and click Settings.
Fader allows one to select CPU 0-7. Expect only CPU 0-3.

Selecting CPU 4-7 returns error -22 from setCpuAffinity().

So we can refer users to a specific version.
Add app version to the reports for Share.

This will prepare us for #72
so we can always reach the "Cancel" button.

Currently it can only be run from the command line using -ta

Just leave the int main() in the apps folder and have it call a library function SynthMarkCommand(argc,argv).

If you enter -nx then it will just set n to zero with no error message.
It should print an error message and abort.

Please add some more steps to the instructions here:
https://github.com/google/synthmark/blob/master/scripts/android/autopower/README.md

Let's make it very easy.
Please describe connection of each cable, including power and USB.
Please also describe commands for applying voltage.
Every step from start to finish.

Getting the following linker error when building for Android:

undefined reference to `HostCpuManager::mWorkloadHintsEnabled'

Create a scripts folder

Measure VoiceMark 95 then divide that range by 10 steps and run UtilizationMark.

This will make it easier to see when there is a big gap, for example when it jumps to 50 msec.

Some results appear in command line but not on app.
Too many ways to log in SynthMark

mLogTool->log() writes to an std::ostream or printf if ostream not specified. Prefixed with "[log]".
Prefix did not appear in app because of a bug.

SynthMarkResult has a std::string that gets appended to by various tests and then printed at the end. Appears in log after test is complete.

    mLogTool->log(mResult.getResultMessage().c_str());

printf used for errors and some logs
printf("allNotesOn(%d) exceeded maxVoices of %d\n", numVoices, mMaxVoices);

CDD says:
The SynthMark app needs to be run using the "Automated Test" option and achieve the following results:

voicemark.90 >= 32 voices
latencymark.fixed.little <= 15 msec
latencymark.dynamic.little <= 50 msec

The corresponding SynthMark results are:

voice.mark.little = 73.4724
voice.mark.big = 276.446
audio.latency.msec.2.18.18 = 4
audio.latency.msec.2.18.54 = 8

This would involve storing all important timing information in a compact format.
It can be dumped to a VCD format file and viewed using a VCD waveform viewer.
Store:
task awake/asleep
buffer depth
cpu index

Dump:
synthesized ideal burst timing so we can see when a task is late

The Q CDD may recommend specific values. These should be made more obvious.

It would be nice to see the progress of the test.
But we do not want to print from the real-time thread because
it would interfere with the timing.
So we should write the logs to a FIFO and pull from it and print in a background thread.

Threads

• audio callback, generates logs
• test, runs one or more subtests,
• monitor, Java thread for app, main() for shell, polls test, prints logs and exits when test complete
  test.launch()
  do {
  if (test.hasLogs()) print log
  else sleep(100 msec)
  while (test.isRunning)
  drain logs

CPUs may be split 2+2, 2+4, 4+4 or 2+6, etc.
Benchmark each CPU individually.
https://en.wikipedia.org/wiki/ARM_big.LITTLE

Add a new test that gives a clear measure of the time to recover when the work load increases. LatencyMark restarts the test with a bigger buffer when there is a glitch. Add a test that just measures how long it takes to render in real-time again after a workload increase.

REPRODUCE
adb shell synthmark -tj -n16 -w0
adb shell synthmark -tj -n16 -w1

EXPECT
Similar render times because load is not changing.

ACTUAL
Render times with -n16 are very low indicating high utilization.
Also this is printed:
workUnits=16 requestDLBandwidth=0.900000 [###########################___]

SynthMark initially sets the utilization very high (90%) so that it can calibrate the workload. But it does not lower the utilization until the workload changes. After the initial calibration, it could check to see if the utilization is set too high and lower it.

On some Android systems, some cores are reserved for the foreground process.
Background processes should avoid them and foreground processes can use them.

We should call Process.getExclusiveCores() and then display the resulting int array.

https://developer.android.com/reference/android/os/Process?hl=ml#getExclusiveCores()

So we can see how the CPU governor behaves

We just need this in the app. We can always control-C the command line version.

Use Monsoon power meter

Allow SynthMark app to run in background by not stopping it. Use a Fragment?

When the app is launched from an Intent, that should be enabled so we can run other apps in front of SynthMark.

LatencyMark does a linear search, which can be very tedious.
Go up by powers of two then do binary search once bracketed.

I noticed that because of

It doesn't really matter the result of this this condition

We will always use the STATE_RAMP_UP approach.

Run these two commands
synthmark -tj -c3 -n2
synthmark -tj -c3 -n2 -N60

Expect very different result.

Actually they are about the same.

Return error or implement -N option.

About a third of the time. SynthMark would hang when the test was finished. It would then never print the test results.

This was caused by returning AAUDIO_CALLBACK_RESULT_STOP from the callback and then immediately closing the stream. This caused a race condition.

The fix involves returning AAUDIO_CALLBACK_RESULT_CONTINUE and then preventing the callbacks from executing the thread procedure multiple times.

On https://github.com/google/synthmark/blob/master/docs/README.md the link titled "Synthesizer Architecture" links to: https://github.com/google/synthmark/blob/master/docs/Synthesizer.md which does not exist.

I ran the SynthMark App latency test on Android device with 6/74 voices. I expected a high latency because it was a changing workload. But I measured a latency of only 6 msec!

But then I built SynthMark using the NDK and ran it as a command line executable.

adb shell synthmark -tl -n5 -N75 -c1

That measured a latency of 112 msec!

I think the difference between the app and the command line may be because:

1. I am touching the screen when I run the app. That can boost the frequency.
2. The app has a continuously updated status with a rotating icon. That may increase utilization enough to boost the frequency.

Oboe has a tool called StabilizedCallback that forces a consistent CPU utilization by running a fake load. We could measure various fake load techniques in SynthMark.

Could be the number of wakeups that would cause an underrun with a single buffer.... or...

In some cases the slower CPU is identified as "big".
And vice versa.

Open /proc file for each CPU at start of benchmark.
Read frequency periodically, accumulate in time slots.
Display "heat map" ASCII chart that includes CPU index.

It should call its superclass.

This is so that touch boost will not affect the performance result by raising the CPU clock frequency.
Ideally the test should be run after the touch boost has timed out.