mozilla-mobile / android-automation-tools Goto Github PK

We upload releases of Focus/Klar manually to archive.mozilla.org.

This is a source of errors (inconsistent file names, wrong directory structure, forgotten releases) and we should do this automatically as part of our release pipeline.

Like MickeyMoz's regular l10n imports, it'd be great to regularly run a lossless image optimization on registered repositories (saving size is a big deal in Android) and open PRs on all the relevant repositories.

Currently: with #19, we have scripts for a local user to do this process but we don't automatically open PRs (just provide links for the user to open them). Unfortunately, they're probably not robust enough to run on a server. I intend to run it about once a month but it'd be cool to automate.

Sebastian previously gave me info about how MickeyMoz runs:

MickeyMoz runs on taskcluster. We use the "hooks" service to run tasks periodically like cron jobs.

For example that's the task we run for creating the strings import PRs:
https://tools.taskcluster.net/hooks/project-focus/strings-import

In this case the code lives in the focus repository:
https://github.com/mozilla-mobile/focus-android/blob/master/tools/taskcluster/import_strings_and_create_pull_request.sh
https://github.com/mozilla-mobile/focus-android/blob/master/tools/taskcluster/create-pull-request.py

https://plugins.gradle.org/plugin/org.mozilla.android suggests:

plugins {
  id "org.mozilla.android" version "0.1"
}

However, when trying this in Echo Show, this results in the error:

Could not find com.android.tools.build:gradle:3.2.0.
Searched in the following locations:
  - https://plugins.gradle.org/m2/com/android/tools/build/gradle/3.2.0/gradle-3.2.0.pom
Required by:
    project :app > org.mozilla.android:org.mozilla.android.gradle.plugin:0.1 > org.mozilla.android:gradle-plugin:0.1

My best guess is that we don't include the google plugin repository in our gradle plugin definition, which the new syntax requires, but the old syntax works fine (because it uses the config from the application).

We have some code in Focus (and future apps) to generate a version code for release builds. How can we share this code? Could this be a gradle plugin?

https://github.com/mozilla-mobile/focus-android/blob/master/tools/gradle/versionCode.gradle

e.g. to capture #30.

During the fire TV androidX migration mozilla-mobile/firefox-tv#1398, we ran into compile errors because an old version of the android gradle plugin, defined in the buildSrc directory, in conjunction with a new version (defined in the app) was not compatible with jetifier. To avoid this issue, it's very likely we'll need to upgrade the existing version of our android-gradle-plugin definition.

FFTV is currently on 3.3.1; the latest version is 3.3.2.

The configurations are basically duplicated for components, focus, and TV.

fwiw, on Fire TV, I noticed that there appears to be a coverage task we don't use:

./gradlew -Pcoverage tasks | grep -i coverage
createDebugCoverageReport - Creates test coverage reports for the debug variant.

I wonder if this is generated automatically rather than the jacoco task we created and use?

In the Github Browser Sample on Android Archictecture components samples repo, this task is also created and seems to cover UI tests (which we don't measure coverage for).

Like we use "mach" in mozilla-central.

Before building a release build, we should verify:

• No uncommitted changes
• We have a git tag checked out
• The buildVersion matches the git tag (since git tags are unique, this ensures build versions are never re-used)
• API keys were properly added

The first three are already addressed in a firefox-tv buildSrc/ task: https://github.com/mozilla-mobile/firefox-tv/blob/master/buildSrc/src/main/java/org/mozilla/gradle/tasks/ValidateAndroidAppReleaseConfiguration.kt

Each project we set up has to get static analysis up and running. Usually, this is neglected in favor of features and then it becomes a pain to get static analysis working after the fact when many things are already failing. It'd be great if we could provide:

• A list of recommended static analysis tools
• Default configuration files for these tools
• Snippets about how to add and activate these tools

So that setting it up initially is trivial so it gets done early.

Unfortunately there is no easy distribution mechanism for XML files but it only needs to happen once (and there are likely to be local edits to the configs) so I think this is fine.

Focus-android has a list of their CI tools: https://github.com/mozilla-mobile/focus-android/wiki/Automation

And the Firefox TV repo uses the most strict Android lint we can put together (in combination with warningsAsErrors). We're also working on detekt mozilla-mobile/firefox-tv#807 and those are the major configurable kotlin-friendly tools we use.

Summary from Slack: reproducible builds allows others to guarantee that the compiled application you're running comes from the source code you've compiled so you can audit the source code (among other perks). It's not too hard to do with gradle/kotlin but currently, in Focus & friends, we:

• Sometimes bundle API keys
• Dynamically generate the versionCode

Full conversation:

mcomella [14:46]
Are bintray dependencies distributed with bytecode or by source? Wondering about this issue in components where a dependency includes java 8 bytecode: https://github.com/mozilla-mobile/android-components/issues/925
pocmo
#925 Cannot use dependencies with Java 8 bytecode
```The dependency contains Java 8 bytecode. Please enable desugaring by adding the following to build.gradle
android {
    compileOptions {
        sourceCompatibility 1.8
        targetCompatibility 1.8
 Show more
Labels
:building_construction: build
mozilla-mobile/android-componentsToday at 05:59Added by GitHub


nalexander [15:26]
@mcomella in general, they’re distributed as JVM bytecode (.aar/.jar/.class files).

mcomella [15:39]
@nalexander Is that a problem for us? Would we prefer to build by source?

nalexander [15:39]
@mcomella I think it’s an advantage for us — shipping compiled JVM bytecode is a hell of a lot faster than building from source!
And building Google Support Libraries from source — when it’s available — probably requires checking out most of the Android source tree… it’s a huge endeavour.
Be thankful that the JVM engineers worked this out 20 years ago!
30 years ago?  Early 90s, I think.

mcomella [15:43]
:slightly_smiling_face: And I guess if we’re compiling against the libs, they can’t obfuscate their bytecode so we know what we’re getting.

@nalexander Is there any way to verify the authenticity of the source? Like this bytecode was compiled from this source tree beyond doing the compilation ourselves? And, provided we had infinite time, would it be worth us verifying our dependencies are what we think they are regularly?

nalexander [15:44]
@mcomella aye, there’s that — although Java essentially does *no* obfuscation.
It’s entirely possible (and I do it semi-regularly) to decompile Java and read the input Java.
Presumably Kotlin is a little harder to read.
And no, there’s no input-to-output verification that incorporates the sources directly.  (This is exactly like reproducible builds.)
There’s a web of trust that runs through hashes and signatures in the distribution network (Maven!).  But the hash is of the output and doesn’t capture the inputs.
And even if you could make that link, it’s incredibly difficult to capture the toolchains and local configurations…

mcomella [15:48]
Interesting, thanks for the run down

nalexander [15:52]
yw.

mcomella [15:53]
@nalexander Do you think reproduce builds is something we should strive for? It seems like it'd be straight forward with kotlin/gradle provided upstream dependencies don't change published packages but 1) we package API keys sometimes and 2) we generate the app version code

nalexander [15:54]
In general, you should strive for reproducible builds, and you probably are very close to having them.  Don’t put dates/file paths in your source code and you’re probably very close to reproducible already.
Pinning your dependencies exactly helps: i.e., don’t rely on loose semver to take patch upgrades.
But after that, it’s not worth really fighting too hard, in my opinion.

These are in the photon guidelines - https://design.firefox.com/photon/copy/punctuation.html - and I recently came across this in practice: mozilla-l10n/firefoxtv-l10n#7

We may want to check with Delphine to figure out what other export checks could be automated.

After #30, we'll have some tests for our gradle plugin: we should add continuous integration to ensure those tests are run every build and don't fail.

Currently, we create the license.html files (firefox-tv example) by hand which is a lot of work and error-prone: it'd be great to generate the license pages and share this code across all of our projects.

Note that we may require an ability to add in extra dependencies/licenses that wouldn't show up in gradle (e.g. maybe we compile node.js dependencies into the javascript we inject into pages).

i.e.

for repo in registered_moz_repostiories:
  pushd repo_dir
  optimize .
  git commit . -am "No issue: optimize!"
  git push --no-verify # and provide link to user to create PR on GH interface
  popd ..

This can be used by a user (e.g. me) on a regular basis (e.g. monthly) to optimize our assets across a few repositories. Eventually, it'd preferably move to cron job, like MickeyMoz's string imports.

That's the script we have for Focus:
https://github.com/mozilla-mobile/focus-android/blob/master/tools/taskcluster/execute-firebase-test.sh

A bunch of other apps may want to test on Firebase via taskcluster and we should make this as easy as possible.

So instead of:

repositories {
    maven {
        url "https://maven.mozilla.org/maven2"
    }
}

We get:

repositories {
    mozilla()
}

In Focus we have a script that makes sure we only ship the languages we want to ship in our release builds:
https://github.com/mozilla-mobile/focus-android/blob/master/tools/l10n/filter-release-translations.py

This script may be useful for other projects too.

We do have scripts in the Focus repository for a bot that opens PRs for string imports:

https://github.com/mozilla-mobile/focus-android/blob/master/tools/taskcluster/import_strings_and_create_pull_request.sh
https://github.com/mozilla-mobile/focus-android/blob/master/tools/taskcluster/create-pull-request.py

Those scripts are triggered from a taskcluster hook. Can we make them generic and trigger them for multiple projects without every project needing its own hook/scripts?

We have some scripts in Focus for that.

We have been running into problems with slow emulators. Some history is in this issue: mozilla-mobile/focus-android#980

Once we have someone working on this I can elaborate more.

The README doesn't say how to apply the plugin. Currently, only the legacy DSL works (see #37). Here's instructions on how to apply the plugins: https://plugins.gradle.org/plugin/org.mozilla.android We should also link to that page.

Severin mentioned in my PR landing this on FFES that it might be clearer where these tasks come from if we namespace them. While most gradle plugins don't seem to do this, I think this is a case to break with conventions because it's helpful to namespace them so it's clearer to know where they're coming from (like we do our code).

We have some plugin code in our ac repository. Maybe it should live here instead?

GVNightlyVersionVerifierPlugin: This plugin can automatically update a local GeckoView dependency (currently only Nightly and currently only for a very specific configured project) if there's a newer version available. It only updates if the major version hasn't changed.
https://github.com/mozilla-mobile/android-components/blob/master/buildSrc/src/main/java/GVNightlyVersionVerifierPlugin.kt#L15

GitHubPlugin: A plugin that can open pull requests or issues in a project.
https://github.com/mozilla-mobile/android-components/blob/master/buildSrc/src/main/java/GitHubPlugin.kt

Also see docs in PR:
mozilla-mobile/android-components#934

We have 120283 scripts in the Focus repository that do that for one specific secret. In the Fenix repository I'm writing a more generic version of this. However eventually this should live here.

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Focus maintains its own fork of Android2Po so that we can add languages that do not have a matching locale in babbel yet. If we keep this process then it would be a good idea to make this fork available to others.

https://github.com/mozilla-mobile/focus-android/tree/master/tools/l10n

Mike has started the work to create a Mozilla Gradle plugin.

Let's add a helper to setup the Mozilla repository - so that we can just do:

repositories {
    mozilla()
}

Ideally, we'd be using TaskCluster, or similar to publish new releases. See #31 for setting up CI.

Sometimes malicious packages with similar names can be hosted on different packaging sites: https://blog.autsoft.hu/a-confusing-dependency/

We should make sure the dependencies are listed in an optimal order like:

• mozilla maven
• google maven
• jcenter

In #32, we get find the release tasks by checking that they start with assemble and end in Release: this is pretty hacky. The Android gradle plugin might have more context and be able to give these tasks to us.

PyPI?
http://peterdowns.com/posts/first-time-with-pypi.html