anestisb / android-prepare-vendor Goto Github PK

Update extract & vendor generate scripts to also include bootloader & baseband images at the generated vendor blobs.

When JDK is set to 1.7 boot class extraction will fail:

[*] Preparing environment for 'arm' ABI
[!] Boot classes extraction failed
[-] System partition de-optimization failed
...

icyerasor@aosp:/tmp/android_img_repair.IBHeWV/arm$ java -jar /home/icyerasor/android-prepare-vendor/hostTools/Java/oat2dex.jar boot /tmp/android_img_repair.IBHeWV/arm/boot.oat
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/rh/smaliex/Main : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

Solution: (install JDK 1.8 &) update-alternatives for java and javac to use 1.8
Depending on what branch you try to compile, you might have to switch back to 1.7 again before building aosp.

Not sure on how to improve this, maybe mention it int he requirements or as one of the causes of "boot classes extraction failed"?

Modify extract-factory-images.sh to support extracting files for processing directly from loopback mounted raw images. This will save lots of time since rsync a local copy will no longer be required.

nexus 6p in osx 10.11 can't build

Contents of /vendor/etc/audio_effects.conf appear to be different between AOSP builds and production release images for Nexus 6p. The scripts automatically pick the config file when extracting data from production image, although the file is already included from master device.mk here.

Since AOSP build system ignores destination duplicates of PRODUCT_COPY_FILES keeping only the first occurrence, the version we include from the scripts is ignored.

Investigate if we need to prioritize /vendor PRODUCT_COPY_FILES to always be selected from build system by appending them at the generated BoardConfigVendor.mk instead of the current angler-vendor-blobs.mk

diff -u angler/nmf26f/factory_imgs_data/vendor/etc/audio_effects.conf /fast-datavault/aosp_prod/out_release_Linux_7.1.1_r1/aosp_prod/target/product/angler/vendor/etc/audio_effects.conf
--- angler/nmf26f/factory_imgs_data/vendor/etc/audio_effects.conf	2009-01-01 10:00:00.000000000 +0200
+++ /fast-datavault/aosp_prod/out_release_Linux_7.1.1_r1/aosp_prod/target/product/angler/vendor/etc/audio_effects.conf	2016-12-06 18:54:13.928473755 +0200
@@ -18,21 +18,21 @@
   visualizer_hw {
     path /system/lib/soundfx/libqcomvisualizer.so
   }
-  fmas {
-    path /system/lib/soundfx/libfmas.so
+  downmix {
+    path /system/lib/soundfx/libdownmix.so
   }
   proxy {
     path /system/lib/soundfx/libeffectproxy.so
   }
-  loudness_enhancer {
-    path /system/lib/soundfx/libldnhncr.so
-  }
   offload_bundle {
     path /system/lib/soundfx/libqcompostprocbundle.so
   }
   qcom_pre_processing {
     path /system/lib/soundfx/libqcomvoiceprocessing.so
   }
+  loudness_enhancer {
+    path /system/lib/soundfx/libldnhncr.so
+  }
 }
 
 # Default pre-processing library. Add to audio_effect.conf "libraries" section if
@@ -95,8 +95,18 @@
     }
   }
   virtualizer {
-    library fmas
-    uuid 36103c50-8514-11e2-9e96-0800200c9a66
+    library proxy
+    uuid d3467faa-acc7-4d34-acaf-0002a5d5c51b
+
+    libsw {
+      library bundle
+      uuid 1d4033c0-8557-11df-9f2d-0002a5d5c51b
+    }
+
+    libhw {
+      library offload_bundle
+      uuid 509a4498-561a-4bea-b3b1-0002a5d5c51b
+    }
   }
   equalizer {
     library proxy
@@ -187,8 +197,8 @@
     }
   }
   downmix {
-    library fmas
-    uuid 36103c51-8514-11e2-9e96-0800200c9a66
+    library downmix
+    uuid 93f04452-e4fe-41cc-91f9-e475b6d1d69f
   }
   loudness_enhancer {
     library loudness_enhancer
@@ -204,6 +214,15 @@
   }
 }
 
+pre_processing {
+    voice_communication {
+        aec {
+        }
+        ns {
+        }
+    }
+}
+
 # Default pre-processing effects. Add to audio_effect.conf "effects" section if
 # audio HAL implements support for them.
 #
@@ -256,11 +275,6 @@
 #        ...
 #    }
 
-pre_processing {
-    voice_communication {
-        aec {
-        }
-        ns {
-        }
-    }
-}

So, if I reproduce my actions correctly I've done about the following:

cd $HOME/android
make clobber
cd $HOME/android-prepare-vendor
sudo ./extract-all.sh -d bullhead -b MTC19V -o $HOME/android
cd $HOME/android
. build/envsetup.sh
lunch 18
make -j2
make dist
./build/tools/releasetools/sign_target_files_apks -o -d ~/.android-certs out/dist/*-target_files-*.zip signed-target_files.zip
./build/tools/releasetools/img_from_target_files signed-target-files.zip signed-img.zip

I see that the generated makefiles are included:

including ./bullhead/mtc19v/vendor/lge/bullhead/Android.mk ...

However, no vendor.img is included in the final signed-img.zip and I can't locate any intermediaries. When flashing the generated images the system won't detect the SIM card.

There might be a user error on my side, or maybe some changes in the latest version. In case of the latter, I'll happily make all the necessary changes myself in order to get it working, however I have no clue where to begin looking.

I'll appreciate any hints, thank you.

I am pretty sure this is caused by missing something in AOSP / android-prepare-vendor. It might have another local cause though, so it would be good if someone could confirm that they hit this.

root@k3vinlu:/home/k3vinlusec/Downloads/android-prepare-vendor-master# ./execute-all.sh -d bullhead -b NBD90W -o ./bullhead/
[] Setting output base to '/home/k3vinlusec/Downloads/android-prepare-vendor-master/bullhead/bullhead/nbd90w'
....
....
[?] I have read and agree with the above terms and conditions - ACKNOWLEDGE [y|n]: y
[] Downloading image from 'https://dl.google.com/dl/android/aosp/bullhead-nbd90w-factory-d2f11c5f.zip'
--2016-10-10 17:14:11-- https://dl.google.com/dl/android/aosp/bullhead-nbd90w-factory-d2f11c5f.zip
Resolving dl.google.com (dl.google.com)... 216.58.195.78, 2607:f8b0:4005:804::200e
Connecting to dl.google.com (dl.google.com)|216.58.195.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1243266718 (1.2G) [application/zip]
Saving to: ‘/home/k3vinlusec/Downloads/android-prepare-vendor-master/bullhead/bullhead/nbd90w/bullhead-nbd90w-factory-d2f11c5f.zip’

100%[========================================================================================>] 1,243,266,718 5.77MB/s in 3m 31s

2016-10-10 17:17:42 (5.63 MB/s) - ‘/home/k3vinlusec/Downloads/android-prepare-vendor-master/bullhead/bullhead/nbd90w/bullhead-nbd90w-factory-d2f11c5f.zip’ saved [1243266718/1243266718]

[] Processing with 'API-24 config-naked' configuration
[] Extracting '/home/k3vinlusec/Downloads/android-prepare-vendor-master/bullhead/bullhead/nbd90w/bullhead-nbd90w-factory-d2f11c5f.zip'
[*] Unzipping 'image-bullhead-nbd90w.zip'
[-] '/tmp/android_img_extract.MTMVz1/bullhead-nbd90w-factory-d2f11c5f/images/system.img.raw' mount failed
/home/k3vinlusec/Downloads/android-prepare-vendor-master/scripts/extract-factory-images.sh: line 93: fuse-ext2: command not found
[-] Factory images data extract failed

When I run this I get the error in the title.

I am running on Ubuntu 14.04 Desktop. I have installed all the build packages required for AOSP, including openjdk-8-jdk. I turned on the debug in the system script and the following is the console output.

outputfailed.txt

What is going on and how can I fix this?

Currently vendor generate script is marking all framework JARs under /system as pre-signed.

include $(CLEAR_VARS)
LOCAL_MODULE := com.google.widevine.software.drm
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_OWNER := lge
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/$(TARGET_COPY_OUT_SYSTEM)/framework
LOCAL_SRC_FILES := proprietary/framework/com.google.widevine.software.drm.jar
LOCAL_CERTIFICATE := PRESIGNED
LOCAL_MODULE_CLASS := JAVA_LIBRARIES
LOCAL_MODULE_SUFFIX := $(COMMON_JAVA_PACKAGE_SUFFIX)
LOCAL_DEX_PREOPT := false
include $(BUILD_PREBUILT)

However the new DEX repair method appends bytecode back to archive and deletes old signature. When marked as presigned it build chain skips zip-align/re-sign and thus breaking the archive.

Need to investigate if JARs need to be marked as pre-signed when extracted (thus not delete META-INF) or can be treated in same way as APKs and use LOCAL_CERTIFICATE := platform flag.

Thanks to @thestinger for pointing out the problem.

Hey Anestisb,

I have a few nexus devices and no doubt have been using your tool to assist in extraction of vendor. i have noticed in both angler and bullhead that there are a couple of duplicates. I was thinking you may need an ifdef or something to decide whether to pull from the vendor or not. At present i just delete them as the issue arises when pulling from new factory images.

Files i see are; (in android.mk and in /lib and /lib64)
loc_api_v02
libloc_ds_api

from a guess they are gps related eh?

I used this tool to build AOSP Nougat 7.0 on my Nexus 5x, it seems that it works. But when I restart the device each time, there is a dialog like "There's an internal problem with your device...." shown in the screen. I'm not sure if there's a bug in this tool.

https://android.googlesource.com/device/huawei/angler/+/android-7.0.0_r6/aosp/Android.mk

# Create links from /vendor to /system for AOSP without vendor builds
ifneq (,$(filter aosp_angler, $(TARGET_PRODUCT)))
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := AOSPLinks
LOCAL_MODULE_OWNER := google
LOCAL_MODULE_TAGS := optional
LOCAL_POST_INSTALL_CMD := \
    test -s vendor/huawei/angler/proprietary/prebuilt/target/product/msm8994/system/etc/izat.conf || { \
    mkdir -p $(PRODUCT_OUT)/system/etc; \
    ln -sf /vendor/etc/izat.conf $(PRODUCT_OUT)/system/etc/izat.conf; \
    ln -sf /vendor/etc/flp.conf $(PRODUCT_OUT)/system/etc/flp.conf; \
    ln -sf /vendor/etc/sap.conf $(PRODUCT_OUT)/system/etc/sap.conf; \
    ln -sf /vendor/etc/pp_calib_data_SAMSUNG_S6E3HA3X01_5P7_1440P_CMD_DUAL0.xml $(PRODUCT_OUT)/system/etc/pp_calib_data_SAMSUNG_S6E3HA3X01_5P7_1440P_CMD_DUAL0.xml; \
    mkdir -p $(PRODUCT_OUT)/system/etc/data; \
    ln -sf /vendor/etc/data/dsi_config.xml $(PRODUCT_OUT)/system/etc/data/dsi_config.xml; \
    ln -sf /vendor/etc/data/netmgr_config.xml $(PRODUCT_OUT)/system/etc/data/netmgr_config.xml; \
    mkdir -p $(PRODUCT_OUT)/system/etc/cne; \
    ln -sf /vendor/etc/cne/profile.txt $(PRODUCT_OUT)/system/etc/cne/profile.txt; \
    mkdir -p $(PRODUCT_OUT)/system/bin; \
    ln -sf /vendor/bin/imscmservice $(PRODUCT_OUT)/system/bin/imscmservice; \
    ln -sf /vendor/bin/imsdatadaemon $(PRODUCT_OUT)/system/bin/imsdatadaemon; \
    ln -sf /vendor/bin/imsqmidaemon $(PRODUCT_OUT)/system/bin/imsqmidaemon; \
    ln -sf /vendor/bin/ims_rtp_daemon $(PRODUCT_OUT)/system/bin/ims_rtp_daemon; }
include $(BUILD_PHONY_PACKAGE)
endif # aosp_angler

Due to differences in expected locations some symbolic links for angler are not generated. We probably need to generate an additional fake package (BUILD_PHONY_PACKAGE) and introduce a LOCAL_POST_INSTALL_CMD that is aligned with the directory structure for the vendor folder that is created from the scripts.

Not sure how many bits are broken when these files are missing from builds.

I'm getting an issue in which the vendor generation keeps failing.

I have tried your versions over the last few days and still get the same error.

Using the angler NRD90U firmware i get down to this;

basename: invalid option -- '>'
Try 'basename --help' for more information.
[-] Vendor generation failed

is this an issue that is to surpassed? is there anything further required to assist?
This issue occurs whether -g flag used or not.

Thanks

Nougat AOSP sources indicate that some components have dependencies with proprietary shared objects that are not available from Google. Such an example is bullhead loc_api_V02 (https://android.googlesource.com/device/lge/bullhead/+/android-7.0.0_r1/location/loc_api/loc_api_v02/Android.mk#20)

As such we can no longer include the shared objects that are extracted from factory images as part of the PRODUCT_COPY_FILES option. We need to create a separate target for at least all shared objects that are dependencies of AOSP components. Since we can't easily identify those we need to upgrade the toolchain to create a separate target for all shared objects that need to be included from factory images.

The format should be something like this:

include $(CLEAR_VARS)
LOCAL_MODULE := libqmi_cci
LOCAL_SRC_FILES := vendor/lib64/libqmi_cci.so
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/$(TARGET_COPY_OUT_VENDOR)/lib64
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_OWNER := lge
include $(BUILD_PREBUILT)

We also need to identify a compatible way to deal with MULTILIB scenarios. Probably search if 32bit version is also available and set the LOCAL_SRC_FILES_32 flag.

This seems relevant to android-prepare-vendor, since it goes through a lot of pain to get a proper build of vendor.img where verity can be used:

• https://android.googlesource.com/device/lge/bullhead/+/7fecf8ad27e41fc1ccfc586bac8ffe1c66016873
• https://android.googlesource.com/device/huawei/angler/+/4ff4fb8d3ceb14e1c10811acb65c8feafa6347e1

Not sure if you can override it, but you might want to document it.

I wonder if there's any chance of landing the vendor reassembly scripts upstream. Their blob list already expects you to bundle a stock vendor.img. If they actually moved everything they should to vendor, they could support a sane build process again with only that.

Back in the Marshmallow days it was a nice gift the way SmaliEx handled the baksmali -> smali iterations to de-odex and reassemble original DEX files. Unfortunately it seems that project is not keeping up with the all the smali & dexlib changes the way & with the speed we want.

This issue is to track down progress of switching from SmaliEx to smali while manually dealing with error handling for baksmali / smali iterations.

Hey,

just a little note: I used your repo within a virtual machine running Ubuntu 16.04 x64 and experienced a crash because of a too small configured RAM (only 2GB).

Maybe you could advise to have at least 3-4 GB of RAM.

Best,

Huppys

AOSP build system seems to support "LOCAL_PREBUILT_MODULE_FILE" for package makefiles (including APPS & JARS). Investigate if we can use this flag to include pre-optimized bytecode as extracted from factory images.

The drawback with this approach is that we'll have to enable DEX_PREOPT (for at least framework jars) to test if external pre-opts can be included.

Also since the boot.oat & ART image that ship with factory images will not match the offsets of the one we build from AOSP we need to double check that included odex can be repaired with patch2oat at first boot, or worst case scenario patch2oat understands that there is something sketchy and recompile from scratch. If runtime is good enough to catch this case we might be able to also use "DONT_DEXPREOPT_PREBUILTS" on top of "LOCAL_PREBUILT_MODULE_FILE" so that we don't need to pre-optimize framework jars and runtime will repair us at first boot.

It's a long shot, although it might be worth trying since the while de-optimization process is very fragile (see Android N issues with smali) and expensive (lots of time spent de-optimizing app/jars).

Nexus 6P has these symlinks within vendor and at the moment only the 2 for the app libraries are included:

vendor/app/ims/lib/arm64/libimscamera_jni.so
vendor/app/ims/lib/arm64/libimsmedia_jni.so
vendor/etc/firmware/wcd9320/wcd9320_anc.bin
vendor/etc/firmware/wcd9320/wcd9320_mad_audio.bin
vendor/etc/firmware/wcd9320/wcd9320_mbhc.bin
vendor/lib64/eglSubDriverAndroid.so
vendor/lib64/libEGL_adreno.so
vendor/lib64/libGLESv1_CM_adreno.so
vendor/lib64/libGLESv2_adreno.so
vendor/lib64/libq3dtools_adreno.so
vendor/lib64/libq3dtools_esx.so

Hi,

This is not an issue, but rather a question.

The execution printed the following lines:

…
[*] '/priv-app/PrebuiltGmsCore/PrebuiltGmsCore.apk' is multi-dex - adjusting recursive archive adds
[*] '/priv-app/GoogleServicesFramework/GoogleServicesFramework.apk' not pre-optimized with sanity checks passed - copying without changes
[*] '/app/Music2/Music2.apk' is multi-dex - adjusting recursive archive adds
[*] '/app/Photos/Photos.apk' is multi-dex - adjusting recursive archive adds
[-] '/app/PrebuiltExchange3Google/PrebuiltExchange3Google.apk' de-optimization failed consider manual inspection - skipping archive
[*] '/app/Maps/Maps.apk' is multi-dex - adjusting recursive archive adds
…

Does this mean that Google proprietary stuff will be included?

These apks are copied to out/bullhead/mhc19q/factory_imgs_repaired_data/system/priv-app, but not to out/bullhead/mhc19q/vendor/lge/bullhead. so I think the answer is no, but I prefer asking. I don't want google stuff in my build.

A more general question: is the proprietary stuff extracted for the AOSP build the minimal set to make the device work correctly?

I just used this tool for my Nexus 5X AOSP project and I noticed that the file downloaded from google's servers (factory image) ends up with .zip extension, so maybe that should be covered in the script?

Hi,

Thanks for your work.

However, execution fails:

# ./execute-all.sh -d bullhead -b MHC19Q -i bullhead-mhc19q-factory-22ba5e5a.tgz -o out
[*] Setting output base to 'out/bullhead/mhc19q'
[*] Extracting 'bullhead-mhc19q-factory-22ba5e5a.tgz'
[*] Unzipping 'image-bullhead-mhc19q.zip'
[*] Copying files from system parition ...
[*] Copying files from vendor partition ...
[*] Preparing environment for 'arm' ABI
[!] Boot classes extraction failed
[-] System partition de-optimization failed

It doesn't work at all if oat2dex.jar is not updated to the latest version (compiled directly from git tree)

sudo ./execute-all.sh -d bullhead -b npd56n -o tmp -i ~/Downloads/bullhead-npd56n-factory-996cac57.tgz
[sudo] password for danield:
[] Setting output base to 'tmp/bullhead/npd56n'
[] Extracting 'bullhead-npd56n-factory-996cac57.tgz'
[] Unzipping 'image-bullhead-npd56n.zip'
[] Copying files from system partition ...
[] Copying files from vendor partition ...
[] Preparing environment for 'arm' ABI
[] Preparing environment for 'arm64' ABI
[] Start extracting system partition & de-optimize pre-compiled bytecode ...
[-] '/priv-app/GoogleDialer/GoogleDialer.apk' de-optimization failed consider manual inspection - skipping archive
[] '/priv-app/PrebuiltGmsCore/PrebuiltGmsCore.apk' not pre-optimized with sanity checks passed - copying without changes
[-] '/priv-app/CNEService/CNEService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ExternalStorageProvider/ExternalStorageProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/CalendarProvider/CalendarProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Shell/Shell.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GooglePartnerSetup/GooglePartnerSetup.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GCS/GCS.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ConnMO/ConnMO.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Phonesky/Phonesky.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/DMService/DMService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/TeleService/TeleService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/DCMO/DCMO.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/DiagMon/DiagMon.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/CellBroadcastReceiver/CellBroadcastReceiver.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/TelephonyProvider/TelephonyProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/BlockedNumberProvider/BlockedNumberProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/SprintDM/SprintDM.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/SetupWizard/SetupWizard.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GooglePackageInstaller/GooglePackageInstaller.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/EmergencyInfo/EmergencyInfo.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ConnMetrics/ConnMetrics.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/MtpDocumentsProvider/MtpDocumentsProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Settings/Settings.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/SettingsProvider/SettingsProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ConfigUpdater/ConfigUpdater.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/MediaProvider/MediaProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/WallpaperCropper/WallpaperCropper.apk' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/priv-app/CtsShimPrivPrebuilt/CtsShimPrivPrebuilt.apk' not pre-optimized & without 'classes.dex' - skipping
[-] '/priv-app/CarrierEntitlement/CarrierEntitlement.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Velvet/Velvet.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GoogleLoginService/GoogleLoginService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ContactsProvider/ContactsProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/HotwordEnrollment/HotwordEnrollment.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GoogleContacts/GoogleContacts.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/DownloadProvider/DownloadProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GoogleOneTimeInitializer/GoogleOneTimeInitializer.apk' de-optimization failed consider manual inspection - skipping archive
[] '/priv-app/GoogleServicesFramework/GoogleServicesFramework.apk' not pre-optimized with sanity checks passed - copying without changes
[-] '/priv-app/StatementService/StatementService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/ManagedProvisioning/ManagedProvisioning.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/SystemUI/SystemUI.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/TagGoogle/TagGoogle.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/LifeTimerService/LifeTimerService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Telecom/Telecom.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/MmsService/MmsService.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/Entitlement/Entitlement.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/DocumentsUI/DocumentsUI.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/priv-app/GoogleFeedback/GoogleFeedback.apk' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/ext.jar' not pre-optimized & without 'classes.dex' - skipping
[-] '/framework/com.google.android.media.effects.jar' de-optimization failed consider manual inspection - skipping archive
[-] '/framework/services.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/okhttp.jar' not pre-optimized & without 'classes.dex' - skipping
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/org.apache.http.legacy.boot.jar' not pre-optimized & without 'classes.dex' - skipping
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/telephony-common.jar' not pre-optimized & without 'classes.dex' - skipping
[] Skipping framework-res since it doesn't pair with bytecode
[-] '/framework/android.test.runner.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/ims-common.jar' not pre-optimized & without 'classes.dex' - skipping
[-] '/framework/com.google.android.maps.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/voip-common.jar' not pre-optimized & without 'classes.dex' - skipping
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/core-libart.jar' not pre-optimized & without 'classes.dex' - skipping
[-] '/framework/pm.jar' de-optimization failed consider manual inspection - skipping archive
[] '/framework/rcsservice.jar' not pre-optimized with sanity checks passed - copying without changes
[-] '/framework/javax.obex.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/bouncycastle.jar' not pre-optimized & without 'classes.dex' - skipping
[] '/framework/rcsimssettings.jar' not pre-optimized with sanity checks passed - copying without changes
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/core-junit.jar' not pre-optimized & without 'classes.dex' - skipping
[-] '/framework/uiautomator.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/framework.jar' not pre-optimized & without 'classes.dex' - skipping
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/apache-xml.jar' not pre-optimized & without 'classes.dex' - skipping
[] '/framework/cneapiclient.jar' not pre-optimized with sanity checks passed - copying without changes
[-] '/framework/monkey.jar' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/core-oj.jar' not pre-optimized & without 'classes.dex' - skipping
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/conscrypt.jar' not pre-optimized & without 'classes.dex' - skipping
[-] '/framework/wifi-service.jar' de-optimization failed consider manual inspection - skipping archive
[-] '/app/BasicDreams/BasicDreams.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleHome/GoogleHome.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleContactsSyncAdapter/GoogleContactsSyncAdapter.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/UserDictionaryProvider/UserDictionaryProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Maps/Maps.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/RCSBootstraputil/RCSBootstraputil.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Photos/Photos.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PrebuiltExchange3Google/PrebuiltExchange3Google.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleCamera/GoogleCamera.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Hangouts/Hangouts.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/CalculatorGooglePrebuilt/CalculatorGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/YadaYada/YadaYada.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/YouTube/YouTube.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/talkback/talkback.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Tycho/Tycho.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PartnerBookmarksProvider/PartnerBookmarksProvider.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PlayGames/PlayGames.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/FaceLock/FaceLock.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleJapaneseInput/GoogleJapaneseInput.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleVrCore/GoogleVrCore.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleHindiIME/GoogleHindiIME.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PrebuiltDeskClockGoogle/PrebuiltDeskClockGoogle.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/KoreanIME/KoreanIME.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/HiddenMenu/HiddenMenu.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/HTMLViewer/HTMLViewer.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/CalendarGooglePrebuilt/CalendarGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Chrome/Chrome.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Wallet/Wallet.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/NfcNci/NfcNci.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GooglePinyinIME/GooglePinyinIME.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PrebuiltGmail/PrebuiltGmail.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/GoogleTTS/GoogleTTS.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/WebViewGoogle/WebViewGoogle.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/CloudPrint2/CloudPrint2.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/RcsImsBootstraputil/RcsImsBootstraputil.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Bluetooth/Bluetooth.apk' de-optimization failed consider manual inspection - skipping archive
[-] 'tmp/bullhead/npd56n/factory_imgs_data/system/app/CtsShimPrebuilt/CtsShimPrebuilt.apk' not pre-optimized & without 'classes.dex' - skipping
[-] '/app/PrebuiltBugle/PrebuiltBugle.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Videos/Videos.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Drive/Drive.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/DMAgent/DMAgent.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Music2/Music2.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/LatinIMEGooglePrebuilt/LatinIMEGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/PrintSpooler/PrintSpooler.apk' de-optimization failed consider manual inspection - skipping archive
[-] '/app/Stk/Stk.apk' de-optimization failed consider manual inspection - skipping archive
[] System partition successfully extracted & de-optimized at 'tmp/bullhead/npd56n/factory_imgs_repaired_data'
[!] Target device expects to have following img versions when using output system img
[] Booatloder:BHZ11d
[] Baseband:M8994F-2.6.33.2.08
[] Generating blobs for vendor/lge/bullhead
[*] Copying files to 'tmp/bullhead/npd56n/vendor/lge/bullhead' ...
cp: cannot stat ‘tmp/bullhead/npd56n/factory_imgs_repaired_data/system/app/atfwd/atfwd.apk’: No such file or directory
[-] Vendor generation failed

Me again, I had an issue in this week, basically (I don't know exactly with which update, but was with NBD90W's vendor files) the generated otapackage.zip can't be flashed in twrp, when installing starts it gets stopped almost immediately with "Updater process ended with error 6" (and no explaination of what)
So I tried to manually edit the updater-script, checked EOLs, repaced with Unix's ones, reflashed, still error 6.
This week i tried to extract from NRD91N's factory, and still the same error. Finally I tried importing into the aosp old NRD90S's vendor, that was extracted from factory in September(almost) compiled (after a make installclean) then I tried to flash the generated .zip and this time it worked, but I can't understand why.

I made another build of aosp 7.0.0_r6, after i downloaded from the master branch and gave -g option to execute-all.sh. without flashing gapps i'm able to see google's project-fi, when i flash opengapps (haven't tried other gapps) my bullhead just bootloops, i'm leaving a logcat, it might be helpful.

logcat_bootloop_1.txt

TBD

Pulled down the latest code on master branch (although had the same issue with master-dev before your merge with master), here is the log with the error:
./execute-all.sh -d bullhead -b NRD90S -o nexus5x_nougat
[] Setting output base to 'nexus5x_nougat/bullhead/nrd90s'
[] Downloading image from 'https://dl.google.com/dl/android/aosp/bullhead-nrd90s-factory-d6bf1f56.zip'
--2016-09-12 09:33:34-- https://dl.google.com/dl/android/aosp/bullhead-nrd90s-factory-d6bf1f56.zip
Resolving dl.google.com (dl.google.com)... 172.217.19.110, 2a00:1450:4016:804::200e
Connecting to dl.google.com (dl.google.com)|172.217.19.110|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1243254222 (1,2G) [application/zip]
Saving to: ‘nexus5x_nougat/bullhead/nrd90s/bullhead-nrd90s-factory-d6bf1f56.zip’

100%[====================================================================================================================================================================>] 1.243.254.222 410KB/s in 38m 46s

2016-09-12 10:12:20 (522 KB/s) - ‘nexus5x_nougat/bullhead/nrd90s/bullhead-nrd90s-factory-d6bf1f56.zip’ saved [1243254222/1243254222]

[] Processing with 'API-24' configuration
[] Extracting 'nexus5x_nougat/bullhead/nrd90s/bullhead-nrd90s-factory-d6bf1f56.zip'
[*] Unzipping 'image-bullhead-nrd90s.zip'
[-] Invalid input directory structure
Usage: gen-prop-blobs-list.sh [options]
OPTIONS:
-i|--input : Root path of /vendor partition
-o|--output : Path to save generated "proprietary-blobs.txt" file
--sys-list : File list with proprietary blobs in /system partition
--bytecode-list : File list with proprietary bytecode archive files
--dep-dso-list : File list with DSO files with individual targets
[-] 'proprietary-blobs.txt' generation failed

I tried with and without sudo just to be sure, but no luck. I will be available to perform any additional tests you ask :)

Please fix this....
This is the result:
Input:
android-prepare-vendor$ ./execute-all.sh -d bullhead -i /fast-datavault/nexus-vendor-blobs/bullhead/mtc20k/bullhead-mtc20k-factory-4a950470.zip -b mtc20k -o /fast-datavault/nexus-vendor-blobs
output:
[-] 'fusermount' command not found

Bytecode de-optimization util is using baksmali / smali tools, which currently don't support Android N ART changes (JesusFreke/smali#378).

The second (although minor) problem is that old "boot.oat" is now split into multiple files prefixed with boot-XXX.oat and having their own ART image file. We'll probably have to deal with it manually, unless SmaliEx maintainers quickly pick up this and implement it.

Support for Android N is blocked until smali project (and then by extend SmaliEx which we actually use to simplify baksmali -> smali iterations) supports recent ART changes.

java -jar /aosp_b_prod/prepare_vendor_blobs/hostTools/Java/baksmali.jar -x -o $(pwd)/deopt -c "$(find boot64 -name "*.oat" -printf "%f\n" | while read -r file; do echo -ne "$file:"; done)" -d $(pwd)/boot64 atfwd.odex


Error occurred while loading boot class path files. Aborting.
java.lang.StringIndexOutOfBoundsException: String index out of range: 1599362287
        at java.lang.String.checkBounds(String.java:385)
        at java.lang.String.<init>(String.java:462)
        at org.jf.dexlib2.dexbacked.OatFile$1$1.next(OatFile.java:172)
        at org.jf.dexlib2.dexbacked.OatFile$1$1.next(OatFile.java:159)
        at org.jf.dexlib2.analysis.ClassPath.fromClassPath(ClassPath.java:191)
        at org.jf.baksmali.baksmali.disassembleDexFile(baksmali.java:67)
        at org.jf.baksmali.main.run(main.java:113)
        at org.jf.baksmali.main.main(main.java:322)

Error:
/home/lucent/android-prepare-vendor/scripts/generate-vendor.sh: Line 483: PKGS_SSLINKS: unbound variable
Cause:
PKGS_SSLINKS is an empty variable(no packages are added to the variable)

Fix:
Test whether PKGS_SSLINKS is empty before using it.

Error can be recreated with this command:
sudo ./execute-all.sh -d bullhead -b MTC20K -o ../ -g -y --smaliex

Environment:

• elementary OS 0.4 Loki (based on Ubuntu 16.04 / xenial)
• GNU bash, version 4.3.46(1)-release (x86_64-pc-linux-gnu)

I'm not sure what changed to impact this but the marlin/sailfish symlinks are being added as modules but not to PRODUCT_PACKAGES.

Hi again,

I generated vendor using your tool, and built android-6.0.1_r24 for Nexus 5X.

The build was OK.

On boot, it prints There's an internal problem with your device. Contact your manufacturer for details.
This is a known problem that is easily resolved by using the same fingerprint in /system/build.prop and /vendor/build.prop but maybe it could be automated.

I have another problem, an alert Exchange Services stopped.

04-18 22:53:42.440  5051  5065 I ActivityManager: Start proc 8078:com.android.exchange/u0a32 for service com.android.exchange/.service.EasService
04-18 22:53:42.476  8078  8078 W System  : ClassLoader referenced unknown path: /system/app/Exchange2/lib/arm64
04-18 22:53:42.482  8078  8078 D AndroidRuntime: Shutting down VM
04-18 22:53:42.483  8078  8078 E AndroidRuntime: FATAL EXCEPTION: main
04-18 22:53:42.483  8078  8078 E AndroidRuntime: Process: com.android.exchange, PID: 8078
04-18 22:53:42.483  8078  8078 E AndroidRuntime: java.lang.RuntimeException: Unable to instantiate application com.android.exchange.Exchange: java.lang.ClassNotFoundException: Didn't find class "com.android.exchange.Exchange" on path: DexPathList[[zip file "/system/app/Exchange2/Exchange2.apk"],nativeLibraryDirectories=[/system/app/Exchange2/lib/arm64, /vendor/lib64, /system/lib64]]
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.LoadedApk.makeApplication(LoadedApk.java:578)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4685)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.ActivityThread.-wrap1(ActivityThread.java)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1405)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.os.Handler.dispatchMessage(Handler.java:102)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.os.Looper.loop(Looper.java:148)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.ActivityThread.main(ActivityThread.java:5422)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at java.lang.reflect.Method.invoke(Native Method)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
04-18 22:53:42.483  8078  8078 E AndroidRuntime: Caused by: java.lang.ClassNotFoundException: Didn't find class "com.android.exchange.Exchange" on path: DexPathList[[zip file "/system/app/Exchange2/Exchange2.apk"],nativeLibraryDirectories=[/system/app/Exchange2/lib/arm64, /vendor/lib64, /system/lib64]]
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.Instrumentation.newApplication(Instrumentation.java:981)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    at android.app.LoadedApk.makeApplication(LoadedApk.java:573)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    ... 9 more
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    Suppressed: java.io.IOException: No original dex files found for dex location /system/app/Exchange2/Exchange2.apk
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexFile.openDexFileNative(Native Method)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexFile.openDexFile(DexFile.java:295)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexFile.<init>(DexFile.java:80)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexFile.<init>(DexFile.java:59)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexPathList.loadDexFile(DexPathList.java:279)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexPathList.makePathElements(DexPathList.java:248)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.DexPathList.<init>(DexPathList.java:120)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.BaseDexClassLoader.<init>(BaseDexClassLoader.java:48)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at dalvik.system.PathClassLoader.<init>(PathClassLoader.java:65)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at android.app.ApplicationLoaders.getClassLoader(ApplicationLoaders.java:58)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at android.app.LoadedApk.getClassLoader(LoadedApk.java:376)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at android.app.LoadedApk.makeApplication(LoadedApk.java:568)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        ... 9 more
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    Suppressed: java.lang.ClassNotFoundException: com.android.exchange.Exchange
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at java.lang.Class.classForName(Native Method)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at java.lang.BootClassLoader.findClass(ClassLoader.java:781)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at java.lang.BootClassLoader.loadClass(ClassLoader.java:841)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        at java.lang.ClassLoader.loadClass(ClassLoader.java:504)
04-18 22:53:42.483  8078  8078 E AndroidRuntime:        ... 12 more
04-18 22:53:42.483  8078  8078 E AndroidRuntime:    Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack trace available

Do you know if it's related to this tool?

Before using it, I had SIM problems (now it works!) but no "Exchange Services" problem.

Note that I removed the priv-app from out/bullhead/mhc19q/vendor/lge/bullhead/proprietary/priv-app, but I don't think it's related.

Me again, used your script today to generate files and worked like a charm from start till the end. I have a problem not strictly related to your script but maybe you could help me: building the ROM goes smooth and after flashing I get that 'There's an internal problem bla bla' warning. It turns out that my ro.vendor.build.fingerprint and my ro.build.fingerprint are different. Example:
ro.build.fingerprint=Android/aosp_bullhead/bullhead:7.0/NBD90W/mikala10101738:userdebug/test-keys
ro.vendor.build.fingerprint=Android/aosp_bullhead/bullhead:7.0/NBD90W/mikala10101936:userdebug/test-keys

I didnt touch any of the build scripts from AOSP tree and I didnt have this before on previous nougat release. Do you maybe know what is the problem? On your tests, do you get the same fingerprint for both?

Thanks in advance, I appreciate your help!

I just checked out binaries page on google (https://developers.google.com/android/drivers) and it turns out there are binaries now for 5x and 6p. Im gonna try and and make a build with these, anyone else had a shot with these yet?

I'm not sure if this problem will be reproducible for you, but it happens consistently here on Arch Linux. I can look into it deeper but for now I think I'll just edit the generated output to fix it.

I end up with libdmengine being incorrectly referenced like this:

$ ls /system/priv-app/DMService/lib/arm/ -l                     
lrw-r--r-- root     root              2009-01-01 00:00 libdmengine.so -> /system/lib/lib/libdmengine.so
lrw-r--r-- root     root              2009-01-01 00:00 libdmjavaplugin.so -> /system/lib/libdmjavaplugin.so

It can be seen in the generated Android.mk that the libdmjavaplugin.so library is fine, but libdmengine.so is not:

include $(CLEAR_VARS)
LOCAL_MODULE := libdmjavaplugin_32.so
LOCAL_MODULE_CLASS := FAKE
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_OWNER := huawei
include $(BUILD_SYSTEM)/base_rules.mk
$(LOCAL_BUILT_MODULE): TARGET := /system/lib/libdmjavaplugin.so
$(LOCAL_BUILT_MODULE): SYMLINK := $(PRODUCT_OUT)/$(TARGET_COPY_OUT_SYSTEM)/priv-app/DMService/lib/arm/libdmjavaplugin.so
$(LOCAL_BUILT_MODULE): $(LOCAL_PATH)/Android.mk
$(LOCAL_BUILT_MODULE):
        $(hide) mkdir -p $(dir $@)
        $(hide) mkdir -p $(dir $(SYMLINK))
        $(hide) rm -rf $@
        $(hide) rm -rf $(SYMLINK)
        $(hide) ln -sf $(TARGET) $(SYMLINK)
        $(hide) touch $@

include $(CLEAR_VARS)
LOCAL_MODULE := libdmengine_32.so
LOCAL_MODULE_CLASS := FAKE
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_OWNER := huawei
include $(BUILD_SYSTEM)/base_rules.mk
$(LOCAL_BUILT_MODULE): TARGET := /system/lib/lib/libdmengine.so
$(LOCAL_BUILT_MODULE): SYMLINK := $(PRODUCT_OUT)/$(TARGET_COPY_OUT_SYSTEM)/priv-app/DMService/lib/arm/libdmengine.so
$(LOCAL_BUILT_MODULE): $(LOCAL_PATH)/Android.mk
$(LOCAL_BUILT_MODULE):
        $(hide) mkdir -p $(dir $@)
        $(hide) mkdir -p $(dir $(SYMLINK))                                                                                                  
        $(hide) rm -rf $@
        $(hide) rm -rf $(SYMLINK)
        $(hide) ln -sf $(TARGET) $(SYMLINK)
        $(hide) touch $@

Hi,

I have tried to re-generate vendors for the two new Google Pixels :

$ PATH=/home/tgayet/_aosp-git-repos/android-prepare-vendor/_tmp-jdk/jdk1.8.0_101/bin:$PATH ; ./execute-all.sh --device sailfish --buildID NDE63V --output $(pwd) --yes

[-] 'sailfish' is not supported
I wanna know if you have ever tried to rebuild vendors for the new pixel (pixel and pixel XL : marlin and sailfish) ?

I have both mobile on my side.

What do you need to investigate in order to add the two support for your script ?

"marlin" for Pixel XL

Version Download SHA-256 Checksum
7.1.0 (NDE63H, Oct 2016) Link 96adcecf136d0389569e28a3396eba92880d77df3ceb8e39b0a765d99ddf0929
7.1.0 (NDE63L, Oct 2016) Link 71a65e7ef49f88a8f376831378984c019e8f078dbd39499eb4ca9e1a475df9f0
7.1.0 (NDE63P, Oct 2016) Link dcdaaa51d4b4c0a67c4b670f76fe53b852b38d7b2ce65399ac56799bfc1ca7bb
7.1.0 (Europe, NDE63U, Nov 2016) Link 24e8abeb4e1f09bff31577b46661110bf7d5745abb9269b638f6fb86a338d271
7.1.0 (NDE63V, Nov 2016) Link a66866ba4b8f9f1baa856828d46a747d769098add86098bece50854983ecea3e
7.1.0 (Verizon, NDE63X, Nov 2016) Link 8b520419e32141f68d9f12d7e302bfb77885ed5587908d30b2a0e94ed07038a7

"sailfish" for Pixel

Version Download SHA-256 Checksum
7.1.0 (NDE63H, Oct 2016) Link 43ba5f8130a58770edce6cb0c6c43920798e038cd7cdec2c54f22a1dd474a7c8
7.1.0 (NDE63L, Oct 2016) Link 49876a7c9e8f14da672dad27a0e404cccbb1330a4c9ebd398c34583a576eba6c
7.1.0 (NDE63P, Oct 2016) Link aaaa2f1a592ba43de10396938220c4ed65ff3341edd01afd4b5a4f506b39042c
7.1.0 (Europe, NDE63U, Nov 2016) Link 3fa367096c251cd5ab1a6c465cb60fc075b08b999148bbf0036f1533e3efd364
7.1.0 (NDE63V, Nov 2016) Link 1b3155ff623e8a872a788c54e650a0077175f54d723a185f52fff8a3170478ff
7.1.0 (Verizon, NDE63X, Nov 2016) Link dc45f6c83547266bf9e49edeae0978c3d734278febec2f7023aae4bf299e4fec

https://developers.google.com/android/images

Thanks in advance for your reply.

BR
Thierry

Native library resources from pre-optimized APKs at factory images can be any of the following:

• Symbolic links to /system/lib{64} paths: These are effectively system libraries also exposing JNI functionality utilized by some APKs.
• Self-contained DSO that were extracted in subdirectories (lib/arm{64}) when APK was pre-optimized.

Currently scripts automatically deal with the first, although not second case. As such manual work needs to be done at the system-proprietary-blobs-apiXX.txt config file in order to get copied across.

However we want to automate this process so that libraries are automatically included if APK owner is marked for repair.

I haven't looked into this in detail yet but the files are definitely still present with stripped apks alongside the oat files as before.

This project is heavily relying to SmaliEx (baksmali/smali wrapper) in order to de-optimize bytecode targets (APKs/JARs) that we want extract from factory system images as part of the vendor proprietary blobs.

While this approach was relatively working until now, it heavily depends into smali & smaliEx maintainers being always up to speed with all recent ART runtime changes (#20, #18). Unfortunately this is practically not the case since teams need a decent amount of time to adjust to Google changes. Plust we need to do a lot of porting / glueing work to upgrade script to match these changes. Considering how often such changes happen in the Android ecosystem, I was looking for an alternative workaround.

Back in Feb 2015 that I've upstreamed oatdump++ to AOSP (https://android-review.googlesource.com/#/c/134380/), I've pushed a DEX export functionality to dump bytecode from .rodata section of the OAT file. Of course the exported DEX bytecode is not the actual original since the DEX-to-DEX transformations have already been applied (https://github.com/anestisb/oatdump_plus#dex-to-dex-optimisations).

However, to my great surprise when I was researching some other aspects of the dex2oat compiler backends, I've noticed that current implementation front-end is not aborting when DEX-to-DEX transformations have been partially or fully applied already to input. Instead the compiler overrides them when it's not happy with the VTABLE indexes. This makes more sense if someone considers the ART profiling functionality where pre-optimized code maybe need to be recompiled without having the original DEX.

This effectively means that the bytecode we want to repair from /system partition can be extracted with oatdump host tool, re-inserted back to APK/JAR and re-compile (pre-optimize) when building from AOSP.

Theoretically dex2oat can happen either at host (pre-optimize) or directly at target when first boot. Until we're confident that this approach works for all bytecode we want to repair for supported devices, we choose to always pre-optimize so potential errors can be spotted as early as possible.

The execution flow of the repairs is:

1. oatdump to dump bytecode from OAT ELF
2. dexrepair (https://github.com/anestisb/dexRepair) to repair DEX CRC checksum
3. Rename bytecode to match naming conventions (classes.dex, classes2.dex, etc.)
4. jar to append bytecode back to source APK/JAR

• zipalign & resign steps are handled automatically from AOSP build chain.

This new method is a work in progress and still needs lots of testing, however so far results (manual repair bytecode samples for N5x target) seems quite promising.

Hi guys, thanks for this awesome project.
It'd be nice to have a flag for the execute-all.sh to keep the META-INF in the final deodexed apk, it's been useful at least for me today to quickly update angler vendor blobs with the latest February factory image.

Bye and thanks again.

It appears that fuse-ext2 (or OSXFUSE) has a bug when running in latest macOS sierra resulting into missing entries when mounting raw images generated from simg2img.

 diff -u mac_mtc20k.txt linux_mtc20k.txt
--- mac_mtc20k.txt	2017-01-04 20:37:16.000000000 +0200
+++ linux_mtc20k.txt	2017-01-04 20:37:24.000000000 +0200
@@ -3,7 +3,13 @@
 bullhead/mtc20k/factory_imgs_data/system/app/Bluetooth/Bluetooth.apk:9260d0fb26c50ffb3da6e4069cc0e076
 bullhead/mtc20k/factory_imgs_data/system/app/Bluetooth/oat/arm/Bluetooth.odex:da52b4290679e2159c5ec6b6214c4c87
 bullhead/mtc20k/factory_imgs_data/system/app/BluetoothMidiService/BluetoothMidiService.apk:91073d348f48f85c3a307c8166afa8bf
+bullhead/mtc20k/factory_imgs_data/system/app/BluetoothMidiService/oat/arm64/BluetoothMidiService.odex:4530a1517c06b6a0023cbc6542adb629
+bullhead/mtc20k/factory_imgs_data/system/app/BullheadLayout/BullheadLayout.apk:972b77d8fabefa9aeb29986b0b7f0cd9
+bullhead/mtc20k/factory_imgs_data/system/app/BullheadLayout/oat/arm64/BullheadLayout.odex:877862149469b58a9fc93d20f663ad30
 bullhead/mtc20k/factory_imgs_data/system/app/CalculatorGoogle/CalculatorGoogle.apk:345130b0cb4d3d970bae2327cb92bd72
+bullhead/mtc20k/factory_imgs_data/system/app/CalculatorGoogle/oat/arm64/CalculatorGoogle.odex:a145a9a2d2a06ce150164dca0b8ff889
+bullhead/mtc20k/factory_imgs_data/system/app/CalendarGooglePrebuilt/CalendarGooglePrebuilt.apk:57c0f2b401d27b0ce8ee65e063ce67c8
+bullhead/mtc20k/factory_imgs_data/system/app/CalendarGooglePrebuilt/oat/arm64/CalendarGooglePrebuilt.odex:87f6974202332bcd1f7760f72d45a164
 bullhead/mtc20k/factory_imgs_data/system/app/CaptivePortalLogin/CaptivePortalLogin.apk:d813a4369791e8f7d2c70fc835251a9d
 bullhead/mtc20k/factory_imgs_data/system/app/CaptivePortalLogin/oat/arm64/CaptivePortalLogin.odex:2464de47c7d45f983b19c910f8e0330c
 bullhead/mtc20k/factory_imgs_data/system/app/CertInstaller/CertInstaller.apk:7aacd43b3d1aa685b6512ac9ccb0dd7b
@@ -22,6 +28,10 @@
 bullhead/mtc20k/factory_imgs_data/system/app/Drive/oat/arm64/Drive.odex:7284406140576278410fcb5ff386fffe
 bullhead/mtc20k/factory_imgs_data/system/app/FaceLock/FaceLock.apk:1b6ca424757c862222661a72df8a65b1
 bullhead/mtc20k/factory_imgs_data/system/app/FaceLock/oat/arm64/FaceLock.odex:d73f36cd52f87e086a5c207475dadad0
+bullhead/mtc20k/factory_imgs_data/system/app/GoogleCamera/GoogleCamera.apk:df17d60a00d9532b12327cbe5f8007b7
+bullhead/mtc20k/factory_imgs_data/system/app/GoogleCamera/oat/arm64/GoogleCamera.odex:52961de2ff049b6f523680fde62af739
+bullhead/mtc20k/factory_imgs_data/system/app/GoogleContactsSyncAdapter/GoogleContactsSyncAdapter.apk:b99ededa6897f8ebd54eee652be710b0
+bullhead/mtc20k/factory_imgs_data/system/app/GoogleContactsSyncAdapter/oat/arm64/GoogleContactsSyncAdapter.odex:53bcba9825a42fb1c938af62f70c1dfe
 bullhead/mtc20k/factory_imgs_data/system/app/GoogleHindiIME/GoogleHindiIME.apk:033ab854fe0c07a3fc2ea0d3bffc4c93
 bullhead/mtc20k/factory_imgs_data/system/app/GoogleHindiIME/oat/arm64/GoogleHindiIME.odex:609bd8e75f620ba301b649cbb35fe835
 bullhead/mtc20k/factory_imgs_data/system/app/GoogleHome/GoogleHome.apk:0b3a06e3e003bd140b0ec8525ad60c08
@@ -35,14 +45,29 @@
 bullhead/mtc20k/factory_imgs_data/system/app/HTMLViewer/HTMLViewer.apk:c54264b488461304e22349cb171c4f43
 bullhead/mtc20k/factory_imgs_data/system/app/HTMLViewer/oat/arm64/HTMLViewer.odex:83487ea0307c92c0683500b658b87706
 bullhead/mtc20k/factory_imgs_data/system/app/Hangouts/Hangouts.apk:0101359b4a10468bd8ce5e649b208389
+bullhead/mtc20k/factory_imgs_data/system/app/Hangouts/oat/arm/Hangouts.odex:6b77481a562b1575e4c5d4d6083d55a1
+bullhead/mtc20k/factory_imgs_data/system/app/HiddenMenu/HiddenMenu.apk:a50731a5ab478950a7d3156eb667204f
+bullhead/mtc20k/factory_imgs_data/system/app/HiddenMenu/oat/arm64/HiddenMenu.odex:e5fd1b68b799a360d42c382cc238ea7c
 bullhead/mtc20k/factory_imgs_data/system/app/KeyChain/KeyChain.apk:eaca5540bb6e2641bc84cad3645042e2
+bullhead/mtc20k/factory_imgs_data/system/app/KeyChain/oat/arm64/KeyChain.odex:1944e27cfa31628cc07258de2f96f585
+bullhead/mtc20k/factory_imgs_data/system/app/KoreanIME/KoreanIME.apk:3a5ec342bb3c67e3ccbe424063dd7cf8
+bullhead/mtc20k/factory_imgs_data/system/app/KoreanIME/oat/arm64/KoreanIME.odex:cdd61bb0198cd5da7ad23440ec09d8f1
 bullhead/mtc20k/factory_imgs_data/system/app/LatinImeGoogle/LatinImeGoogle.apk:1f9c02b03027b4341765bc8b4805730b
 bullhead/mtc20k/factory_imgs_data/system/app/LatinImeGoogle/oat/arm64/LatinImeGoogle.odex:d13a725625e62444fa9f15b66bccbfe1
 bullhead/mtc20k/factory_imgs_data/system/app/LiveWallpapersPicker/LiveWallpapersPicker.apk:d257afad6612d0b212f7d8e2214c2e5e
+bullhead/mtc20k/factory_imgs_data/system/app/LiveWallpapersPicker/oat/arm64/LiveWallpapersPicker.odex:1d1b27c77ec2153625e6b96bddd7e8fc
+bullhead/mtc20k/factory_imgs_data/system/app/Maps/Maps.apk:7af9c6808885afcbf9bf637ddc2b5b53
+bullhead/mtc20k/factory_imgs_data/system/app/Maps/oat/arm64/Maps.odex:e3fc7f5b67161a8959babd936a6a4198
+bullhead/mtc20k/factory_imgs_data/system/app/Music2/Music2.apk:2c27b51d325f700f5cdc70a29fcc719a
+bullhead/mtc20k/factory_imgs_data/system/app/Music2/oat/arm64/Music2.odex:93b9d3d6fdb4ad8899e1ace61cd042d2
 bullhead/mtc20k/factory_imgs_data/system/app/NfcNci/NfcNci.apk:07fbce22c357f7421d4a8ca07d439c4d
 bullhead/mtc20k/factory_imgs_data/system/app/NfcNci/oat/arm64/NfcNci.odex:df252f4a6e554b1eaf6c598181e6bd7c
 bullhead/mtc20k/factory_imgs_data/system/app/PacProcessor/PacProcessor.apk:7f327097aa51f4b9175080cb672d6707
 bullhead/mtc20k/factory_imgs_data/system/app/PacProcessor/oat/arm64/PacProcessor.odex:cf28fe91df20be6e39cc047d87d0eb8a
+bullhead/mtc20k/factory_imgs_data/system/app/PartnerBookmarksProvider/PartnerBookmarksProvider.apk:6744e3b58a25809310e22160357322d4
+bullhead/mtc20k/factory_imgs_data/system/app/PartnerBookmarksProvider/oat/arm64/PartnerBookmarksProvider.odex:09f83086a12340e6df5ae688ea81dd30
+bullhead/mtc20k/factory_imgs_data/system/app/Photos/Photos.apk:092f2cefe3fc0e2fb5853a337cd7e5fa
+bullhead/mtc20k/factory_imgs_data/system/app/Photos/oat/arm/Photos.odex:2a4a232db130312ce13312c8eb7d3c15
 bullhead/mtc20k/factory_imgs_data/system/app/PlayAutoInstallConfig/PlayAutoInstallConfig.apk:c100f9a4b66da1de8f2d444294b9484a
 bullhead/mtc20k/factory_imgs_data/system/app/PlayAutoInstallConfig/oat/arm/PlayAutoInstallConfig.odex:74c9ed973ea8530f982484c8521e04b5
 bullhead/mtc20k/factory_imgs_data/system/app/PlayAutoInstallConfig/oat/arm64/PlayAutoInstallConfig.odex:dc217da7e5447974c46c045da498ec78
@@ -58,6 +83,10 @@
 bullhead/mtc20k/factory_imgs_data/system/app/PrebuiltGmail/oat/arm64/PrebuiltGmail.odex:9f7dcc10355f825964093dfd9d34b430
 bullhead/mtc20k/factory_imgs_data/system/app/PrintSpooler/PrintSpooler.apk:a03dbfa5cc1557898bc54a2bb5f06729
 bullhead/mtc20k/factory_imgs_data/system/app/PrintSpooler/oat/arm64/PrintSpooler.odex:95d80a2428416dea1022837f45384a71
+bullhead/mtc20k/factory_imgs_data/system/app/RCSBootstraputil/RCSBootstraputil.apk:fd431a92e71618ab380559b6d863b61a
+bullhead/mtc20k/factory_imgs_data/system/app/RCSBootstraputil/oat/arm64/RCSBootstraputil.odex:9746eaf757e23f5b7c59caacf2b132cb
+bullhead/mtc20k/factory_imgs_data/system/app/RcsImsBootstraputil/RcsImsBootstraputil.apk:e9dc9c65bcd2a22881e669458a20a26f
+bullhead/mtc20k/factory_imgs_data/system/app/RcsImsBootstraputil/oat/arm64/RcsImsBootstraputil.odex:a1252868a5999b033ca424cd6a7edcfe
 bullhead/mtc20k/factory_imgs_data/system/app/SetupSmartDeviceOverlay/SetupSmartDeviceOverlay.apk:ab554c944e42d5aeec398254baaa078c
 bullhead/mtc20k/factory_imgs_data/system/app/SetupSmartDeviceOverlay/oat/arm64/SetupSmartDeviceOverlay.odex:3d8a5dfba122875acdf9ae0df2838adf
 bullhead/mtc20k/factory_imgs_data/system/app/Stk/Stk.apk:3d575c24a169f6b0d981cfe8b93eb681

the GURL have been updated :

The old url and currently used in the scripts/download-nexus-image.sh script is : https://developers.google.com/android/nexus/images

The correct url is : https://developers.google.com/android/images

Indeed 'nexus' doesn't appear in the url anymore !

$ ./execute-all.sh --device bullhead --buildID MTC20K --output $(pwd)

[*] Setting output base to '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k'
[-] Image URL not found
[-] Images download failed

By fixing the url the script is working fine again :

$ ./execute-all.sh --device bullhead --buildID MTC20K --output /home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors --yes
[*] Setting output base to '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k'

--{ Google Terms and Conditions
Downloading of the system image and use of the device software is subject to the
Google Terms of Service [1]. By continuing, you agree to the Google Terms of
Service [1] and Privacy Policy [2]. Your downloading of the system image and use
of the device software may also be subject to certain third-party terms of
service, which can be found in Settings > About phone > Legal information, or as
otherwise provided.

[1] https://www.google.com/intl/en/policies/terms/
[2] https://www.google.com/intl/en/policies/privacy/

[?] I have read and agree with the above terms and conditions - ACKNOWLEDGE [y|n]: yes
[*] Downloading image from 'https://dl.google.com/dl/android/aosp/bullhead-mtc20k-factory-4a950470.zip'
--2016-11-03 11:50:05-- https://dl.google.com/dl/android/aosp/bullhead-mtc20k-factory-4a950470.zip
Résolution de dl.google.com (dl.google.com)… 216.58.204.142, 2a00:1450:4007:812::200e
Connexion à dl.google.com (dl.google.com)|216.58.204.142|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 964936536 (920M) [application/zip]
Enregistre : «/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/bullhead-mtc20k-factory-4a950470.zip»

100%[=============================================================================================================================================================================================>] 964,936,536 3.51MB/s ds 4m 30s

2016-11-03 11:54:35 (3.41 MB/s) - «/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/bullhead-mtc20k-factory-4a950470.zip» enregistré [964936536/964936536]

[] Processing with 'API-23 config-naked' configuration
[] Extracting '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/bullhead-mtc20k-factory-4a950470.zip'
[] Unzipping 'image-bullhead-mtc20k.zip'
[] '20' bytecode archive files will be repaired
[] Repairing bytecode under /system partition using oat2dex method
[] Preparing environment for 'arm' ABI
[] Preparing environment for 'arm64' ABI
[] Start processing system partition & de-optimize pre-compiled bytecode
[!] '/framework/cneapiclient.jar' not pre-optimized with sanity checks passed - copying without changes
[!] '/framework/framework-res.apk' not pre-optimized & without 'classes.dex' - copying without changes
[] '/framework/framework.jar' is multi-dex - adjusting recursive archive adds
[!] '/framework/rcsimssettings.jar' not pre-optimized with sanity checks passed - copying without changes
[!] '/framework/rcsservice.jar' not pre-optimized with sanity checks passed - copying without changes
[] System partition successfully extracted & repaired at '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/factory_imgs_repaired_data'
[] Generating blobs for vendor/lge/bullhead
[] Copying radio files '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/vendor/lge/bullhead'
[] Copying product files & generating 'bullhead-vendor-blobs.mk' makefile
[] Generating 'device-vendor.mk'
[] Generating 'AndroidBoardVendor.mk'
[] Bootloader:BHZ10r
[] Baseband:M8994F-2.6.32.1.13
[] Generating 'BoardConfigVendor.mk'
[] Generating 'vendor-board-info.txt'
[] Generating 'Android.mk'
[] Gathering data from 'vendor/app' APK/JAR pre-builts
[] Gathering data from 'proprietary/app' APK/JAR pre-builts
[] Gathering data from 'proprietary/framework' APK/JAR pre-builts
[] Gathering data from 'proprietary/priv-app' APK/JAR pre-builts
[] Generating signatures file
[] All actions completed successfully
[*] Import '/home/tgayet/_aosp-build-tests/test_bullhead_android6-vendors/bullhead/mtc20k/vendor' to AOSP root

Because the old url is redirected to the new one, maybe we can say to curl to follow http redirection by using the -L parameter.

I have cloned the git repo on the master branch (b2ab9a3).

BR
Thierry GAYET

There are a ton of blobs that are not need for Angler. AOSP reproduces many of them.

I have already submitted a pull request for some that were missed that are needed. Would you like me to slim down the ones that are not needed and submit another pull request?

Also I noticed that the AOSP makelinks script you have appended to a makefile in vendor is not working. The one in the tree does work, when called. See: krash867/android_device_huawei_angler@faf48e2

Thank you for your awesome work. This tool is a real gem of a find for me.

Google appears to have upgraded the JS part of the page performing an explicit post request to acknowledge ToS is accepted by user. As such the links index page content is not revealed until user triggers that request.

The request is against the "profile/acknowledgeNotification" URI and seems to bind with a XSRF token generated by the developers subdomain and NID generated from master google TLD. The second part is kind of weird since if user has never visited any google.com TLD before browser cache doesn't have a valid NID to use and thus cannot access the developers page (you can reproduce if clear all cookies & cache and jump directly to nexus devs website).

Investigate if the above mechanism can be performed from command line, otherwise the auto download feature needs to be stripped out and users have to manually perform image downloads prior to generating the vendor blobs.