• Parse and process release files into CSV format, then store to BigQuerry for later use.

Modeled after ETL pipeline, this project handels everything from extraction to load

• Extraction
  • Release Parser extracts wanted data from target folder
    • <Release_Name>-FileList.csv, meta data for all target files, size, etc.
    • <Release_Name>-FeatureList.csv
    • <Release_Name>-ExeList.csv
    • <Release_Name>-ReleaseContent.csv
    • <Release_Name>-PropertiesList.csv
    • <Release_Name>-ServicesList
  • Added in function to extract content from APEX files, as well as parse symbolic links
• Transformation
  • The extracted data is written and stored as CSV files
• Load
  • Use the script inside uploadScript

1. Clone and open the project, run uberjar
2. Edit config, point path to jar to output uberjar
3. Create a output folder
4. program args -i -o -a 28 -f <SubDir1, SubDir2>
   • -f allows you to specify what are the sub dir to be parsed
   • If not specified, all files and dir in the folder will be parsed
5. Set JRE to jdk-14
6. Run and check output folder for result

This peoject can be adapted for Google Cloud Build with the following steps:

• Clone this project in Google Cloud Shell
• Create a new JRE Docker image
  • Clone Google Cloud Build community images
  • gcloud builds submit --config=cloudbuild-ndk-jre11.yaml --substitutions=_ANDROID_VERSION=29
  • Builds image with cloudbuild-ndk-jre11.yaml
• Pull the built image to local
• Setup buckets
• Build gcloud builds submit --config=cloudbuild-ndk-jre11.yaml --substitutions=_ANDROID_VERSION=29
  • ./testCloudBuildLocal.sh does the same thing.

Results are stored inside bucket gs://* your_bucket_ID *, this includes releaseParser.jar and index.html.

releaseParser.sh helps you run releaseParser test.sh is an example of releaseParser.sh

Here are some command lines that can be used for testing docker image

• docker image inspect
  • Inspects the image
  • Used for chekcing if the paths inside are setup correctly
• docker run -t -d --rm --name sdk bash
  • Runs docker immage
• docker exec -ti sdk sh -c "<COMMAND_PROMP>"
  • Executes <COMMAND_PROMP> on the running docker image
• docker stop <CONTAINER_ID>
  • Stops the running docker
• docker ps
  • Checks the running docker status

• The key value pair of ro.product.cpu.abilist,arm64-v8a,armeabi-v7a,armeabi inside PropertiesList can not be uploaded properly
• Finish data visualization at DataStudio

• Automate testing and uploading