opengapps / apkcrawler Goto Github PK

View Code? Open in Web Editor NEW

319.0 319.0 83.0 1.2 MB

APK Crawler is a tool to automatically retrieve APKs from various internet sources

License: Other

Python 100.00%

apkcrawler's Introduction

Open GApps

Getting the latest pre-built Open GApps

The latest version of pre-built Open GApps can be found at https://opengapps.org, hosted on SourceForge.

Support Project
Q&A Forum
Development Forum
Support Chat

Support for the pre-built packages from OpenGApps.org

If you have any questions, check out the Open GApps Wiki, especially the FAQ since it'd answer most of the questions. If you can't find the answer to your question use the XDA Q&A Thread or join us on Gitter to receive support. Don't forget to add at least the Open GApps installer debug log and if experiencing Force Closures also include a logcat.

If you did find a bug in the Pre-built OpenGApps.org packages you can report it at the XDA Open GApps Development Thread. Remember to include at minimum the Open GApps installer debug log and if applicable a logcat.

Please don't file directly any GitHub issues to file problems with the Pre-built packages. The GitHub issues tracker is only used for issues concerning the Open GApps Project scripts themselves.

Build your own Open GApps

The example git commands assume you have a GitHub account and have set-up SSH authentication.

If you want to build your own version of Open GApps, you'll need to fetch the git sources:

To initialize your local repository using the Open GApps source tree, clone the main repository with the command:

git clone [email protected]:opengapps/opengapps.git

Then sync the submodules to get the original APK sources as provided by Google. Take note that these repositories are very large (in the order of GiBs). You can also use this command to update the sources at a later moment to their most recent version:

./download_sources.sh [--shallow] [arch]

--shallow will order to fetch only the latest snapshot of the APKs (reduces space used and amount of data to be retrieved by git, by not fetching the APKs' history)
arch can be one of the following "arm, arm64, x86, x86_64" to fetch only data required for specified architecture (note that fallback architectures will be fetched too)

To build Open GApps you'll need the Android build tools installed and set-up in your $PATH. If you use Ubuntu you can check out @mfonville's Android build tools for Ubuntu.

To build Open GApps for all platforms and all Android releases:

make

To build Open GApps for a specific Android release on a specific platform, define both the platform and the API level of that release, seperated by a dash and optionally add the variant with another dash.

Two examples (for building for Android 6.0 on ARM):

make arm-23

make arm-23-stock

To add updated source APKs to the sources archive (you can add more than one at once):

./add_sourceapp.sh [/path/to/the/files/you/want/to/add.apk]* [beta] [/apps/that/should/be/marked/as/beta.apk...]*

For contributors, updated sources can be uploaded. Either without an argument for every architecture, or with one or more arguments for a subset of architectures:

./upload_sources.sh [archs]*

If you want an overview of the locally available sources:

./report_sources.sh

You can add extra arguments to report_sources to do your more advanced bidding too:

./report_sources.sh ( ([sdk] [archs]*) || [arch-sdk] ) && [hash] || [max*mb] || [min*mb] || [nobeta] || [nohelp] || [noleanback] || [nosig]

License

The Open GApps Project itself is licensed under the GPLv3 with an addendum called the Open GApps installable zip exception.

A short explanation of these license terms and the terms used by the pre-built OpenGApps.org packages:

The license and its exception are very comparable to the GNU Compiler Collection.
The Open GApps Project itself is like a "compiler" that is licensed under the GPLv3.
The "Installable zips" produced by the Open GApps Project is a assorted product of which its components adhere to various different licenses. E.g. the Open GApps installer scripts are still GPLv3 licensed.
The author of an "installable zip" can choose the license for this assorted end-product themselves, as long as any changes to the version of The Open GApps Project compiler used during the creation of the "installable zip" are published under the GPLv3 with the "installable zip" too. Also the individual components within the "installable zip" are still licensed under their respective terms.
The pre-built packages from OpenGApps.org are made available under the terms that they can be freely used for personal use only, and are not allowed to be mirrored to the public other than OpenGApps.org

apkcrawler's People

Contributors

Stargazers

Watchers

Forkers

bgiesing vaginessa nagyistge therealssj laudarch stethd czhiraga ru4 isaacandy themenucha ajgappmark wangkai2014 kyuhlee enterstudio hitnkt renbingfei alam00179 ssizemore lfany lucas-developer suhasbhairav primmus qzshucsz twindaciouz sojhal limber0117 dinneo emtee40 quinndiggity sshyran sandsux wintersmilesb101 syafiqahismail liberty1776dtom manedible acidburn0zzz intuitsavage94 jamiejackherer mixianghang huashuolee fakegit pombredanne hhy5277 jayd2446 bigboss9682 ofskymohsen zepcp lqtian mao-tool keefc crackercat kingking888 neaplus pbt98 chp656s ninovalboskoni tiagodiz ashishbijlani reverse-dusk trythat1 mmdevlpr stuff2600 hzj415909583 nikita-bithat appstarbd mastershanecsw blueyes27 clintonkildepstein terrorizer1980 nightincode lili2lala jarhow msgpo juicethadon wangzhenyan12 wuhdafak cccp37 uvbs chanhduc yaroo6969

apkcrawler's Issues

Support *.en.uptodown.com

@mfonville:
I found another source: http://google-drive.en.uptodown.com/android
They have a very simple download URL: http://google-drive.en.uptodown.com/android/download
and here they show all (old) versions: http://google-drive.en.uptodown.com/android/old

Support http://www.apktrunk.com/

The have the .33 versions of the docs.editors :-)

Support http://www.yasdl.com/

Google App version name

The last number after the dot but before the architecture is variable per sdk level; we should treat it as such to be future proof ~~(this also applies to opengapps buildscripts)~~

Keep track of each aptoide store when it was crawled for the last time

To prevent crawling too many pages of the really big stores

Latest playstore APK possibly triggers use of new api...

Running the playstorecrawler with the latest playstore APK (com.android.vending, ver=7.1.12.I-all [0] [PR], vercode=80711200 - link ) causes "message.payload.deliveryResponse.appDeliveryData.downloadAuthCookie" googleplay.py:327 to be None.

This propagates from googleplay.py:208 where, "message.payload.deliveryResponse.status" == 2

This code still works as expected when running against older playstore versions (com.android.vending, ver=7.0.18.H-all [0], vercode=80701800 - link )

cc: @therealssj

If an APK download fails; retry

Should probably become part of apkdownload-helper

(Latin) Keyboard version name

For com.google.android.inputmethod.latin the architecture versioncode is now also part of the versionname. Meanwhile they did drop the architecture name itself from the versionname.
We should account for this variable part of the versionname (also in the opengapps build code)

Also we should keep an eye out if this will be applied to the other keyboard-apps too.

Create an apkdownload-helper class

a new "to be downloaded"-url would be added to the apkdownload-helper instance;
the helper would only do 1 download at a time per host
e.g.:
so 1 at aptoide can be parallel with 1 at apk-dl
but not 2 at aptoide at the same time

It could work with passing the avi-object (that contains the URL that needs to be downloaded) and we could make optional fields like "Stream=False" and "Header=''" to pass the necessary tricks.

APK land

like http://apk.land/apk/google-docs-161521133

Support http://freeapkpro.co/

Also have sometimes rare APKs

download URLs look like: http://freeapkpro.co/file/download-maps-apk.apk

but often the downloads only result in zero bytes.

Use aptoide malware detection

If an APK is ok:
malware.status:"scanned"
and
malware.reason.signature_validated.status:"passed"
malware.reason.signature_validated.signature_from:"market"
malware.reason.scanned.status:"passed"

But if:
malware.status:"warn" the APK is not signed

and more research is necessary:
malware.reason.signature_validated.status:"failed"
in combination with:
malware.reason.signature_validated.signature_from:"market"
means that the APK is incomplete and should simple be ignored at all cost

but if it is combined with:
malware.reason.signature_validated.signature_from:"user" the APK might be complete but from e.g. a factory image, but it can also be a completely fake signature.
In that case the APK can be saved to err.filename.apk.err and output only stderr with big warnings that this APK is untrusted and potentially really harmful, but that it could be an interesting app from e.g. an Android One device.

Support www.plazza.ir

It might be a bit harder to parse since it is in Farsi, but they do offer often quite rare versions of the APKs.
Luckily the page URLs are very simple, with just the package name.
Full version info is provided on the page nd there is a regular download button. So it should be easy.

androidapksfree.com

Example URL with detailed version history: http://www.androidapksfree.com/version-history/google-photos-apk-version-history/

Support dlapks.com

Example link: http://dlapks.com/com.google.android.apps.docs/

Support http://apk.91.com/

Aptoide "old store" detection

Some aptoide stores "die" because their user doesn't upload any new APKs anymore.
Maybe the crawler could indicate if a store has not had any updates for e.g. more than 2 months; suggesting to suspend it from our repo list. (we could e.g. comment the line with that store)

Generate a list of missing APKs

The intermediate output of apkhelper can also be useful to search for missing APKs on Google. Could we find a way to output this in a convenient way?

Support http://globalapk.co/

fix apkmirror case where number of versioncodes exceed display

http://www.apkmirror.com/apk/google-inc/google-play-services/ sorts by versioncode (not upload date) and only shows the highest 15 apks.

If all 15 have the same same version, then additionally crawl: http://www.apkmirror.com/uploads/?app=google-play-services

The 2nd page sorts by upload date but shows 30 apks

Some vernames are "as good as the vercode"

If a vername is provided for a certain (whitelist) of apps, it can safely be assumed that we know what the vercode would have no surprises, and that if the vername matches our local version the APK doesn't need to be downloaded.

Support http://www.apk20.com/

Support apkleecher.com

Support www.apk4now.com

Gets barely updates, so really low priority.

Add support for pulling from Nexus images

Per opengapps/opengapps#70,

@Lekensteyn 's tool (https://github.com/Lekensteyn/make-gapps-zip) could be used to extend apkcrawler's functionality to getting files from Nexus images.

Nothing

Usage

Is there any way to crawl different apks not only opengapps support apks?

Fix apkhelper comparison

VERY IMPORTANT AND CRITICAL BUG:
Currently hyphens crash the whole compare.
It should deal with ( and ) as found in e.g. GMS, Google Play Games and Google Camera.

Add leanback support

APKs from the Nexus Play have been added to the repo thus starting the path towards leanback support by Open GApps. The apkcrawler needs to be update to support pulling these from various sources

Set working directory for APK storage

When downloading an APK to pass it to add_sourceapp.sh it would be useful to be able to let the APK be downloaded to e.g. a temporary location.

Support apk-dl.com

apk-dl.com often has a less popular version of the app.
Downloading from their website should be relatively trivial.

Support 1mobile.com

Extend set of apps

Since we now have the super package, and the AndroidTV package is in the works, the set of packages should probably be extended to cover these new apps :-)

Support http://getandroidapp.org/

apks.org often have multiple versions of each APK, often including less popular ones, downloading from their website could help getting a more complete APK set.

http://birddl.com/

F-Droid support?

Are there plans to add f-droid support to the crawler?

Playstore crawler is offered sdk24 Chrome for all CPUs, but downloading results in 500 error

The playstore crawler is offered the correct vercode for each Chrome sdk24 apk for all CPU's

arm: 278512402
arm64: 278512452
x86: 278512412
x86_64: 278512462

... but when attempting to download each apk the request results in a HTTP500 error

cc: @therealssj

Make compatible with Python 3

When using Python 3, the script doesn't work.

Terminal Output

[bgiesing@bgiesing-TOSHIBA opengapps]$ ./add_sourceapp.sh $(python ~/opengapps/apkcrawler/apkcrawler.py docs)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
    msg = self.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
    return fmt.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 565, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.4/logging/__init__.py", line 326, in getMessage
    msg = str(self.msg)
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 79, in __str__
    import urlparse
ImportError: No module named 'urlparse'
Call stack:
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 387, in <module>
    main(sys.argv[1:])
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 350, in main
    logging.debug(ai)
Message: <__main__.ApkInfo object at 0x7fae73853e10>
Arguments: ()
Traceback (most recent call last):
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 387, in <module>
    main(sys.argv[1:])
  File "/home/bgiesing/opengapps/apkcrawler/apkcrawler.py", line 353, in main
    maxVersionByName = sorted(ai.versions)[-1]
TypeError: unorderable types: ApkVersionInfo() < ApkVersionInfo()

Log

2015-09-04 20:59:44,815 INFO/main(11255): Fetching Information for: Docs
2015-09-04 21:03:36,701 INFO/main(11373): Fetching Information for: Docs
2015-09-04 21:05:48,403 INFO/main(11386): Fetching Information for: Docs
2015-09-04 21:16:48,791 DEBUG/main(11682): ['Docs']
2015-09-04 21:16:48,791 INFO/main(11682): Fetching Information for: Docs
2015-09-04 21:16:48,806 DEBUG/getAppVersions(11682): Requesting2: http://www.apkmirror.com/apk/google-inc/docs/
2015-09-04 21:16:50,072 DEBUG/printDictionary(11682): "Docs 1.4.332.17.30"         - /apk/google-inc/docs/docs-1-4-332-17-30-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.33"         - /apk/google-inc/docs/docs-1-4-332-17-33-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.34"         - /apk/google-inc/docs/docs-1-4-332-17-34-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.35"         - /apk/google-inc/docs/docs-1-4-332-17-35-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.36"         - /apk/google-inc/docs/docs-1-4-332-17-36-2-android-apk-download/
2015-09-04 21:16:50,073 DEBUG/printDictionary(11682): "Docs 1.4.332.17.44 (arm64)" - /apk/google-inc/docs/docs-1-4-332-17-44-arm64-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.33"         - /apk/google-inc/docs/docs-1-4-352-08-33-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.34"         - /apk/google-inc/docs/docs-1-4-352-08-34-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.08.35"         - /apk/google-inc/docs/docs-1-4-352-08-35-3-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.09.30"         - /apk/google-inc/docs/docs-1-4-352-09-30-android-apk-download/
2015-09-04 21:16:50,074 DEBUG/printDictionary(11682): "Docs 1.4.352.09.33"         - /apk/google-inc/docs/docs-1-4-352-09-33-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.34"         - /apk/google-inc/docs/docs-1-4-352-09-34-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.35"         - /apk/google-inc/docs/docs-1-4-352-09-35-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.36"         - /apk/google-inc/docs/docs-1-4-352-09-36-android-apk-download/
2015-09-04 21:16:50,075 DEBUG/printDictionary(11682): "Docs 1.4.352.09.44 (arm64)" - /apk/google-inc/docs/docs-1-4-352-09-44-arm64-android-apk-download/