project-artist / dexterous Goto Github PK

Library and standalone CLI tool for apk/dex merging, repackaging and signing. Can also get used as a dex analyzer framework.

License: Other

Java 99.91% HTML 0.06% Python 0.03%

analysis android artist dex merge tool

dexterous's People

Contributors

Stargazers

Watchers

Forkers

terry2012 alfink mcintoshmatthew81 androidpal jackyglony keqizwl yonotoken zgh913913 supernghia89 freemanzyq dandycheung liuyun201990 crackercat diankuangliuxu nearlee2008

dexterous's Issues

Cannot merge codelib into systemserver

When I try to use the current dexterous version @ fd5a845 to merge a codelib into the systemserver (services.jar), the following error occurs:

Partially merging CodeLib into services.jar using dexterous
I/Dexterous: Loading DexFile: services.apk:classes.dex
I/Dexterous: Loading DexFile: services.apk:classes2.dex
I/Dexterous: DONE Loading 01 Dexfiles (services.apk)
I/Dexterous: Loading DexFile: app-debug.apk:classes.dex
I/Dexterous: Loading DexFile: app-debug.apk:classes2.dex
I/Dexterous: DONE Loading 01 Dexfiles (app-debug.apk)
I/Dexterously: # MULTIDEX File. DexFile Count: 2
I/Dexterously: 
I/Dexterously: # DEXFILE: services.apk:classes.dex #############################################
I/Dexterously: 
I/Loggy: ----------------------------------------------------------------------------
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                     HEADER | OFF:        0 | SIZE:        1 |
I/Loggy: Section |TYPE:                 STRING_IDS | OFF:      112 | SIZE:    63478 |
I/Loggy: Section |TYPE:                   TYPE_IDS | OFF:   254024 | SIZE:     4637 |
I/Loggy: Section |TYPE:                  PROTO_IDS | OFF:   272572 | SIZE:     9860 |
I/Loggy: Section |TYPE:                  FIELD_IDS | OFF:   390892 | SIZE:    19115 |
I/Loggy: Section |TYPE:                 METHOD_IDS | OFF:   543812 | SIZE:    34535 |
I/Loggy: Section |TYPE:                 CLASS_DEFS | OFF:   820092 | SIZE:     2616 |
I/Loggy: Section |TYPE:   ANNOTATION_SET_REF_LISTS | OFF:   903804 | SIZE:      352 |
I/Loggy: Section |TYPE:            ANNOTATION_SETS | OFF:   908676 | SIZE:     3517 |
I/Loggy: Section |TYPE:                 CODE_ITEMS | OFF:   945024 | SIZE:    24583 |
I/Loggy: Section |TYPE: ANNOTATION_DIRECTORY_ITEMS | OFF:  4128108 | SIZE:     2239 |
I/Loggy: Section |TYPE:                 TYPE_LISTS | OFF:  4193532 | SIZE:     6329 |
I/Loggy: Section |TYPE:          STRING_DATA_ITEMS | OFF:  4262876 | SIZE:    63478 |
I/Loggy: Section |TYPE:            DEBUG_INFO_ITEM | OFF:  5763692 | SIZE:    22550 |
I/Loggy: Section |TYPE:            ANNOTATION_ITEM | OFF:  6587462 | SIZE:     3781 |
I/Loggy: Section |TYPE:        ENCODED_ARRAY_ITEMS | OFF:  6656830 | SIZE:      539 |
I/Loggy: Section |TYPE:           CLASS_DATA_ITEMS | OFF:  6670105 | SIZE:     2604 |
I/Loggy: Section |TYPE:                   MAP_LIST | OFF:  6880792 | SIZE:        1 |
I/Loggy: ----------------------------------------------------------------------------
I/Dexterously: 
I/Dexterously: # DEXFILE: app-debug.apk:classes.dex #############################################
I/Dexterously: 
I/Loggy: ----------------------------------------------------------------------------
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                       null | OFF:       -1 | SIZE:        0 |
I/Loggy: Section |TYPE:                     HEADER | OFF:        0 | SIZE:        1 |
I/Loggy: Section |TYPE:                 STRING_IDS | OFF:      112 | SIZE:    21234 |
I/Loggy: Section |TYPE:                   TYPE_IDS | OFF:    85048 | SIZE:     2352 |
I/Loggy: Section |TYPE:                  PROTO_IDS | OFF:    94456 | SIZE:     3489 |
I/Loggy: Section |TYPE:                  FIELD_IDS | OFF:   136324 | SIZE:    10394 |
I/Loggy: Section |TYPE:                 METHOD_IDS | OFF:   219476 | SIZE:    17438 |
I/Loggy: Section |TYPE:                 CLASS_DEFS | OFF:   358980 | SIZE:     1626 |
I/Loggy: Section |TYPE:                 CODE_ITEMS | OFF:   411012 | SIZE:    11730 |
I/Loggy: Section |TYPE:            DEBUG_INFO_ITEM | OFF:  1237269 | SIZE:    10561 |
I/Loggy: Section |TYPE:                 TYPE_LISTS | OFF:  1463400 | SIZE:     2079 |
I/Loggy: Section |TYPE:          STRING_DATA_ITEMS | OFF:  1484594 | SIZE:    21234 |
I/Loggy: Section |TYPE:            ANNOTATION_ITEM | OFF:  2021194 | SIZE:     2040 |
I/Loggy: Section |TYPE:           CLASS_DATA_ITEMS | OFF:  2054433 | SIZE:     1516 |
I/Loggy: Section |TYPE:        ENCODED_ARRAY_ITEMS | OFF:  2151369 | SIZE:      291 |
I/Loggy: Section |TYPE:            ANNOTATION_SETS | OFF:  2167192 | SIZE:     1916 |
I/Loggy: Section |TYPE:   ANNOTATION_SET_REF_LISTS | OFF:  2189004 | SIZE:      144 |
I/Loggy: Section |TYPE: ANNOTATION_DIRECTORY_ITEMS | OFF:  2191632 | SIZE:     1436 |
I/Loggy: Section |TYPE:                   MAP_LIST | OFF:  2237768 | SIZE:        1 |
I/Loggy: ----------------------------------------------------------------------------
I/MethodFilter: Annotationtype used for whitelisting:Lsaarland/cispa/artist/codelib/CodeLib$Inject;
Exception in thread "main" java.lang.IllegalArgumentException: position=88867372 length=2237988
	at comm.android.dex.Dex.open(Dex.java:191)
	at comm.android.dx.merge.MethodFilter.getString(MethodFilter.java:136)
	at comm.android.dx.merge.MethodFilter.getMethodString(MethodFilter.java:150)
	at comm.android.dx.merge.MethodFilter.initializeWhitelist(MethodFilter.java:230)
	at comm.android.dx.merge.MethodFilter.<init>(MethodFilter.java:387)
	at comm.android.dex.Dex.setWhitelistedAnnotation(Dex.java:358)
	at saarland.cispa.dexterous.cli.Dexterously.mergeCodeLib(Dexterously.java:617)
	at saarland.cispa.dexterous.cli.Main.main(Main.java:49)

It does NOT occur when I use the dexterous version from before the annotation-based whitelisting patch (tested: 3253508).

@alfink do you have an intuition what is going wrong here?

Library distribution: JitPack

Given that we already build an Android library, we could as well publish so that it can be included in, e.g., a build.gradle file.

The fastest way to do so seems to be https://jitpack.io/, which automatically builds a project from a GitHub repo and exposes the result, given there is a certain task available. It should be straightforward to set this up, judging from the log at https://jitpack.io/#Project-ARTist/dexterous

Merging fields

Merging fails when whitelisting class fields.

Logcat:

11-03 21:20:19.688  7541 12227 D Dexterous: Dexterous() Init:
11-03 21:20:19.688  7541 12227 D Dexterous: > App:            /data/app/de.alfink.regbug-1/base.apk
11-03 21:20:19.688  7541 12227 D Dexterous: > CodeLib:        /data/user/0/saarland.cispa.artist.artistgui/app_modules/saarland.cispa.artist.module.regbug/codelib.apk
11-03 21:20:19.689  7541 12227 D Dexterous: > CodeLibName:    codelib.apk
11-03 21:20:19.689  7541 12227 D Dexterous: > CodeLibDexName: codelib.apk:classes.dex
11-03 21:20:19.689  7541 12227 D Dexterous:   > DexFile:        base.apk:classes.dex
11-03 21:20:19.689  7541 12227 D Dexterous:   > DexFile:        codelib.apk:classes.dex
11-03 21:20:19.690  7541 12227 I MethodFilter: Annotationtype used for whitelisting:Lsaarland/cispa/artist/codelib/CodeLib$Inject;
11-03 21:20:19.690  7541 12227 D MethodFilter: Field whitelisted:10
11-03 21:20:19.690  7541 12227 D MethodFilter: Field whitelisted:11
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:12
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:13
11-03 21:20:19.691  7541 12227 D MethodFilter: Field whitelisted:14
11-03 21:20:19.692  7541 12227 D MethodFilter: Field whitelisted:15
11-03 21:20:19.692  7541 12227 D MethodFilter: Field whitelisted:16
11-03 21:20:19.692  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:fiveargs
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:fourargs
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:onearg
11-03 21:20:19.693  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:sevenargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:sixargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:threeargs
11-03 21:20:19.694  7541 12227 D MethodFilter: ColdelibWhitelisting	Method annotated:twoargs
11-03 21:20:19.695  7541 12227 I MethodFilter: CodelibWhitelisting	Class whitelisted: 17 (Lsaarland/cispa/artist/codelib/CodeLib;)
11-03 21:20:19.695  7541 12227 I MethodFilter: class_annotations_off: 3720
11-03 21:20:19.695  7541 12227 I MethodFilter: f/m/p:7/7/0
11-03 21:20:19.696  7541 12227 I Dexterous: MERGING DEX codelib.apk:classes.dex into base.apk:classes.dex
11-03 21:20:19.697  7541 12227 D Dexterous: DexMerger	mergeStringIds...
11-03 21:20:19.698  7541 12227 W DexMerger: Whitelisted: 20 - LL
11-03 21:20:19.749  7541 12227 W DexMerger: Whitelisted: 21 - LLL
11-03 21:20:19.751  7541 12227 W DexMerger: Whitelisted: 22 - LLLL
11-03 21:20:19.751  7541 12227 W DexMerger: Whitelisted: 23 - LLLLL
11-03 21:20:19.752  7541 12227 W DexMerger: Whitelisted: 24 - LLLLLL
11-03 21:20:19.753  7541 12227 W DexMerger: Whitelisted: 25 - LLLLLLL
11-03 21:20:19.754  7541 12227 W DexMerger: Whitelisted: 26 - LLLLLLLL
11-03 21:20:19.755  7541 12227 W DexMerger: Whitelisted: 30 - Ldalvik/annotation/MemberClasses;
11-03 21:20:19.795  7541 12227 W DexMerger: Whitelisted: 32 - Ljava/lang/Object;
11-03 21:20:19.797  7541 12227 W DexMerger: Whitelisted: 43 - Lsaarland/cispa/artist/codelib/CodeLib;
11-03 21:20:19.803  7541 12227 W DexMerger: Whitelisted: 57 - argument1
11-03 21:20:19.843  7541 12227 W DexMerger: Whitelisted: 58 - argument2
11-03 21:20:19.844  7541 12227 W DexMerger: Whitelisted: 59 - argument3
11-03 21:20:19.845  7541 12227 W DexMerger: Whitelisted: 60 - argument4
11-03 21:20:19.845  7541 12227 W DexMerger: Whitelisted: 61 - argument5
11-03 21:20:19.846  7541 12227 W DexMerger: Whitelisted: 62 - argument6
11-03 21:20:19.847  7541 12227 W DexMerger: Whitelisted: 63 - argument7
11-03 21:20:19.847  7541 12227 W DexMerger: Whitelisted: 71 - fiveargs
11-03 21:20:19.868  7541 12227 W DexMerger: Whitelisted: 72 - fourargs
11-03 21:20:19.870  7541 12227 W DexMerger: Whitelisted: 78 - onearg
11-03 21:20:19.925  7541 12227 W DexMerger: Whitelisted: 85 - sevenargs
11-03 21:20:19.950  7541 12227 W DexMerger: Whitelisted: 86 - sixargs
11-03 21:20:19.953  7541 12227 W DexMerger: Whitelisted: 88 - threeargs
11-03 21:20:19.958  7541 12227 W DexMerger: Whitelisted: 89 - twoargs
11-03 21:20:19.962  7541 12227 W DexMerger: Whitelisted: 91 - value
11-03 21:20:19.968  7541 12227 D DexMerger$IdMerger: Blacklisted: 49
11-03 21:20:19.968  7541 12227 D Dexterous: DexMerger	mergeTypeIds...
11-03 21:20:19.980  7541 12227 D DexMerger$IdMerger: Blacklisted: 14
11-03 21:20:19.980  7541 12227 D Dexterous: DexMerger	mergeTypeLists...
11-03 21:20:20.001  7541 12227 D Dexterous: DexMerger	mergeProtoIds...
11-03 21:20:20.025  7541 12227 D DexMerger$IdMerger: Blacklisted: 6
11-03 21:20:20.025  7541 12227 D Dexterous: DexMerger	mergeFieldIds...
11-03 21:20:20.029  2228  2228 D wpa_supplicant: wlan0: Control interface command 'SIGNAL_POLL'
11-03 21:20:20.044  2228  2228 D wpa_supplicant: CTRL-DEBUG: global_ctrl_sock-sendto: sock=9 sndbuf=163840 outq=0 send_len=48
11-03 21:20:20.092  7541 12227 W System.err: java.lang.IllegalArgumentException: Expected an unsigned short: -1
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dex.Dex$Section.writeUnsignedShort(Dex.java:673)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dex.FieldId.writeTo(FieldId.java:59)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$5.write(DexMerger.java:633)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$5.write(DexMerger.java:607)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:353)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeFieldIds(DexMerger.java:635)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeDexesMethods(DexMerger.java:201)
11-03 21:20:20.098  7541 12227 W System.err: 	at comm.android.dx.merge.DexMerger.mergeMethodsOnly(DexMerger.java:244)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLibReference(Dexterous.java:234)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLibReference(Dexterous.java:220)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeMethodIds(Dexterous.java:107)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.dexterous.Dexterous.mergeCodeLib(Dexterous.java:121)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.artist.artistgui.instrumentation.stages.InstrumentationStagesImpl.mergeCodeLib(InstrumentationStagesImpl.java:182)
11-03 21:20:20.098  7541 12227 W System.err: 	at saarland.cispa.artist.artistgui.instrumentation.InstrumentationTask.run(InstrumentationTask.java:69)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
11-03 21:20:20.098  7541 12227 W System.err: 	at java.lang.Thread.run(Thread.java:761)

Fine-grained merging

Instead of merging all classes, types and methods, it would be preferable to allow for selective, fine-grained merging. This avoids wasting IDs on Android helper classes that are not really used and allows for merging codelib packages that feature bigger libraries. With this approach, a codelib class could define a small interface to the large library and only those few class methods are assigned IDs in target dex files (see 64k method ID limit).
This could, e.g., be controlled with blacklists or whitelists provided via commandline arguments.

Too many field IDs. field ID not in [0, 0xffff]: 65536

Exception in thread "main" comm.android.dex.DexIndexOverflowException: Too many field IDs. field ID not in [0, 0xffff]: 65536

DevNote: Zip-/APKfile filename encoding

I leave this here as a reference, in case errors with filenames occur at a large scale testing later:

Digging through https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT D.3 (APPENDIX D - Language Encoding (EFS)):

It's either CP437 (original encoding), UTF-8 or user defined via the 0x0008 Extra Field

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.