reddr / libscout Goto Github PK

19:47:55 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: null 19:47:55 ERROR TplCLI : java.lang.NullPointerException at java.util.TreeMap.getEntry(TreeMap.java:347) at java.util.TreeMap.containsKey(TreeMap.java:232) at java.util.TreeSet.contains(TreeSet.java:234) at de.infsec.tpl.modules.libmatch.LibCodeUsage.checkUsage(LibCodeUsage.java:78) at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:204) at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84) at de.infsec.tpl.TplCLI.main(TplCLI.java:141)

I would like to use this tool, can u please let me know if there are any tutorial kind

#10 Before running the LibScout jar, and in order to make it work you should be in the LibScout/ directory or you will get this ERROR : Could not open [./logging/logback.xml]. this error means that when you were developping the solution you gave the code the partial parth (logical path using the ./ unix notion).

this situation prevent this program to be run from outside the LibScout folder, which means less flexibility. i hope you guys could solv this. you can add an option for the command java -jar LibScout.jar -o match -a lib/android-X.jar -p -s -d $someapp.apk in which we can indicate the logback.xml file location

Is there a way to retrieve the mappings from obfuscated to real class/package name? This would be very useful if I want to do some analysis on the library files as found in the app - which for various reasons may not 100% match the files in the original library JAR (e.g. due to dead-code elimination).

Example:

{
	"libName": "Facebook",
	"package-mappings": {
		"a.a.a": "com.facebook.applinks".
		"a.a.b": "com.facebook.login",
		...
	},
	...
}

Or even

{
	"libName": "Facebook",
	"class-mappings": {
		"a.a.a.a": "com.facebook.applinks.AppLinkData".
		"a.a.a.b": "com.facebook.applinks.FacebookAppLinkResolver",
		...
	},
	...
}

isMethodSyntetic is not a method in IMethod class in com.ibm.wala.classLoader . There is a method named isSynthetic which is defined instead.
Here is the link https://www.ibm.com/support/knowledgecenter/SSS9LM_9.0.0/com.ibm.rational.appscansrc.utilities.osx.doc/api/com/ibm/wala/classLoader/IMethod.html

i tried to create a profile and try to find a match but i got this (ps the apk used and the second phase use the given retrofit library)

java -jar ../build/LibScout.jar -o profile -a ../lib/android-23.jar -x /root/Downloads/retrofit.xml /root/Downloads/retrofit-2.4.0.jar
10:18:48,756 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
10:18:48,757 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:18:48,757 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
10:18:48,763 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
10:18:48,918 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

java -jar ../build/LibScout.jar -o match -a ../lib/android-23.jar /root/Downloads/dgi_proguard.apk
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
10:24:45,355 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
10:24:45,514 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

java.lang.ArrayIndexOutOfBoundsException: 636
at android.content.res.StringBlock.getShort(Unknown Source)
at android.content.res.StringBlock.getString(Unknown Source)
at android.content.res.AXmlResourceParser.getName(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.loadClassesFromBinaryManifest(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest$1.handleManifest(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.handleAndroidManifestFile(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.loadManifestFile(Unknown Source)
at de.infsec.tpl.LibraryIdentifier.parseManifest(Unknown Source)
at de.infsec.tpl.LibraryIdentifier.identifyLibraries(Unknown Source)
at de.infsec.tpl.TplCLI.main(Unknown Source)

Hi,

I'm trying to use LibScout with an APK containing 100+ dex files. I found out (after increasing the log levels) that currently only the first 10 classes.dex files are processed, resulting in an incomplete package tree for APKs with 10+ dex files.
The cause of this is the regex in the WALA code: classes[1-9]{0,1}\\.dex.
I think it might be a good idea to get rid of this limitation.

I changed the regex in the DexFileModule.class file to classes[0-9]{0,9}\\.dex and every dex file is successfully processed now. But now I get an exception when exporting to JSON:

ERROR TplCLI                    : [FATAL EXCEPTION] analysis aborted: null
ERROR TplCLI                    : java.lang.NullPointerException
        at de.infsec.tpl.profile.ProfileMatch$Export.<init>(ProfileMatch.java:120)
        at de.infsec.tpl.profile.ProfileMatch.export(ProfileMatch.java:132)
        at de.infsec.tpl.stats.AppStats$Export.<init>(AppStats.java:130)
        at de.infsec.tpl.stats.AppStats.export(AppStats.java:48)
        at de.infsec.tpl.utils.Utils.obj2JsonFile(Utils.java:487)
        at de.infsec.tpl.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:207)
        at de.infsec.tpl.TplCLI.main(TplCLI.java:155)

Also, I know that you use Joana's WALA fork but any plans to publish your patches?

I would like to express my appreciation for the creation of this tool in the first place.

I found an minor error regarding the json file name created by Library Detection. In the process of generating the JSON output file through Library Detection (-o match), any occurrence of '.aar' or '.jar' in the file name is automatically removed. For instance, if the APK name is "com.aarsomething", the final JSON file will be named as "comsomething.json".

Thank you.

Hi Derr,

I'm trying to use Libscout for detecting the third party dependencies from apk file. However, I'm not succeeded yet by using the Libscout. Below are the steps I did, after downloading Libscout code.

Step 1: Generated the jar file using the ant script
Step 2: Downloaded the Android SDK from the Android Studio
Step 3: Downloaded Profiles from "LibScout Profiles" https://github.com/reddr/LibScout-Profiles
Step 4: Executed the below command to run the Libscout.jar on apk file, find the results as an attachment.
123.log
Calendar.log
diva-beta.log
LeafPic.log

java -jar c:\LibScout\build\LibScout.jar -opmode match -a c:\LibScout\lib\android.jar (downloaded in step2) -p c:\LibScout\profiles\profiles\md5\Android (Downloaded in Step3) -d c:\LibScout\log c:\LibScout\apk

Please correct me if there is any process wrong,

Also, I would like to request you to share "Original Library SDK" with me.

when i run LibSout using -j option:
java -jar build/LibScout.jar -o match -a lib/android-23.jar -p profiles/md5/ -j json/ -m /root/Downloads/dgi_proguard.apk

inside the json/ folder i found the json files stored in other folders arborescence, i wonder is it based on the application package?

when i run my test i just got the list of existing libraries but i don't got any information about the exact version of the library. can you help plz?

I'm getting build errors as soon as I open the project in eclipse. Please find the below screenshot and help me in fixing them.

Dear LibScout Team

Currently, I am integrating LibScout into MobSF for a university project. In my testcase, I have an Android APK depending on OkHttp version 2.3.0. Referring to this repository's README.md, this version should be detected as vulnerable. However, this does not happen.

Specifics:
Java OpenJDK 8, tested on Ubuntu 20 LTS x86-64 and macOS Mojave

How to replicate:
Since the vulnerability is still present in the app's latest version, I am not going to disclose the exact APK used. Therefore, the placeholder VULN.apk

(executed in LibScout root directory, PROFILE_PATH is pointing to this):

java -jar build/libs/LibScout.jar -a <SDK_PATH> -p <PROFILE_PATH> -o match <VULN.apk>

Enabling the comments in the config file does display some information, but not the important [SECURITY] section. Has this been found after the latest update to above repo, or are we using the tool wrongly?

Yours sincerely

I pulled the newest commit and tried to recompile the project, but i get failure messages from the java compiler.

clean:
[delete] Deleting directory /home/markus/bachelor/tools/LibScout/bin
[delete] Deleting directory /home/markus/bachelor/tools/LibScout/build
compile:
[mkdir] Created dir: /home/markus/bachelor/tools/LibScout/bin
[echo] Using Java version 1.8.
[javac] Compiling 65 source files to /home/markus/bachelor/tools/LibScout/bin
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:22: error: cannot find symbol
[javac] import android.util.TypedValue;
[javac] ^
[javac] symbol: class TypedValue
[javac] location: package android.util
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:28: error: cannot find symbol
[javac] import android.util.TypedValue;
[javac] ^
[javac] symbol: class TypedValue
[javac] location: package android.util
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:215: error: cannot find symbol
[javac] if (valueType!=TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:288: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_STRING) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:303: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_FLOAT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:313: error: cannot find symbol
[javac] if (valueType>=TypedValue.TYPE_FIRST_INT &&
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:314: error: cannot find symbol
[javac] valueType<=TypedValue.TYPE_LAST_INT)
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:328: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:129: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_STRING) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:132: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_ATTRIBUTE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:135: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:138: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_FLOAT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:141: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_INT_HEX) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:144: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_INT_BOOLEAN) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:147: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_DIMENSION) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:149: error: cannot find symbol
[javac] DIMENSION_UNITS[data & TypedValue.COMPLEX_UNIT_MASK];
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:151: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_FRACTION) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:153: error: cannot find symbol
[javac] FRACTION_UNITS[data & TypedValue.COMPLEX_UNIT_MASK];
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:155: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_COLOR_INT && type<=TypedValue.TYPE_LAST_COLOR_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:155: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_COLOR_INT && type<=TypedValue.TYPE_LAST_COLOR_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:158: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_INT && type<=TypedValue.TYPE_LAST_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:158: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_INT && type<=TypedValue.TYPE_LAST_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/StringItems.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 22 errors

In XMLParser.java, the date format is "dd.MM.yyyy", but in the xml template, the suggested date format is 'DD/MM/YYYY'. Not a big deal, but may be good to have it fixed.

PS C:\Users\z\Desktop\Lib-tools\LibScout-2.3.2> java -jar .\LibScout.jar -o profile -x C:\Users\z\Desktop\ttst\library.xml C:\Users\z\Desktop\ttst\aboutlibraries-6.0.0.jar
22:08:01 INFO LibScoutConfig : This is LibScout 2.3.2
22:08:01 INFO LibraryProfiler : Process library: aboutlibraries-6.0.0.jar
22:08:01 INFO LibraryProfiler : Library description:
22:08:01 INFO LibraryProfiler : name: aboutlibraries
22:08:01 INFO LibraryProfiler : category: Utilities
22:08:01 INFO LibraryProfiler : version: 6.0.0
22:08:01 INFO LibraryProfiler : release-date: 21.05.2017
22:08:01 INFO LibraryProfiler : comment: no
22:08:02 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
22:08:02 ERROR TplCLI : com.ibm.wala.ipa.cha.ClassHierarchyException: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
how can i solve this error

In the WU run on 1.2TB of APKs, I'm using the JSON output. It outputs mostly using a directory hierarchy based on the segments of the Java Package Name. But sometimes it outputs straight into the json/ dir:

$ ls -1 json/org.mariotaku.twidere*.json json/org/mariotaku/*.json
json/org/mariotaku/org.mariotaku.imageviewergl_1_1.json
json/org/mariotaku/org.mariotaku.twidere_135_135.json
json/org/mariotaku/org.mariotaku.twidere_166_166.json
json/org/mariotaku/org.mariotaku.twidere_181_181.json
json/org/mariotaku/org.mariotaku.twidere_196_196.json
json/org/mariotaku/org.mariotaku.twidere_213_213.json
json/org/mariotaku/org.mariotaku.twidere_260_260.json
json/org/mariotaku/org.mariotaku.twidere_370_370.json
json/org/mariotaku/org.mariotaku.twidere_371_371.json
json/org/mariotaku/org.mariotaku.twidere_379_379.json
json/org.mariotaku.twidere_389_0.json
json/org.mariotaku.twidere_393_0.json
json/org.mariotaku.twidere_394_0.json

was it created manually?

I downloaded an open source code from gitee (https://gitee.com/DaLianZhiYiKeJi/xpay?_from=gitee_search), and I divided it into two steps to verify the libscout test results

Step 1: I compile the downloaded code into apK, and then use libscout to analyze the apK. The result is that the SDK contained in the APK can be detected. For example, the okhttp library was successfully detected 。

Step 2:, I compiled the downloaded code into apK using ProGuard confusion, and then analyzed the apK using LibScout. The okHTTP library I just detected was not detectable.

The question is: can LibScout support this scenario after ProGuard confusion ？

Thanks。

Hi, when I use Library Detection (-o match), It seems appear some errors.

I input the command:
java -jar ./build/libs/LibScout.jar -o match -p ./profiles/Cloud -a ./sdk/android.jar -j ./json app.apk
The profiles are generated by myself, from some SDK jar files.
I also change the asserts/library.xml.

And then errors appear:
the command output is
`
11:07:08 DEBUG HashTree : Generate hash tree..

11:07:14 DEBUG HashTree : - generated 698 package hashes.

11:07:14 DEBUG HashTree : - generated 8939 clazz hashes.

11:07:14 DEBUG HashTree : - generated 60936 method hashes.

11:07:14 DEBUG HashTree : => Library Hash: b66cf0802b150b4e24eac7a566f61709

11:07:14 INFO AppProfile : - generated app hash trees (in 5 sec, 636 ms)

11:07:14 INFO AppProfile :

11:07:14 INFO LibraryIdentifier : = Scan for library root packages (heuristic) =

11:07:14 INFO LibraryIdentifier :

11:07:14 INFO LibraryIdentifier : = Match profiles =

11:07:14 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: null

11:07:14 ERROR TplCLI : java.lang.NullPointerException
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:183)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84)
at de.infsec.tpl.TplCLI.main(TplCLI.java:141)
`
Please help me why does this issue occur and how to solve this issue?

I've run my desirable command on my windows 64-bit machine, and got such this result:

19:30:58 INFO PackageTree : ظ¤é ظ¤é ظ¤£ظ¤�ظ¤� d (3)

A related conversation is: here

the tool was working just fine but today when i run a new test here is what it said:

java -jar LibScout.jar -o match -a ../lib/android-23.jar /root/Downloads/dgi_proguard.apk
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
11:36:07,461 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
11:36:07,623 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

No profiles found in ./profiles. Check your settings!

i have all the files in place './logging/logback.xml' and i have the profiles you offered. i don't know what is the source of this error.

where can i find [logback.groovy] and [logback-test.xml]

I get this NPE when working with profiles I generated myself from the attached file.

$ libscout --opmode match --android-sdk /home/fdroid/android-sdk/platforms/android-29/android.jar --libscout-conf /home/fdroid/code/reddr/LibScout/config/LibScout.toml --log-dir /data/ttt-apks/libraries/run-match.sh-1598623728-logs --profiles-dir ./profiles --json-dir /data/ttt-apks/random-google-play//json /data/ttt-apks/random-google-play/
Exception in thread "main" java.lang.NullPointerException
        at de.infsec.tpl.profile.LibProfile$LibProfileComparator.compare(LibProfile.java:68)
        at de.infsec.tpl.profile.LibProfile$LibProfileComparator.compare(LibProfile.java:65)
        at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
        at java.base/java.util.TimSort.sort(TimSort.java:234)
        at java.base/java.util.Arrays.sort(Arrays.java:1515)
        at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
        at de.infsec.tpl.profile.Profile.loadLibraryProfiles(Profile.java:87)
        at de.infsec.tpl.TplCLI.main(TplCLI.java:122)

Here is my proposed fix for --opmode match:

--- a/src/de/infsec/tpl/profile/Profile.java
+++ b/src/de/infsec/tpl/profile/Profile.java
@@ -81,6 +81,10 @@ public abstract class Profile implements Serializable {
                        // de-serialize library profiles
                        for (File f : Utils.collectFiles(profilesDir, new String[]{LibraryProfiler.FILE_EXT_LIB_PROFILE})) {
                                LibProfile lp = (LibProfile) Utils.disk2Object(f);
+                               if (lp == null || lp.description == null) {
+                                   System.err.println("ERROR: " + f + " produces a null profile!");
+                                   continue;
+                               }
                                profiles.add(lp);
                        }
 

PyzeLibrary_2.5.0.libv.zip

Unable to execute the jar file, please help me, if there I did any wrong in below steps.

1. Generated .jar using eclipse
2. copied android API 28 jar file to Libscout\lib\android.jar
3. downloaded profiles from https://github.com/reddr/LibScout-Profiles.git and extracted to profiles directory.
4. Copied an apk to the Libscout\apk\123.apk directory
5. Tried the below command
   java -jar build\LibScout.jar --opmode match -a lib\android.jar -p profiles\profiles\md5\Android -d log $apk\123.apk
6. I'm getting an error as below
   Command line parsing failed:
   Argument is no valid file or directory!
   usage: LibScout --opmode match -a <path-to-android.jar> $path-to-app(-dir)

I do not know much about java and mobile app development. So I do not understand terms like android.jar file, library-description-files, etc. however, I would like to run this tool.

I have the apk file of the android app I want to profile. I used the java -jar LibScout.jar -o profile ... command but I do not know what or how to generate the SDK android.jar , library_xml and library_file.

Please help ASAP?

When running LibScout on a directory that has no APKs, it gives an error that made me think that I had wrong command line flags:

$ mkdir /tmp/newdir
$ java -jar build/LibScout.jar -o match -a lib/android-23.jar -p profiles  /tmp/newdir/
 
Command line parsing failed:
You have to provide a path to a single application file or a directory

Hi!
Iv'e ran the crawler of the android repo json file, and downloaded the libs.
Since there are new versions of the libraries, i want to create new profiles for each, and then run the match command on an apk.
I ran the library-profile-generator.sh, and it works fine, but there is not output of the new profiles to anywhere.
Would love some help,
Thank you.

$python3 ./scripts/library-scraper.py ./scripts/library-specs/google-libraries.json


== maven/jcenter scraper ==
- Load library info from ./scripts/library-specs/google-libraries.json
- Store libs to my-lib-repo/
- Update libraries (skip alpha/beta versions):
  # check library com.google.android.play::core [Android]   (g:"com.google.android.play" AND a:"core")
    - retrieved meta data for 31 versions:
       - update version: 1.6.5   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.6.5
       - update version: 1.7.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.0
       - update version: 1.7.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.1
       - update version: 1.7.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.2
       - update version: 1.7.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.3
       - update version: 1.8.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.0
       - update version: 1.8.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.2
       - update version: 1.8.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.3
       - update version: 1.9.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.9.0
       - update version: 1.9.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.9.1
       - update version: 1.10.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.0
       - update version: 1.10.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.1
       - update version: 1.10.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.2
       - update version: 1.10.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.3
  # check library com.android.support.constraint::constraint-layout-solver [Android]   (g:"com.android.support.constraint" AND a:"constraint-layout-solver")
    - retrieved meta data for 10 versions:
       - update version: 1.0.2   type: jar  date: n/a  target-dir: my-lib-repo/Android/com.android.support.constraint::constraint-layout-solver/1.0.2

Hi, I got an error msg when using the Library Detection function (-o match). I downloaded the pre-generated profile files.

The command is:
java -jar build/libs/LibScout.jar -o match -p ../LibScout-Profiles -c config/LibScout.toml -a sdk/android.jar apks/103443com.rarlab.rar.apk

The error msg is:
20:40:04 INFO LibScoutConfig : This is LibScout 2.3.2
20:40:04 INFO Profile : Load library profiles:
20:41:31 INFO Profile : Loaded 8542 profiles in 1 min, 27 sec, 451 ms
20:41:31 INFO Profile :
20:41:31 INFO LibraryIdentifier : Process app: 103443com.rarlab.rar.apk
20:41:31 INFO LibraryIdentifier : = Manifest Parser =
20:41:31 INFO LibraryIdentifier : Package name: com.rarlab.rar
20:41:31 INFO LibraryIdentifier : Version code: 96
20:41:31 INFO LibraryIdentifier : minSdkVersion: 19
20:41:31 INFO LibraryIdentifier : targetSdkVersion: 29
20:41:31 INFO LibraryIdentifier : SharedUserId: - none -
20:41:31 INFO LibraryIdentifier : Library dependencies: - none -
20:41:31 INFO LibraryIdentifier :
20:41:31 INFO LibraryIdentifier : Found 402 unique libraries in 8542 library profiles
using /var/folders/zt/qpw4hbwn0_z6cl37q5v4p5d80000gn/T/name1738988254421587180_classes.dex
20:41:35 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
20:41:35 ERROR TplCLI : com.ibm.wala.ipa.cha.ClassHierarchyException: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
at com.ibm.wala.ipa.cha.ClassHierarchy.(ClassHierarchy.java:296)
at com.ibm.wala.ipa.cha.ClassHierarchy.(ClassHierarchy.java:209)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.make(ClassHierarchyFactory.java:78)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.makeWithRoot(ClassHierarchyFactory.java:89)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.makeWithRoot(ClassHierarchyFactory.java:53)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.createClassHierarchy(LibraryIdentifier.java:112)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:143)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84)
at de.infsec.tpl.TplCLI.main(TplCLI.java:141)

I googled this error msg, it seems to be a runtime error for wala. See (http://wala.sourceforge.net/wiki/index.php/UserGuide:Troubleshooting) The possible solution is to change the java_runtime_dir property in the wala.properties file. However, I guess you didn't use this properties file, right? Please correct me if it's wrong. By the way, I am using java 1.8.0_60.

Thank you very much!

Libscout seems to be detecting Facebook libraries for every application - including the empty project template within Android Studio 3.2.1.
libscout.txt

i cloned LibScout after last commit and when running the command:
java -jar build/libs/LibScout.jar -o match -a lib/android-23.jar -p profiles/ -j ./json/ app-release.apk
the json files are no more created. i already checked privileges issues and gives writing privileges to all for json folder and subfolders and still not working.
i tried it with differents androif-sdk versions.

Hi,
I am scanning a mobile app using LibScout. We are using the okhttp3 library and the version used is 3.10.0. This is done using gradle in Android Studio : 'com.squareup.okhttp3:mockwebserver:3.10.0'.

When I scan this binary using LibScout it reports 3.6.0 version. I was wondering which file is used within okhttp3 to fingerprint the version.

Good afternoon, I'm a bachelor's student in computer science at the Universidade de Brasilia and I'm doing research using libscout as a tool for detecting external libraries. My goal is to verify if the security errors pointed out by the cognicrypt (https://www.eclipse.org/cognicrypt/) and cryptoguard (https://www.cryptoguard.com/) tools are mostly native or external code. I decided to use libscout as a helper tool thanks to the article 'Automated Third-Party Library Detection for Android Applications: Are We There Yet?'. I already got some results, however, talking to my tutor and analyzing some of the tool's outputs, we observed that sometimes the analysis of a certain application registers external libraries only in the "Library Identifier" step and does not show in the "ProfileMatch" step. This is not something that prevents the completion of my undergraduate work, however, it would be interesting to understand why this happens. I've already tried changing the proximity value of the cluster, nevertheless, the library continues to appear only in its identification and not in the match step.
The following attachment illustrates this. Okio3 and HTTP show in the Library Identify step but not in the ProfileMatch.

It is worth adding that I am using the flags to dull a full profile match and that I am using the profile dataset available here on the wiki.

libscout_result.txt

I was trying around different versions of android.jar and i can't figure out the difference when using LibScout.
so i wonder how does the android.jar version used either on profiling or matching affect the result?
what would happend if i create a profile with a given version and try to match it using an other android-X.jar version?

isn't possible to update the database when running
java -jar build/LibScout.jar -o db -p profiles/md5/ -s stats/
because i got this error message
12:19:40 INFO SQLStats : Generate DB from stats mode!
12:19:40 INFO SQLStats : Loaded 4965 profiles from disk
12:19:40 WARN SQLStats : DB file appStats.sqlite exists! -- No updates will be performed -Abort!
so i have to delete the existant file each time i wanna run the command

after i execute a matching test with an APK file and generate the result with the json option, the generated json file didn't contain the comments info about libraries i've put previousely inside library.xml files.

is there any option to get this comments after the tests

LibScout uses JOANA's WALA fork that adopted and improved ScanDroid's dex frontend to allow Android app analyses. At the time of integration, the state was still immature and a lot of integral parts were missing, i.e. multi-dex support. During LibScout's development, some issues have been fixed and workaround patches have been applied to a custom version (lib/wala.jar) that is statically linked during compilation time.

Relying on a custom version is definitively sub-optimal, both in terms of integration and maintainability.
To be able to move from this old and customized WALA to upstream WALA it needs to be checked, whether all features currently used in LibScout (through custom patches etc.) are supported in the WALA master.

I already checked some of LibScout's patches against WALA master. I commented on the following (important) ones that are either unresolved or should already work with the latest WALA version:

• [wala_dex_dexfilemodule_multidex_support.patch]
  Adds multi-dex support to WALA dex frontend that was based on dexlib1
  [Status: resolved] WALA master uses dexlib2 that has builtin support for multi-dex. This would also solve #19 .

• [wala_cha_workaround_unresolvable_superclasses.patch]
  WALA's CHA builder discards classes when their superclasses cannot be resolved. This affects library classes that extend other library classes not being in the classpath. However, the same library class might be resolved within an app where all dependencies are statically linked. This is a workaround patch that forces the CHA builder to keep such classes by changing it's superclass to java.lang.Object to allow code reuse detection.
  [Status: resolved] Addressed in issue wala/WALA#322. Instead of rewriting to Object, WALA's patch creates a synthetic PhantomSuperClass.

• [wala_add_check_synthethic_methods.patch]
  Addition of isMethodSynthetic call as a workaround (see also #16). This became necessary since a proper implementation of isSynthetic for IMethod subclasses (DexIMethod, ShrikeBTMethod,..) caused some unintended side-effects. Did not dig deeper, as this affects all IMethod subclasses.
  [Status: not implemented] WALA master still returns false for isSynthetic calls.

• [wala_dex_ssa_fix_for_methods_without_parameters.patch]
  WALA creates an empty IR for static initializer methods without arguments. This patches fixes the SSA Builder.
  [Status: probably not, needs to be checked]

I'm going to test WALA master and create issues for the open problems.
CC @eighthave