joernio / joern Goto Github PK

Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. Discord https://discord.gg/vv4MH284Hc

Home Page: https://joern.io/

License: Apache License 2.0

Scala 76.44% Shell 0.72% C 0.06% Dockerfile 0.01% SuperCollider 0.06% Assembly 0.03% ANTLR 1.00% Java 20.97% C++ 0.01% JavaScript 0.01% Kotlin 0.03% Python 0.01% PHP 0.59% Batchfile 0.05% HCL 0.01% Slim 0.02%

code-analysis query-language code-browser fuzzy-parsing syntax-tree dataflow controlflow cpg java cpp

joern's People

Stargazers

Watchers

Forkers

clccc lphd zarulshahrin tainaefe uuuunotfound leitingtod japroc sukwon0709 kodoque1 sk1tch hlmu verazuo zenhumany chubbymaggie dukebarman thuanpv z420 derpsteb shuangyinren jingkai92 terry2012 hylkefoeken l0s aimeike wxmandrew ash1n2 zeyu-zh wubozhi crackercat japkeerat trietptm-on-coding-algorithms lily79 thapr0digy neodks askaskn lparis 5l1v3r1 diewufeihong m1cm1c zeph1912 mpollmeier ilwoof colorlight maxwellzk laozhudetui p0p0p0 joernchen r4b3rt davidbakereffendi iamsajidkhan debach zhaojianrui source-code-reading-tools iohexo linshig0ng afwu ratherto rmallof tai-rex wooseunghoon pai-po harite gifrop jiet07 iljavs nashid usama7628674 marciopocebon dmott-eyelock zu1kbackup sanying123 jingqiangliu zhouzhenghui taku65maku zqtang-chn r3aker jadyhill marvinmw drone29a tonghuaroot xee5ch latkes mahmoudzamani infernalheaven rezaduty geekwish zaidozaid strogo zkcod timb-machine-mirrors ekmixon pombredanne charygao li-zhenyuan wyu0hop patricksmith scala-steward maochen-stu popthink tomconner

joern's Issues

Struct Typedef not supported

Hello,

I am trying to parse

#include <stdlib.h>

#define BIG 10000

typedef struct {
    int value;
    struct node *next;
} node;

void free_list(struct node *head) {
    struct node *q;
    int k;
    k = 0;
    for (struct node *p = head; p != NULL; p = q) {
        q = p->next;
        free(p);
	k += BIG;

	for(int q = 0; q <= MID; q+=1) {
	  int g += SMALL;
	}
    }

    if(SMALL < g + q) {

      return BIG;
    }

    else {

      return MID;
    }
}

But the parser thinks that the struct's name is <unnamed>. Is there a way to get the name of this struct? And have the name be "node".

Thanks!

joern-parse error parsing

test.c

void test()
{
  int v1;
  unsigned int v2;

  if ( (*(int (__cdecl **)(signed int))(v1))(v2) != -1 )
  {
  }
  else
  {
  }
}

./joern-parse test.c

Error parsing function test. skipping.

program into a dead loop

My computer configuration environment is :mac os catalina10.15.1 / intellijIDE community2019.2 / last joern version / scala v2.12.10

I wrote two test file bout "Example - CVE-2016-6480 (Linux Kernel)". as follows:

1: first make a cpgloader

class CpgLoad
{
  def loadCpg(files: File): (Cpg, String) =
  {
    val inputFilenames = Set(files.pathAsString)
    val outputFilename = "/tmp/cpg.bin.zip"
    // Create a CPG using the C/C++ fuzzy parser
    val fuzzyc2Cpg = new FuzzyC2Cpg(outputFilename)
    fuzzyc2Cpg.runAndOutput(inputFilenames, Set(".c"))
    // Link CPG fragments and enhance to create semantic CPG
    Cpg2Scpg.run(outputFilename, dataFlow = true, "joern-cli/src/main/resources/default.semantics")
    // Load the CPG
    (CpgLoader.load(outputFilename), outputFilename)
  }

}

2: second I want to run and get the flows , but the program dead loop

class demotest extends WordSpec with Matchers
{
  val files=File(getClass.getClassLoader.getResource("testcode/aacraid"))
  val cpgload=new CpgLoad
  val cpg=cpgload.loadCpg(files)
  val sinkArguments = cpg._1.method.name("copy_from_user").parameter.argument
  println(sinkArguments.reachableByFlows(cpg._1.identifier).p)
}

Support for PHP Extension

The previous version of joern (https://github.com/octopus-platform/joern/tree/dev/projects/extensions/joern-php) looks to have the ability to ingest PHP ast and populate its database, I was wondering if the same thing is possible with currently supported branch of joern or if I should be using the previous version for that feature?

`runScript(filename)` should check if CPG is already loaded

Looks like run(filename) (without the cpg parameter) loads cpg.bin.zip by deafult. This can be quite slow for large graphs. Can we check if a graph is already loaded and use that if so?

Joern's architecture implementation

Hello, I'm reading the source code of joern now, but need a lot of questions, can you provide me how to implement the source code architecture of joern? Or joern's flow chart and technology used, thank you very much

Website: allow linking to subsections

We currently cannot link to subsections in the documentation. It would be nice to have this feature so that we can point people closer to what they're looking for.

Incorrect type for first identifier declaration.

I have a simple test code

#ifndef FUNCTIONS_H
#define FUNCTIONS_H

void call_function();
int glob;

#endif // FUNCTIONS_H

When I generate the CPG and query database for DECLARES relationship, I get type

define FUNCTIONS_H void ( )

of declaration call_function. as shown

Whereas, The type for next variable glob is correct.

COLUMN_NUMBER_END and LINE_NUMBER_END not supported.

According to the schema described in https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/codepropertygraph/src/main/resources/schemas/base.json . Some nodes should have COLUMN_NUMBER_END and LINE_NUMBER_END values, but joern doesn't appear to be following the schema and will not provide COLUMN_NUMBER_END and LINE_NUMBER_END. Maybe I am not looking in the right spot?

Thanks!

Exception on querying some projects

Hello I tried to use joern to analyze a project.
./joern-parse worked successfully
but ./joern-query -f scripts/list-funcs.scala and other script raises Exception.

Meanwhile, when i use sbt semanticcpg/console of project https://github.com/ShiftLeftSecurity/codepropertygraph, it works fine

Full Exception stacktrace:

Exception in thread "main" java.lang.RuntimeException: Edge of type REACHING_DEF with direction OUT not supported by class ControlStructureDb
	at io.shiftleft.overflowdb.OdbNode.storeAdjacentNode(OdbNode.java:419)
	at io.shiftleft.overflowdb.OdbNode.storeAdjacentNode(OdbNode.java:401)
	at io.shiftleft.overflowdb.OdbNode.addEdge(OdbNode.java:248)
	at io.shiftleft.overflowdb.NodeRef.addEdge(NodeRef.java:111)
	at io.shiftleft.passes.DiffGraphApplier.tinkerAddEdge$1(CpgPass.scala:212)
	at io.shiftleft.passes.DiffGraphApplier.$anonfun$addEdges$4(CpgPass.scala:208)
	at io.shiftleft.passes.DiffGraphApplier.$anonfun$addEdges$4$adapted(CpgPass.scala:205)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at io.shiftleft.passes.DiffGraphApplier.addEdges(CpgPass.scala:205)
	at io.shiftleft.passes.DiffGraphApplier.applyDiff(CpgPass.scala:159)
	at io.shiftleft.passes.CpgPass.$anonfun$createApplyAndSerialize$1(CpgPass.scala:80)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:455)
	at scala.collection.Iterator$$anon$20.next(Iterator.scala:882)
	at scala.collection.Iterator$$anon$20.next(Iterator.scala:878)
	at scala.collection.Iterator.foreach(Iterator.scala:937)
	at scala.collection.Iterator.foreach$(Iterator.scala:937)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
	at io.shiftleft.passes.CpgPass.createApplySerializeAndStore(CpgPass.scala:64)
	at io.shiftleft.layers.DataFlowRunner.$anonfun$run$1(DataFlowRunner.scala:13)
	at io.shiftleft.layers.DataFlowRunner.$anonfun$run$1$adapted(DataFlowRunner.scala:13)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at io.shiftleft.layers.DataFlowRunner.run(DataFlowRunner.scala:13)
	at io.shiftleft.joern.CpgLoader$.load(CpgLoader.scala:22)
	at $line5.$read$$iw$$iw.<init>(<console>:14)
	at $line5.$read$$iw.<init>(<console>:21)
	at $line5.$read.<init>(<console>:23)
	at $line5.$read$.<init>(<console>:27)
	at $line5.$read$.<clinit>(<console>)
	at $line5.$eval$.$result$lzycompute(<console>:5)
	at $line5.$eval$.$result(<console>:5)
	at $line5.$eval.$result(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:735)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.callEither(IMain.scala:739)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.evalEither(IMain.scala:754)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.$anonfun$evalEither$3(Scripted.scala:208)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at scala.Console$.withIn(Console.scala:226)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.$anonfun$evalEither$2(Scripted.scala:208)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at scala.Console$.withErr(Console.scala:192)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.$anonfun$evalEither$1(Scripted.scala:207)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at scala.Console$.withOut(Console.scala:163)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.evalEither(Scripted.scala:206)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.$anonfun$eval$1(Scripted.scala:226)
	at scala.tools.nsc.interpreter.Scripted.withScriptContext(Scripted.scala:124)
	at scala.tools.nsc.interpreter.Scripted$WrappedRequest.eval(Scripted.scala:225)
	at scala.tools.nsc.interpreter.Scripted.eval(Scripted.scala:181)
	at io.shiftleft.joern.JoernQuery$.$anonfun$new$1(JoernQuery.scala:19)
	at io.shiftleft.joern.JoernQuery$.$anonfun$new$1$adapted(JoernQuery.scala:8)
	at scala.Option.foreach(Option.scala:257)
	at io.shiftleft.joern.JoernQuery$.delayedEndpoint$io$shiftleft$joern$JoernQuery$1(JoernQuery.scala:8)
	at io.shiftleft.joern.JoernQuery$delayedInit$body.apply(JoernQuery.scala:6)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at io.shiftleft.joern.JoernQuery$.main(JoernQuery.scala:6)
	at io.shiftleft.joern.JoernQuery.main(JoernQuery.scala)

Installation script

Currently, joern is simply unzipped and used from the target directory, as many people do for other JVM-based programs such as IntelliJ and Burp Suite. It would be cleaner though to have an installer that allows us to place joern programs in the path. This will then only simplify packaging for distros.

Fail to run test case

Hi, after successful building joern by sbt stage, I tried to follow the quickstart but get following result.

(py36) ➜  joern git(master):  ./joern-parse  joern-cli/src/test/resources/testcode/free
readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path
You need to run 'sbt stage' first

It looks like it cannot recognize the following parameter is a dirname, however when I tried following command, it still didn't work.

./joern-parse  joern-cli/src/test/resources/testcode/free/free.c

I run this command in joern dir, which absolutely contains the two files.

Does anyone have ideas about this situation? Thx!

Support installation via script on macOS

Attempting to run the joern-install.sh script on macOS produces the following error:

readlink: illegal option -- f
usage: readlink [-n] [file ...]

System information:

sh-3.2$ uname -a
Darwin host 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

Stack Overflow Error

When parsing ftp://ftp.gnu.org/gnu/gsl/gsl-latest.tar.gz (GNU Scientific Library). I get the following Stack Overflow Error using the latest version of Joern:

dev@ubuntu:~/joern$ ./joern-parse ../gsl-2.5/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/dev/Projects/joern/joern-cli/target/universal/stage/lib/org.apache.logging.log4j.log4j-slf4j-impl-2.11.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/dev/Projects/joern/joern-cli/target/universal/stage/lib/org.slf4j.slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Error parsing function gsl_movstat_apply_accum. skipping.
Exception in thread "main" java.lang.StackOverflowError
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1529)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at 
org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at 

....
org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1513)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1496)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1586)
	at org.antlr.v4.runtime.atn.ParserATNSi

Docu-tests for navigating types

We need docu-tests for navigating types, including examples of how to navigate in the type hierarchy and jump to members and methods of classes. We also need to talk about type parameters here.

Update documentation to reflect changes in install/build

We changed the way joern is built (it's now sbt createDistribution as the README says) and how it is installed. We need to update the documentation in docs accordingly.

Pulling Out Code Like In Old Joern

I noticed that LINE_NUMBER_END and COLUMN_NUMBER_END were recently removed from the base schema.

Does this mean users will no longer be able to extract code like the older version of joern supported? In the past you could do the following:

echo "getFunctionsByParameter('*len*').id" | joern-lookup -g | joern-location | joern-code > dump.c

Create releases on each build

It might also nice to have releases so that folks can just download and start working with them. A release could be drafted through Github with compiled tarballs

Include `scripts` in build

We need to ensure that scripts are automatically built via sbt compile and that they are made available to the IDE (IntelliJ) in a way where it knows where to find dependencies. This will make script writing in IntelliJ a lot more effective.

Compiler scripts in `scripts` directory on builds

We want to ensure that scripts continue to work on changes in joern/codepropertygraph. Let's begin by compiling scripts on build, then creating tests for scripts and running them automatically.

#define not supported.

#include <stdlib.h>

#define BIG 10000

typedef struct {
    int value;
    struct node *next;
} node;

void free_list(struct node *head) {
    struct node *q;
    int k;
    k = 0;
    for (struct node *p = head; p != NULL; p = q) {
        q = p->next;
        free(p);
	k += BIG;

	for(int q = 0; q <= MID; q+=1) {
	  int g += SMALL;
	}
    }

    if(SMALL < g + q) {

      return BIG;
    }

    else {

      return MID;
    }
}

The parser does not isolate the #define. The #define is not represented int he code property. Is there a way to get access to #defines?

Argument tainting

Hi!
I started to look at this new version of Joern and did some experiments with simple C code:
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv)
{
char str[80];
strcpy(str, argv[1]);
printf(str);
return 0;
}
I wrote the following in my main.scala:
val arg = cpg.call.name("printf").filter(call => call.argument.code("str")).argument
val result = arg.reachableByFlows(cpg.identifier).p
result.toStream.foreach(x => print(x + "\n"))
I was expecting to get a path from "printf(str)" to the second argument of the main function but without tainting the arguments like I read in the older Joern version I don't think it is possible.
Instead I only got:
| tracked | lineNumber| method| file |
|======================================================|
| printf(str)| 7 | main | tests/mytest/free1.c|

Is still possible to taint arguments with this version of Joern? If so, can you give me some hints on how to do so? Thanks!

Replacements for `joern-location`/`joern-code`

Old-joern had a feature where locations of AST nodes were returned as strings (joern-location). These strings could then be fed to tools such as joern-code to print the code of the AST. We need a similar feature in new-joern. Also see #85

This is the basis for a lot of other scripts I have planed.

Joern quits without reporting errors for errors in predef.sc

When there are problems in predef.sc, joern quits and does not provide any information on the error. This makes it hard to debug predef.sc.

Generate API documentation via scaladoc

In Ocular, we generate API documentation from the code. The Ocular docs, however, are not too useful to Joern users because they point to all sorts of functionality that is only available in Ocular. It would be better if we generated documentation for Joern as well.

Website: navbar shouldn't scale

The navigation bar width, and in particular the logos of ShiftLeft, TUB, and UGO should not be scaled relative to the window size. The logos become way too big.

Error after reproducing kernel examples from joern documentation

I've tried to reproduce kernel examples with joern (https://joern.io/docs/kernelexamples/):

# ./joern-parse ~/linux/drivers/scsi/aacraid/
# ./joern
joern> loadCpg("cpg.bin.zip")
... # commands from documentation
joern> println(sinkArguments.reachableByFlows(cpg.identifier).l.size)
559 # Another number than in docs, maybe it's not an error
... # commands from documentation
joern> reachingDefs1.intersect(reachingDefs2).foreach(elem => println(elem.code))
java.lang.ClassCastException: io.shiftleft.codepropertygraph.generated.nodes.Call cannot be cast to io.shiftleft.codepropertygraph.generated.nodes.Identifier
  scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:321)
  scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:977)
  scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:977)
  ammonite.$sess.cmd8$.<init>(cmd8.sc:1)
  ammonite.$sess.cmd8$.<clinit>(cmd8.sc)

Source/Binary releases

We are in need for scripts to generate source/binary releases. For now, I would say that we do not release on each new merge to master. Instead, let's trigger the release manually.

Docu-tests for navigating method bodies

Any suggestion for Scala compiling is too slow?

Hi there,

We try to test a couple of c++ codes but it takes too much time for compiling. Do you have any suggestion?

Many thanks!

Website: "Edit on Github" button missing in docs

With the previous documentation theme, we had an "Edit on Github" button. That was pretty nice because it allowed to quickly fix bugs in the documentation. Can we get such a button back?

Automatically run `~/.joern/predef.sc` on startup

Since some of the code between the Joern and Ocular shell are shared, we end up picking up the startup file predef.sc in ~/.shiftleft/ocular. That's fine, but we should also pick up ~/.joern/predef.sc.

joern-parse.bat: "input line too long" when building classpath

I ran into problems with the extremely long APP_CLASSPATH variable in joern-parse.bat when running joern-cli on Windows. The command was too long and the script would die when trying to set the classpath. I got around it by simply passing a wildcard (%APP_LIB_DIR%/*) to the -cp argument at the end of the script.

I'd file a PR but I can't find the script in this repo.

Real bug queries

Hi Fabian, thx for your outstanding work.

I have used the joern for C/C++ for several weeks and read the documents, but I cannot find examples for the types of vulnerabilies in the paper. Wonder why u did not push them on the github. Hope u could show them to more developers.

Thank you!

Remove `joern-query`

Since we now have the ammonite-based Joern shell, we no longer require joern-query as the same can be achieved via

loadCpg(...); $query

I would suggest we remove joern-query and all references to it in the documentation. Opinions?

Exception of "GC overhead limit exceeded" occur when prase bintuils 2.28

Hi there,

I try to use the following command ./joern-parse ./binutils-2.28/ to prase the CPG for binutils, but an Exception. Is the CPG extracted in this way complete? What happened?

[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_reloc.

Error parsing function write_rc_dialog_control. skipping.
Error parsing function write_rc_messagetable. skipping.
Error parsing function insert_reg_alias. skipping.
Error parsing function mmix_cons. skipping.
Error parsing function ppc_target_format. skipping.

Error parsing function rx_include. skipping.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_case.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_case.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label default_case.
Error parsing function Arm_exidx_cantunwind::do_fixed_endian_write. skipping.
Error parsing function lay_out_got. skipping.
Error parsing function General_options::parse_R. skipping.
Error parsing function locale_charset. skipping.
Error parsing function lang_output_section_statement_lookup. skipping.
Error parsing function get_DW_TAG_name. skipping.
Error parsing function print_and_abort. skipping.
Error parsing function (extract_number). skipping.
Error parsing function (print_partial_compiled_pattern). skipping.
Error parsing function byte_compile_range. skipping.
Error parsing function truncate_wchar. skipping.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_S.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_S.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_Q.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_P.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_L.
[main] WARN io.shiftleft.fuzzyc2cpg.cfg.AstToCfgConverter$ - Unable to wire goto statement. Missing label case_B.
Error parsing function main. skipping.
Error parsing function zlibCompileFlags. skipping.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at io.shiftleft.proto.cpg.Cpg$PropertyValue$1.parsePartialFrom(Cpg.java:3587)
        at io.shiftleft.proto.cpg.Cpg$PropertyValue$1.parsePartialFrom(Cpg.java:3581)
        at com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2361)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$Property.<init>(Cpg.java:8143)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$Property.<init>(Cpg.java:8095)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$Property$1.parsePartialFrom(Cpg.java:8751)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$Property$1.parsePartialFrom(Cpg.java:8745)
        at com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2361)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node.<init>(Cpg.java:7387)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node.<init>(Cpg.java:7331)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$1.parsePartialFrom(Cpg.java:9555)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$Node$1.parsePartialFrom(Cpg.java:9549)
        at com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2361)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct.<init>(Cpg.java:7236)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct.<init>(Cpg.java:7192)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$1.parsePartialFrom(Cpg.java:12744)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct$1.parsePartialFrom(Cpg.java:12738)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:215)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:232)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:237)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
        at com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:317)
        at io.shiftleft.proto.cpg.Cpg$CpgStruct.parseFrom(Cpg.java:11950)
        at io.shiftleft.codepropertygraph.cpgloading.ProtoCpgLoader.getNextProtoCpgFromStream(ProtoCpgLoader.java:193)
        at io.shiftleft.codepropertygraph.cpgloading.ProtoCpgLoader.loadFromProtobufDirectory(ProtoCpgLoader.java:137)
        at io.shiftleft.codepropertygraph.cpgloading.ProtoCpgLoader.loadFromProtoZip(ProtoCpgLoader.java:43)
        at io.shiftleft.codepropertygraph.cpgloading.CpgLoader.load(CpgLoader.scala:68)
        at io.shiftleft.codepropertygraph.cpgloading.CpgLoader$.load(CpgLoader.scala:38)
        at io.shiftleft.joern.CpgLoader$.load(CpgLoader.scala:20)
        at io.shiftleft.joern.Cpg2Scpg$.run(Cpg2Scpg.scala:14)
        at io.shiftleft.joern.JoernParse$.parse(JoernParse.scala:24)
        at io.shiftleft.joern.JoernParse$.$anonfun$new$1(JoernParse.scala:13)

Make `joern-query`/`joern-parse` use `joernd`

joern-query and joern-parse are currently scala scripts. In addition to the JVM startup time, joern-query spends considerable time loading the graph. With the new cpgclientlib, we can now replace joern-query and joern-parse with Python scripts that interact with joernd. As a result, JVM startup time is no longer a problem and we can keep loaded graphs cached in joernd.

scalafmt

We don't have automated code formatting installed here yet.

Scripts for AST, CFG, and PDG extraction

Let's create scripts that will extract ASTs, CFGs, and PDGs for all functions and write them to a file. This will enable the research community to use Joern as a tool for the extraction of graph-based intermediate program representations via the fuzzy parser.

Exception: Invalid request: <Response [400]>

I follow the Importing Code part in the Joern Documentation, but when I try to create Code CPG with cpg-create, the following problems were encountered,''Exception: Invalid request: <Response [400]>''.

I follow the Document step by step, and I don't know what's wrong with it, I would appreciate it if you could help me out

How to get AST, CFG graph

Hi, may I ask how can I get AST, CFG graph with this tool. We can just use the simple interfaces supplied by queryprimitives.

More informative `-help` message for `joern`

When running joern -help, we end up mainly with information about the JVM as opposed to information about --script and --params as one would except. Let's make sure that when a user types joern -h or joern --help, a cleaned up info text appears.

Add `updateDependencies` script

In other CS repos, we have an updateDependencies script, which automatically updates dependencies to its newest versions. Can we get the same here?

Crash on parsing of Blender 2.8

To reproduce, run ./joern-parse on the blender source code at: https://github.com/sobotka/blender/releases/tag/v2.80

[main] ERROR io.shiftleft.joern.JoernParse$ - Failed to enhance CPG.
java.util.EmptyStackException
	at java.util.Stack.peek(Stack.java:102)
	at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.enterTemplate_decl(CModuleParserTreeListener.java:113)
	at io.shiftleft.fuzzyc2cpg.ModuleParser$Template_declContext.enterRule(ModuleParser.java:3505)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:42)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
	at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.walkTree(AntlrParserDriver.java:177)
	at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.parseAndWalkTokenStream(AntlrParserDriver.java:123)
	at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.parseClassContent(CModuleParserTreeListener.java:198)
	at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.exitDeclByClass(CModuleParserTreeListener.java:167)
	at io.shiftleft.fuzzyc2cpg.ModuleParser$DeclByClassContext.exitRule(ModuleParser.java:2174)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
	at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.walkTree(AntlrParserDriver.java:177)
	at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.parseAndWalkFile(AntlrParserDriver.java:78)
	at io.shiftleft.fuzzyc2cpg.FuzzyC2Cpg.createCpgForCompilationUnit(FuzzyC2Cpg.scala:171)
	at io.shiftleft.fuzzyc2cpg.FuzzyC2Cpg.$anonfun$runAndOutput$1(FuzzyC2Cpg.scala:87)
	at io.shiftleft.fuzzyc2cpg.FuzzyC2Cpg.$anonfun$runAndOutput$1$adapted(FuzzyC2Cpg.scala:87)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.parallel.immutable.ParHashSet$ParHashSetIterator.foreach(ParHashSet.scala:81)
	at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:974)
	at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
	at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
	at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
	at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:971)
	at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:170)
	at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
	at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
	at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
	at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
	at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
	at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
	Suppressed: java.util.EmptyStackException
		... 43 more
		Suppressed: java.util.EmptyStackException
			at java.util.Stack.peek(Stack.java:102)
			at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.enterTemplate_decl(CModuleParserTreeListener.java:113)
			at io.shiftleft.fuzzyc2cpg.ModuleParser$Template_declContext.enterRule(ModuleParser.java:3505)
			at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:42)
			at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
			at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
			... 28 more
			Suppressed: java.util.EmptyStackException
				at java.util.Stack.peek(Stack.java:102)
				at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.enterTemplate_decl(CModuleParserTreeListener.java:113)
				at io.shiftleft.fuzzyc2cpg.ModuleParser$Template_declContext.enterRule(ModuleParser.java:3505)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:42)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
				at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.walkTree(AntlrParserDriver.java:177)
				at io.shiftleft.fuzzyc2cpg.parser.AntlrParserDriver.parseAndWalkTokenStream(AntlrParserDriver.java:123)
				at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.parseClassContent(CModuleParserTreeListener.java:198)
				at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.exitDeclByClass(CModuleParserTreeListener.java:167)
				at io.shiftleft.fuzzyc2cpg.ModuleParser$DeclByClassContext.exitRule(ModuleParser.java:2174)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47)
				at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30)
				... 28 more
				Suppressed: java.util.EmptyStackException
					at java.util.Stack.peek(Stack.java:102)
					at io.shiftleft.fuzzyc2cpg.parser.modules.CModuleParserTreeListener.enterTemplate_decl(CModuleParserTreeListener.java:113)
					at io.shiftleft.fuzzyc2cpg.ModuleParser$Template_declContext.enterRule(ModuleParser.java:3505)
					at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:42)
					at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
					at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
					... 28 more

Port old malloc-to-memcpy queries

Here are some queries for old-joern that we should showcase for new joern:
https://github.com/vlad902/joern-traversals/blob/master/queries/malloc_to_memcpy.yaml

Global variables not detected

int x;

int main() {
    return 0;
}

shows no reference to variable x when run through joern.

Looking at fuzzyc2cpg 1 and 2, it appears this should be handled.

How to transfer the python program from the old joern to the shiftleft-joern quickly?

I have wrote some python programs running on the old Joern framework, they have some bugs but hard to debug because of REST. So I want to rewrite them with the new Joern.
The new version is more powerful and complicated from the old one. I want to know is the Joern completed basically? Is it the right time to start the rewrite work now?
When I learned the new version, I felt the documentation and samples are bit sparse. Although I have used the old Joern, but I felt difficult to understand the schema of the new Joern clearly. Would you provide more detailed documentation.
I'm eager to experience the power of the shiftleft-joern.
Thanks for the great work very very much!
@fabsx00 @mpollmeier @ml86 @

Travis build for joern

codepropertygraph and fuzzyc2cpg are built by travis but the integration project joern is not. As we are adding integration tests to this repo, we want automated builds here as well. Ideally, the build also generates API documentation and passes it via a webhook to joern.io - if that's possibly. Fun fact: generating the documentation on a 1GB DO droplet is not possible.

Support Scala Native

To keep the voracious JVM in the cage.

Scala Native is an optimizing ahead-of-time compiler and lightweight managed runtime designed specifically for Scala. It features:

Low-level primitives.
Seamless interop with native code.
Instant startup time.

They are working on the upcoming 0.4.0 release right now.

Docu-tests for navigating the control flow graph

We need docu-tests to show how to operate directly on the control flow graph.

joernio / joern Goto Github PK

joern's People

Stargazers

Watchers

Forkers

joern's Issues

Recommend Projects

Recommend Topics

Recommend Org