Better support for external functions about mcsema HOT 5 CLOSED

Yeah the current approach is pretty awful. In granary, I had a way of extracting and wrapping most functions from the Linux kernel and libc. I even started to use the same script to generate the std_defs.txt files from pre-processed system header files. Still, the current format leaves a lot to be desired. I think we can still adapt this script though.

Here's the gist of the idea, and it builds directly off of what granary did: use the cparser script to generate a header file containing all system function declarations, and their dependent types (this is doable, modulo a few parse errors). An alternative, possibly better solution is to use Clang for this. So we want to produce a bitcode file with every external as a declared function, then link this in with lifted bitcode, and find a way to adapt the arguments (similar to right now).

Var args are indeed a problem. Right now we punt on it and pretend that functions like printf take 12 to 16 arguments. It's kind of ugly.

In terms of inline assembly, what do you mean? The easiest solution by far is not to model argument passing explicitly. Instead, use the existing detach mechanisms with a callback. This would avoid the problem entirely, solving it by doing the usual marshalling of the reg state struct into native state.

from mcsema.

The detach mechanism is what I meant by inline assembly (I forgot that portion of the code existed).

Since the function definition list is used in get_cfg.py, I'm hesitant to actually suggest a bitcode file as the way to represent this data in an interchange format, but the LLVM function prototypes definitely seem the way to go.

from mcsema.

I don't think the bitcode vs. the function list approaches are mutually exclusive. That is, cfg_to_bc can consult whatever was put into the CFG proto by get_cfg.py (via the text defs), but it if has better information from a prototype in its bitcode, then it could prefer that.

This will bring mcsema slightly closer to remill, insofar as remill takes a "base" input bitcode module to start with (in remill's case, it contains all instruction semantics), and extends it from there. In this case, cfg_to_bc could be instruction to take in a path to a bitcode file, and lift into that, rather than creating a new llvm::Module.

Thoughts?

from mcsema.

The master branch, when using --explicit_args has available a new option, --library /path/to/bitcode.bc, which is an IR or bitcode file containing declarations of the lifted program's externals.

from mcsema.

Closing this external functions have changed significantly, and ABI libraries (now in progress) should take care of complex argument scenarios.

from mcsema.