Git Product home page Git Product logo

loom's Introduction

Loom: LLVM instrumentation library

Loom is a general-purpose library for adding instrumentation to software in the LLVM intermediate representation (IR) format. It is currently capable of generating static instrumentation points that expose values (e.g., function parameters and return values) to software-defined instrumentation functions.

Loom is part of the CADETS (Causal, Adaptive, Distributed, and Efficient Tracing System) project as part of DARPA's Transparent Computing program.

Get it

Loom itself is entirely contained within this GitHub repository, but it requires that you build LLVM from source. In order to run Loom's test suite, you must also build a matching version of Clang.

$ git clone https://github.com/cadets/llvm -b cadets llvm
$ git clone https://github.com/cadets/clang -b cadets llvm/tools/clang
$ git clone https://github.com/cadets/loom loom

Build it

Building Clang/LLVM

First, build LLVM using CMake and Ninja. The directions below assume a Release build, but a Debug build is also possible.

$ mkdir llvm/Release
$ cd llvm/Release
$ cmake -G Ninja -D CMAKE_BUILD_TYPE=Release ..
$ ninja
[go get some coffee]

Building LOOM

  1. Set your PATH to include the version of LLVM you just built:
$ export PATH=/path/to/LLVM/Release/bin:$PATH

or:

$ setenv PATH /path/to/LLVM/Release/bin:$PATH
  1. Configure Loom build:
$ cd /path/to/Loom
$ mkdir Release
$ cd Release
$ cmake -G Ninja -D CMAKE_BUILD_TYPE=Release ..
  1. Build Loom:
$ ninja
  1. (optional) run test suite:

    a. Install libxo, which is used by Loom to emit structured output:

$ pkg install libxo    # or whatever your package manager command is
b. Run the Loom test suite:
$ ninja check

Use it

opt pass

Loom is structured as a set of libraries for LLVM instrumentation, but can be run most straightforwardly as an LLVM opt pass. This requires creating an instrumentation policy file such as:

#
# There are two instrumentation strategies:
#  * callout:  call an instrumentation function like __loom_called_foo
#  * inline    inject instrumentation inline with instrumented events
#
strategy: callout

#
# When using callout instrumentation, the generated instrumentation functions
# are named __loom_{event-specific-name} by default. This configuration value
# allows the `__loom` prefix to be overridden.
#
hook_prefix: __test_hook

#
# Loom can automatically log events and their immediate values (e.g., when
# logging a call to foo(1, 2, 3.1415), emit "foo", 1, 2 and 3.1415) without
# any processing. Strategies include:
#
#  * none      do not use the simple logger
#  * printf    just print the event using printf()
#  * xo        use libxo to emit a structured representation (e.g., json)
#
logging: printf

#
# Loom can report events via FreeBSD's ktrace(1) mechanism, either from
# the kernel ("kernel") or from userspace via the utrace(2) system call.
#
ktrace: utrace

#
# Loom currently serializes event information for, e.g., ktrace reporting
# using libnv. Support for other serialization schemes can be added.
#
serialization: nv

#
# Specify how/when functions should be instrumented.
#
# Functions can be instrumented on the caller side (before/after the call)
# or the callee side (function prelude / return sites). This configuration
# value should be a list of entries, each specifying:
#
#  * `name`: the name of the function being instrumented (language-mangled)
#  * `caller`: (optional) list of directions to instrument calls (entry/exit)
#  * `callee`: (optional) list of directions to instrument functions
#
functions:
    - name: foo
      caller: [ entry, exit ]

    - name: bar
      caller: [ entry ]
      callee: [ exit ]

#
# Specify how/when structure fields should be instrumented.
#
# We can instrument both reads and writes to structure fields.
# These fields must currently be identified by number rather than name.
# The `structures` config value is a list of entries containing:
#
#  * `name`: the structure name
#  * `fields`: a list of structure field descriptions:
#    * `name`: field name
#    * `operations`: list of operations to instrument (`read` or `write`)
#
structures:
  - name: baz
    fields:
      - name: refcount
        operations: [ read, write ]

and then running opt:

$ opt -load /path/to/LLVMLoom.so -loom -loom-file /path/to/instr.policy

Loom's opt pass has options that can be seen in the help/usage output:

$ opt -load /path/to/LLVMLoom.so -help | grep loom

Instrumenting FreeBSD

To build FreeBSD with Loom, you need to clone our IR-augmented version of FreeBSD. Then, you can use our loom-fbsdmake script as a drop-in wrapper around make:

$ git clone https://github.com/cadets/freebsd -b cadets freebsd-cadets
$ cd freebsd-cadets
$ export LLVM_PREFIX=/path/to/LLVM/Release
$ export LOOM_LIB=/path/to/Loom/Release/lib/LLVMLoom.so
$ /path/to/Loom/scripts/loom-fbsdmake buildworld buildkernel
$ /path/to/Loom/scripts/loom-fbsdmake buildenv   # etc.

loom's People

Contributors

amstrnad avatar bkidney avatar leestephenn avatar trombonehero avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

loom's Issues

Revist use of RegEx in file scope matching

Currently, when a filename is specified to limit the scope of instrumentation it matches using regular expressions. This may cause unexpected behaviour and should be revisited in the future.

Simple FBT

Perform function boundary tracing on simple, easy-to-trace functions (e.g., no structures passed by value or tail calls).

See 133742e for a FBT test (currently XFAIL).

Function name wildcards in policy files

Allow policy files to describe what to do on sets of functions matched by... globs? Full regexes?

This is possible because we don't have to enumerate all of the values passed to instrumentation.

'loom-nv-debug' registered more than once

I receive this error while running loom.

opt: CommandLine Error: Option 'loom-nv-debug' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

I built Loom using LLVM 3.8. Where I used a precompiled version and I built LLVM from source for building Loom.

I used this command for building Loom
cmake -G Ninja -DCMAKE_CXX_COMPILER=lvm/Release/bin/clang++ -DCMAKE_C_COMPILER=/llvm/Release/bin/clang-3.8 -D CMAKE_BUILD_TYPE=Release ..

Using LOOM on Ubuntu

Hi,

I am using LOOM on Ubuntu and had some questions. I have created a fork of LOOM here.

There is a small program which I am building and executing using the script here and it is working fine. To build, there are minor differences when compared with tests in LOOM here.

Using the file LOOM policy file here gives the error in the file here. The change that causes the error to be generated is the function name '[a-zA-Z0-9_]+' in the policy file. The reason for using that function name is to instrument all functions.

  1. Can you please give me any insight on what could be causing the issue mentioned above?
  2. In the function here, is it possible to get the values of strings logged (for example, "hello") rather than the value of the pointer (for example, 0xabcdef)?

Thank you.

ninja failed

$ ninja
[7/14] Building CXX object src/CMakeFiles/LLVMLoom.dir/Strings.cc.o
FAILED: src/CMakeFiles/LLVMLoom.dir/Strings.cc.o 
/usr/bin/c++   -DGTEST_HAS_RTTI=0 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/nobackup/llvm/include -I/nobackup/llvm/Release/include -fPIC -fvisibility-inlines-hidden -
Werror=date-time -std=c++11 -w -fdiagnostics-color -ffunction-sections -fdata-sections -std=c++11 -Wall -O3 -DNDEBUG -fPIC   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS  -fno-exceptio
ns -fno-rtti -MMD -MT src/CMakeFiles/LLVMLoom.dir/Strings.cc.o -MF src/CMakeFiles/LLVMLoom.dir/Strings.cc.o.d -o src/CMakeFiles/LLVMLoom.dir/Strings.cc.o -c ../src/Strings.cc
../src/Strings.cc: In function ‘std::__cxx11::string loom::Join(const std::vector<std::__cxx11::basic_string<char> >&, const string&)’:
../src/Strings.cc:42:3: error: ‘ostream_iterator’ was not declared in this scope
   ostream_iterator<string> Iterator(Stream, Delimiter.c_str());
   ^
../src/Strings.cc:42:26: error: expected primary-expression before ‘>’ token
   ostream_iterator<string> Iterator(Stream, Delimiter.c_str());
                          ^
../src/Strings.cc:42:62: error: ‘Iterator’ was not declared in this scope
   ostream_iterator<string> Iterator(Stream, Delimiter.c_str());
                                                              ^
[9/14] Building CXX object src/CMakeFiles/LLVMLoom.dir/PolicyFile.cc.o
FAILED: src/CMakeFiles/LLVMLoom.dir/PolicyFile.cc.o 
/usr/bin/c++   -DGTEST_HAS_RTTI=0 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/nobackup/llvm/include -I/nobackup/llvm/Release/include -fPIC -fvisibility-inlines-hidden -
Werror=date-time -std=c++11 -w -fdiagnostics-color -ffunction-sections -fdata-sections -std=c++11 -Wall -O3 -DNDEBUG -fPIC   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS  -fno-exceptio
ns -fno-rtti -MMD -MT src/CMakeFiles/LLVMLoom.dir/PolicyFile.cc.o -MF src/CMakeFiles/LLVMLoom.dir/PolicyFile.cc.o.d -o src/CMakeFiles/LLVMLoom.dir/PolicyFile.cc.o -c ../src/PolicyFile.cc
../src/PolicyFile.cc:147:14: error: specialization of ‘template<class T, class EnableIf> struct llvm::yaml::SequenceTraits’ in different namespace [-fpermissive]
 struct yaml::SequenceTraits<vector<T>> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:200:8: error:   from definition of ‘template<class T, class EnableIf> struct llvm::yaml::SequenceTraits’ [-fpermissive]
 struct SequenceTraits {
        ^
../src/PolicyFile.cc:159:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<InstrStrategy::Kind> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:168:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<Policy::KTraceTarget> {
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:178:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<Policy::Direction> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:187:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<SerializationType> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:196:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<SimpleLogger::LogType> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:206:14: error: specialization of ‘template<class T> struct llvm::yaml::MappingTraits’ in different namespace [-fpermissive]
 struct yaml::MappingTraits<FnInstrumentation> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:54:8: error:   from definition of ‘template<class T> struct llvm::yaml::MappingTraits’ [-fpermissive]
 struct MappingTraits {
        ^
../src/PolicyFile.cc:216:14: error: specialization of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ in different namespace [-fpermissive]
 struct yaml::ScalarEnumerationTraits<FieldOperation> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:99:8: error:   from definition of ‘template<class T> struct llvm::yaml::ScalarEnumerationTraits’ [-fpermissive]
 struct ScalarEnumerationTraits {
        ^
../src/PolicyFile.cc:225:14: error: specialization of ‘template<class T> struct llvm::yaml::MappingTraits’ in different namespace [-fpermissive]
 struct yaml::MappingTraits<FieldInstrumentation> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:54:8: error:   from definition of ‘template<class T> struct llvm::yaml::MappingTraits’ [-fpermissive]
 struct MappingTraits {
        ^
../src/PolicyFile.cc:234:14: error: specialization of ‘template<class T> struct llvm::yaml::MappingTraits’ in different namespace [-fpermissive]
 struct yaml::MappingTraits<StructInstrumentation> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:54:8: error:   from definition of ‘template<class T> struct llvm::yaml::MappingTraits’ [-fpermissive]
 struct MappingTraits {
        ^
../src/PolicyFile.cc:243:14: error: specialization of ‘template<class T> struct llvm::yaml::MappingTraits’ in different namespace [-fpermissive]
 struct yaml::MappingTraits<PolicyFile::PolicyFileData> {
              ^
In file included from ../src/PolicyFile.cc:40:0:
/nobackup/llvm/include/llvm/Support/YAMLTraits.h:54:8: error:   from definition of ‘template<class T> struct llvm::yaml::MappingTraits’ [-fpermissive]
 struct MappingTraits {
        ^
[13/14] Building CXX object src/CMakeFiles/LLVMLoom.dir/OptPass.cc.o
ninja: build stopped: subcommand failed.

Structures passed by value

LOOM should be able to instrument functions that accept structures by value or use sret. The details of how to do this may be ABI-dependant. Steps:

  1. write tests for structure-by-value parameters and (separately) for sret,
  2. explore ABI-dependence of Clang output and
  3. teach LOOM how to deal with these details.

Structure field access

LOOM should be able to instrument all accesses (reads and write) to structure fields, at least when software is playing by the rules of the type. Ideally we'd be able to track pointers to such fields using escape analysis and information flow propagation techniques from SOAAP, but at the very least, we should warn when "the rules" are broken.

There may be implications here for policy descriptions (including the PolicyFile class).

Loom not instrumenting with latest Cadets LLVM

I downloaded the most recent version of Loom and built the most recent version of LLVM from https://github.com/cadets/llvm/tree/cadets and https://github.com/cadets/clang/tree/cadets. While it builds fine, when I try to run it on a small file, I get the error:
.../llvm-loom/Release/bin/opt: symbol lookup error: .../loom-raw/Release/lib/LLVMLoom.so: undefined symbol: _ZNK4llvm5Twine3strEv

In that and the rest of the commands, '...' stands for the path on my machine to the folder where I installed them.

The exact command I used was
.../llvm-loom/Release/bin/opt -load .../loom-raw/Release/lib/LLVMLoom.so -loom -loom-file really_small.policy -o Instreally_small.bc < really_small.opt

I have attached really_small.policy and really_small.opt, which was generated from the attached really_small.c. (All of them have .txt added to their file names to get GitHub to attach them.)

Note that my primary work uses an old version of Loom, so I can continue it while this remains open.

really_small.c.txt
really_small.opt.txt
really_small.policy.txt

DTrace output

Write a LOOM DTrace provider that can expose events from LOOM instrumentation. This will require dynamic registration of DTrace probes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.