Git Product home page Git Product logo

respire's Introduction

Respire

C/C++ CI

Introduction

Respire provides logic to describe and execute an arbitrary file dependency graph. It is a build system tool, like make, however unlike make it offers the full power of Python to setup build rules enabling a more high level description of a dependency graph. Respire is compatible with both Python 2 and 3.

SCons is a similar build system tool that offers the expressiveness of Python to describe a build dependency graph, however SCons is known to be slow to execute. On the other hand, Respire caches the results of executed Python scripts into easy to parse low-level file dependencies so that unless the Python logic is modified, the Python interpreter will not need to be invoked. This low-level cached file format (currently stored as JSON) serves a similar role as the input to the ninja build system, in that it's meant to be parsed quickly, and not to be analyzed by humans.

A core set of functionality is provided enabling direct specification of a per-file dependency graph. It can be tedious to define large projects in this way, so higher-level utilities are available on top of the core functionality. This enables, for example, C++ projects to be defined in terms of modules, like executables, static libraries and dynamic libraries. No magic is involved in the creation of these libraries, and so anyone is free to build their own in order to create a domain specific language for specifying build targets for custom tools or other programming languages.

Demos

Executing Respire on a project for the first time

Here is an animation showing what happens when Respire is used to build the Respire core project itself. Note the -g command line parameter which enables a visualization of the build process which will open and render in a browser as a web page.

Animation of Respire's build dependency graph while building.

Note that in the above animation there are two phases:

  1. The beginning where the Python scripts which define the build dependency graph are executed.
  2. The second phase where the actual build instructions are executed.

In the first phase, the dependency graph can be seen in the process of updating, as more Python is executed and the results are collected by the Respire core engine.

In the next animation, starting from where we ended in the first animation we see Respire being invoked to run the unit and end-to-end tests packaged with Respire itself.

Note also that multiple tasks are executed in parallel, one per thread available on the system.

The grey nodes are nodes that are defined in the build files and available for building, but were not requested to be built at that time. In this example, they are packaging and testing steps. The next animation goes on to run those build steps.

Re-running respire to build tests and packaging

Animation of Respire's build dependency graph while packaging and testing.

Note that:

  • We do not rebuild items that had already been built and whose dependencies have not changed.
  • We do not have to wait for any Python to execute this time (e.g. "phase 1" from above), because the results from the Python execution have been cached from the first run.

In this second example, we are packaging the Respire project and running the tests, both tasks which require the Respire core project to be built already, as was done in the first animation.

The two circles of nodes on the right represent the packages, where most of the nodes represent Python file copies, where we copy the Python source code representing the Respire environment into a directory that also contains the Respire core executable. Two copies are made, one for packaging and shipping, and one for testing.

The testing one is used to enable end-to-end tests where test Python build steps are run, which ultimately invoke Respire core, and the resulting file system state is tested for correctness.

Building Respire

While Respire can be built by Respire, it can also be built using CMake, in order to bootstrap the process.

Build with CMake

There is a CMakeLists.txt file in the src/ directory. Respire is confirmed to be buildable on Windows using MSVC and on Linux with g++.

Once built, you will need to install the result, e.g. by running

cmake --build . --target install

and then add the ${CMAKE_OUT}/package/python directory to your PYTHONPATH environment variable.

Typically, Respire projects provide a build entry point Python script which must be executed in order to kick off the build. See src/example/SimpleMultiSourceCPP/build.py for an example entry point script.

Build with Respire

Once Respire is installed on a system (e.g. see Build with CMake), then Respire can be built using itself by running:

python src/build.py

By default, the output will appear in a newly created out/ folder. By default a debug build is generated and the Respire binary executable is not packaged with the associated Python files. In order to produce the final packaged output, call:

python src/build.py -t package

which will output the results into the directory out/debug/package (e.g. you could set your PYTHONPATH to this directory in order to use this new instance of Respire for future Respire builds).

In order to build a release build of respire, and package the results, call

python src/build.py -t package -c release

and the results will be available in out/release/package.

Example

Small C++ Project

The directory src/example/SimpleMultiSourceCPP contains a small example of how to use Respire's high level C++ layer to define a small C++ project. For convenience, the build file for this example project is inlined here:

import respire.buildlib.cc as cc 
import respire.buildlib.cc_toolchains.discovery as cc_discovery
import respire.buildlib.modules as modules

def EntryPoint(registry, out_dir):
  toolchain = cc_discovery.DiscoverHostToolchain()
  configured_toolchain = cc.ToolchainWithConfiguration(
      toolchain, cc.Configuration())

  main_module = modules.ExecutableModule(
      'simple_multi_source_cpp', registry, out_dir, configured_toolchain,
      sources=[
        'main.cc',
        'database.cc',
        'database.h',
      ])

  for output in main_module.GetOutputFiles():
    registry.Build(output)

respire's People

Contributors

aabtop avatar

Watchers

 avatar

respire's Issues

Consider introducing a build database for improved "has file been modified" checks.

The article https://apenwarr.ca/log/20181113 goes in to detail on why using only the file modification time would produce both false positives and false negatives on whether a file is out of date or not. One way to address this is to introduce the concept of a build database where we track more information about the last time a file was built (such as file size, tracking the generation of generated file, etc...). Respire currently only uses the file modification time to check if a re-build is necessary.

The advantages of implementing this are to have more correct builds, and fewer re-builds of files that do not need to be re-built. The disadvantages are that it adds a considerable complication and centrality (though perhaps it can be decentralized on a per-registry-file basis) to the build. Currently, Respire (core) does not maintain any "special" files or databases when running. It's not clear to me if the pros outweigh the cons here, but the article above is really interesting so I wanted to at least track it.

Here is where we decide if a file needs to be re-built:

bool should_rebuild = AnyInputNewerThanOutputs(
, and poking around there you'll see that it ultimately just depends on file modification times.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.