ccbrown / needy Goto Github PK

View Code? Open in Web Editor NEW

54.0 4.0 4.0 550 KB

A C++ library dependency helper.

Home Page: https://ccbrown.github.io/needy/

License: MIT License

Python 99.93% Shell 0.07%

python cpp dependancy-manager cli

needy's Introduction

Needy

Needy is tool that aims to make C++ library dependencies as magical as possible. Dependencies are declared in a file known as the "needs file", usually by simply adding a source URI. Then Needy will download and build those dependencies for you.

For example, by creating a needs.yaml file in your project that looks like this:

libraries:
    catch:
        repository: [email protected]:philsquared/Catch.git
        commit: v1.3.0

You can then use a simple command invocation (needy satisfy) to download and build Catch for your target platforms. Once integrated with your build system, adding, updating, or modifying dependencies in any way becomes a trivial matter.

Needy is extremely capable, so be sure to check out the examples directory or the documentation to see some more things you can do.

needy's People

Contributors

Stargazers

Watchers

Forkers

vmrob bittorrent pombredanne

needy's Issues

Needy as a language-agnostic dependency builder

It seems to me that, while Needy is geared towards the build/configuration of C/C++ applications, it is almost entirely language agnostic and could easily handle other languages by supporting their build systems.

I'm thinking, at this point, that this trend should continue so that it really can be used as a language-agnostic dependency build tool. This ticket isn't a call to action, but instead an invitation to discuss the possibility of using needy for those types of uses. An example use case might be setting up environments for python projects using virtualenv.

Assert that somefile.x is present after building.

If there's a problem with a build, this will save time by erroring out immediately, and making it really clear where the error occurred. Usage might look something like...

libraries:
    hiredis:
        repository: [email protected]:redis/hiredis.git
        commit: v0.13.2
        project:
            ensures: lib/libhiredis.a

way to specify installed pre-requisites

for example, if a build requires clang 3.6 to be installed, maybe you could specify that with something like prerequisites: ["clang:3.6+"]

Better handling of artifacts with incorrect checksums

When Needy finds that an artifact has an incorrect checksum but hasn't downloaded it, it should attempt to re-download it after determining the checksum is incorrect.

A related issue is that needy doesn't appear to try to re-download artifacts if the checksum is declared the same between changes in build configuration. Specifically, if the download URL changes, Needy should probably re-download the artifact.

Needy error when changing from git repository to directory

When Needy changes from repository to directory, it doesn't delete the previous source folder leading to os exceptions during the build.

windows support

We need to make sure it works.

Add pkgconfig paths to dependent libraries

As an example, if a library depends on another and knows how to use .pc files, needy should automatically include that path in the PKG_CONFIG_PATH environment variable for the dependent library.

Example:

ffmpeg:
    repository: git://source.ffmpeg.org/ffmpeg.git
    commit: n3.0.2
    dependencies:
        - x264
    project:
        environment:
            PKG_CONFIG_PATH: '{{ build_directory('x264') }}/lib/pkgconfig:{current}'
        build-steps:
            - ./configure --prefix={build_directory} --disable-yasm --enable-gpl --enable-libx264
            - make -j8 install

Ideally, PKG_CONFIG_PATH would not need to be overridden.

ffmpeg:
    repository: git://source.ffmpeg.org/ffmpeg.git
    commit: n3.0.2
    dependencies:
        - x264
    project:
        build-steps:
            - ./configure --prefix={build_directory} --disable-yasm --enable-gpl --enable-libx264
            - make -j8 install

Add remote cache implementation

Needy should be able to publish built artifacts to s3 and should probably use a redis server to implement the required file locking. Additionally, needy should support lock-free read-only of the same s3 cache.

The use case here is that if a development team has a ci server they wouldn't mind receiving pre-built artifacts from, they could have read-only access to the s3 cache for published artifacts while the ci server provides those artifacts to s3.

Add caching of sources

For downloads, caching should be trivial. For directory sources, see #102. For git sources, externalizing the git directory and storing that might be the easiest thing to do.

Need a way to declare external dependencies in libraries for generated build files

As an example, we find ourself having to do this in our Jamfiles quite a bit. It would be great it we could specify these external dependencies in needy so that when we did generate our relevant xcconfig and jamfiles, we reduced duplication and centralized the definitions of these libraries and their dependencies.

lib libSDL2 : ../third-party/needs//sdl2
:
    <name>SDL2
: :
    [ conditional <target-os>darwin :
        <framework>AudioUnit
        <framework>Carbon
        <framework>Cocoa
        <framework>CoreAudio
        <framework>ForceFeedback
        <framework>Foundation
        <framework>IOKit
        <source>libiconv
    ]
    [ conditional <target-os>appletv :
        <framework>CoreAudio
        <framework>CoreGraphics
        <framework>GameController
        <framework>QuartzCore
        <framework>UIKit
    ]
    [ conditional <target-os>iphone :
        <framework>CoreAudio
        <framework>CoreGraphics
        <framework>CoreMotion
        <framework>GameController
        <framework>QuartzCore
        <framework>UIKit
    ]
;

Platform-wide caching (experimental)

A consolidation of issues with the experimental platform-wide caching feature:

zero byte files seem to occasionally get placed into the cache
directory sources aren't properly cached since their source isn't currently factored into their key (related: #110)
keying based on factors such as environment variables or paths severely hurts cachability

And potential features to consider:

remote cache implementation
caching of sources (downloads, repos, etc.)
caching of universal binaries

Parallel build argument difficulties

It's less than obvious how to make needy build in parallel when the parameter to -j|--concurrency is optional:

$ needy satisfy -j openssl
usage: needy satisfy [-h] [-t TARGET] [-u UNIVERSAL_BINARY] [-j [CONCURRENCY]]
                     [-f] [-d VERBOSITY_LEVEL]
                     [--minimum-osx-version MINIMUM_OSX_VERSION]
                     [--minimum-tvos-version MINIMUM_TVOS_VERSION]
                     [--minimum-ios-version MINIMUM_IOS_VERSION]
                     [--android-api-level ANDROID_API_LEVEL]
                     [--android-toolchain ANDROID_TOOLCHAIN]
                     [--android-runtime {libstdc++,gnustl_shared}]
                     [library [library ...]]
needy satisfy: error: argument -j/--concurrency: invalid int value: 'openssl'

Can arg-parse easily disambiguate?

The current workaround is something like...

if [[ $(uname) == 'Darwin' ]] ; then
    NUM_CORES = $(sysctl -n hw.ncpu)
else
    NUM_CORES = $(nproc)
fi

./needy/scripts/needy generate jamfile --satisfy-args="-j $NUM_CORES"

Unsupported/unrecognized library keys don't produce errors

The following project won't produce any errors (assuming foo exists and has something in it).

libraries:
    foo:
        directory: foo
        asdfasdf:
            - echo hello

Add patch support

It would be desirable to be able to apply a list of patches to a source directory prior to project type determination and prior to a build. Currently, the best way to do this varies between download types (git apply {{ needs_file|dirname }}/some-patch.patch, patch -p1 < some-patch.patch, among others) and a unified syntax for applying this without the boilerplate would be greatly desired.

Patch file contents should be factored into the configuration hashes.

Make configure script detection more robust

Currently, autotools configure scripts are the only ones supported. A new configure, make, install builder should be included when a configure script is present but isn't an Autotools script. Ideally, we should be able to detect the presence of Autotools configure scripts without running ./configure --version as some configure scripts don't respect those flags and will start the configure process.

As an example, FFmpeg could be more easily configured:

ffmpeg:
    repository: git://source.ffmpeg.org/ffmpeg.git
    commit: n3.0.2
    dependencies:
        - x264
    project:
        environment:
            PKG_CONFIG_PATH: '{{ build_directory('x264') }}/lib/pkgconfig:{current}'
        build-steps:
            - ./configure --prefix={build_directory} --enable-gpl --enable-libx264
            - make -j8 install

Ideally this would be the library configuration:

ffmpeg:
    repository: git://source.ffmpeg.org/ffmpeg.git
    commit: n3.0.2
    dependencies:
        - x264
    project:
        environment:
            PKG_CONFIG_PATH: '{{ build_directory('x264') }}/lib/pkgconfig:{current}'
        configure-args:
            - --enable-gpl
            - --enable-libx264

Add xcconfig generator

For integrating with Xcode projects.

Add better caching for directory sources

Currently, directory sources can't properly be cached due to build-affecting changes outside the needs configuration.

Add cross-platform dependencies

If a library has a dependency on a host-only need, that relationship cannot currently be set. Needy should add support for declaring cross-platform dependencies.

Add a Needy distribution egg/zip

Instead of a submodule, a single file should be committed to repositories that depend on Needy. A python egg or zip (I've read that a zip with a __main__.py works as expected) would be ideal.

Support for user-defined parameters

This would look something like...

needy satisfy --define fooparameter=bar

libraries:
  somelib:
    {% if definition('fooparameter', 'default') == 'bar' %}
      ... stuff ...
    {% endif %}

The goal would be to better facilitate multiple library build variations.

Add common environment variables to library hash

Caches need to be invalidated when CXXFLAGS, CFLAGS, and LDFLAGS are changed. It would be appropriate to specify these things in the cache configuration.

Add configuration hash to build compatibility for b2 targets

... to ensure that b2 targets are rebuilt when the needs file changes.

Warn/error when pkg-config isn't available

Add support for host-only build tools

In many cases specific build tools are required for compilation on multiple architectures. Examples include NASM, YASM, and GNU make.

Needy should provide a separate top-level directive for tools that are managed much like libraries except that they only target the host architecture.

tools:
    nasm:
        download: 'http://www.nasm.us/pub/nasm/releasebuilds/2.11.06/nasm-2.11.06.tar.bz2'
        checksum: 'eaad6f440d3f8d2172d7fd93bed224d4052b904f'

Scan all files only when library is in dev mode

local files = [ SPLIT_BY_CHARACTERS [ SHELL "find $(sourcedir) -type f -not -path '*/\.*' 2> /dev/null" ] : "\n" ] ;

This drastically increases the runtime of b2 operations and seems to only be necessary for dev-mode libraries. b2 should query Needy for a library's dev-mode status before scanning all of the files.

When a lib has a Needy dependency, building the lib via b2 doesn't build the dependency

As stated in the title, when a lib has a Needy dependency, building the lib via b2 doesn't appear to build the dependency. The workaround is to invoke b2 multiple times.

Add a way to specify external source for dev-mode

When a user would like to modify a needy-managed library, they may wish to develop the library from an external location. I would suggest utilizing dev-mode for this:

needy dev-mode mylib --source=../mylib

When this is done, the sources properties for that library should be ignored and instead the library should be treated as if it had a directory: '../mylib' library property.

Evaluate configure-steps and build-steps in single shell context

So you can do things like...

- export FOO=BAR
- ./buildusingfoo

Graceful handling of git errors

When git isn't installed:

[Errno 2] No such file or directory
Traceback (most recent call last):
  File "/vagrant/needy/scripts/needy", line 9, in <module>
    sys.exit(main(sys.argv))
  File "/vagrant/needy/needy/__main__.py", line 275, in main
    return commands[parameters.command](parameters.args)
  File "/vagrant/needy/needy/__main__.py", line 77, in satisfy
    needy.satisfy_target(needy.target(parameters.target), parameters.library)
  File "/vagrant/needy/needy/needy.py", line 352, in satisfy_target
    library.build()
  File "/vagrant/needy/needy/library.py", line 115, in build
    self.clean_source()
  File "/vagrant/needy/needy/library.py", line 101, in clean_source
    source.clean()
  File "/vagrant/needy/needy/sources/git.py", line 21, in clean
    command(['git', 'clean', '-xfd'], logging.DEBUG)
  File "/vagrant/needy/needy/process.py", line 37, in command
    __log_check_call(cmd, verbosity, env=env)
  File "/vagrant/needy/needy/process.py", line 28, in __log_check_call
    subprocess.check_call(cmd, stderr=devnull, shell=shell, stdout=devnull, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Improving error feedback

Creating a single issue for the various issues where feedback can be improved:

pkg-config is missing
git is missing
git can't establish the authenticity of the host
git is denied permission to the remote
unused or misplaced parameters

Support for "watching" directories or files

Directory sources should watch their source directory for changes, and be considered out-of-date afterwards.

Other sources might also use additional files or directories as inputs to their build, which should affect them the same way.

For example...

{
    'repository': ...,
    'commit:' ...,
    'project': {
        'post-clean': 'apply-patch {needs_file_directory}/mypatch.diff'
    }
}

For that be appropriately rebuilt, we would need to consider the contents of mypatch.diff. So maybe something like this should be possible:

{
    'repository': ...,
    'commit:' ...,
    'watch': '{needs_file_directory}/mypatch.diff',
    'project': {
        'post-clean': 'apply-patch {needs_file_directory}/mypatch.diff'
    }
}

builds should be considered out-dated after needy is updated

This might mean it's time to start versioning.

Add a means to get build configuration from a central repository

We have multiple projects and multiple libraries with similar (if not the same) build configurations:

libraries:
    openssl:
        repository: [email protected]:openssl/openssl.git
        commit: OpenSSL_1_0_1p
        project:
            build-steps:
                {% if platform == 'osx' %}
                - sh ./Configure darwin64-x86_64-cc --prefix={build_directory}
                - make install
                {% elif platform == 'ios' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory}
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/iPhoneOS.platform/Developer CROSS_SDK=iPhoneOS.sdk'
                {% elif platform == 'iossimulator' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -D__STRICT_ANSI__=1 no-asm
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/iPhoneSimulator.platform/Developer CROSS_SDK=iPhoneSimulator.sdk'
                {% elif platform == 'tvos' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -DHAVE_FORK=0
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/AppleTVOS.platform/Developer CROSS_SDK=AppleTVOS.sdk'
                {% elif platform == 'tvossimulator' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -D__STRICT_ANSI__=1 -DHAVE_FORK=0 no-asm
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/AppleTVSimulator.platform/Developer CROSS_SDK=AppleTVSimulator.sdk'
                {% elif platform == 'android' %}
                - sh ./Configure android --prefix={build_directory}
                - make install
                {% elif platform == host_platform %}
                - sh ./config --prefix={build_directory}
                - make install
                {% endif %}

It would be nice if we could back this configuration by a repository. It would be easy to store these externally using a git repository to back it:

Base project:

libraries:
    openssl:
        repository: [email protected]:my-org/needy-recipes.git
        commit: 1.0.0
        recipe: openssl.yaml

Then, in a separate repository:

openssl.yaml @ tag 1.0.0

libraries:
    openssl:
        repository: [email protected]:openssl/openssl.git
        commit: OpenSSL_1_0_1p
        project:
            build-steps:
                {% if platform == 'osx' %}
                - sh ./Configure darwin64-x86_64-cc --prefix={build_directory}
                - make install
                {% elif platform == 'ios' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory}
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/iPhoneOS.platform/Developer CROSS_SDK=iPhoneOS.sdk'
                {% elif platform == 'iossimulator' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -D__STRICT_ANSI__=1 no-asm
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/iPhoneSimulator.platform/Developer CROSS_SDK=iPhoneSimulator.sdk'
                {% elif platform == 'tvos' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -DHAVE_FORK=0
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/AppleTVOS.platform/Developer CROSS_SDK=AppleTVOS.sdk'
                {% elif platform == 'tvossimulator' %}
                - sh ./Configure iphoneos-cross --prefix={build_directory} -D__STRICT_ANSI__=1 -DHAVE_FORK=0 no-asm
                - sh -c 'make install CROSS_TOP=`xcode-select -p`/Platforms/AppleTVSimulator.platform/Developer CROSS_SDK=AppleTVSimulator.sdk'
                {% elif platform == 'android' %}
                - sh ./Configure android --prefix={build_directory}
                - make install
                {% elif platform == host_platform %}
                - sh ./config --prefix={build_directory}
                - make install
                {% endif %}

Move .needy_lock to needs directory

Just so there's one less thing to add to the .gitignore.

Add support for "distribution" projects

"Distribution" projects are projects that are already packaged and should essentially just be copied directly to build_directory. The purpose here would be to simplify the use of pre-build binaries and to eliminate the need to specify simple steps like build-steps: cp * {build_directory}.

To detect a distribution project, I think a file/folder scan for static or shared libraries in lib, headers in include. A detractor would be if any source files are located in source folders.

Conveniently, I think this covers a lot of header-only projects.

Needy should detect whether cmake is installed

Same would apply to other required build tools.

Allow multiple needy instances to run in parallel

Needy should support building independent needs by different concurrently-running instances of needy to allow for better integration with build systems that support parallel building of targets. A simple design might be to lock the build status file before target evaluation and release after a successful build.

generator documentation

Factor dependency configuration hashes into configuration hash

When dependent libraries are changes, it should trigger rebuilds of dependent libraries.

Add .needyconfig files for project and user level configurations

Needy cache configuration is a configuration that would be great to have as a user or global setting instead of a per-project setting.

Allow .needyconfig files to be used at the project level or at the user level.

Add jinja functions to run shell commands

It's often useful to run shell commands to query environment conditions prior to the rendering a needs file:

{% if platform == 'ios' %}
{% set xcrun_platform = 'iphoneos' %}
{% elif platform == 'iossimulator' %}
{% set xcrun_platform = 'iphonesimulator' %}
{% elif platform == 'tvos' %}
{% set xcrun_platform = 'appletvos' %}
{% elif platform == 'tvossimulator' %}
{% set xcrun_platform = 'appletvsimulator' %}
{% endif %}
libraries:
    openssl:
        repository: [email protected]:openssl/openssl.git
        commit: OpenSSL_1_0_1p
        project:
            build-steps:
                {% set openssl_flags = ' --prefix={build_directory}' %}
                {% if platform in ['ios', 'iossimulator', 'tvos', 'tvossimulator'] %}
                  - sh ./Configure iphoneos-cross {{openssl_flags}}
                {% set cross_args = 'CROSS_TOP=$(xcrun --sdk ' + xcrun_platform + ' --show-sdk-platform-path)/Developer CROSS_SDK=$(basename $(xcrun --sdk ' + xcrun_platform + ' --show-sdk-path))' %}
                {% elif platform == host_platform %}
                - sh ./config {{openssl_flags}}
                {% endif %}
                - make depend
                - sh -c 'make install {{cross_args}}'

However, the cacheability of this could be greatly improved if the output of those xcrun commands could be factored into the configuration hash. My proposal is to provide a jinja rule that executes an arbitrary shell command and includes the rendered output.

needy clean
needy clean libfmt
needy clean libfmt -t android

Fix rare race condition with b2 integration

$ ./b2 -j8 toolset=darwin
error: Unable to find file or target named
error:     'Waiting for other needy instances to terminate...'
error: referred to from project at
error:     '/Users/vmrob/development/myproject/needs'