academysoftwarefoundation / rez Goto Github PK
View Code? Open in Web Editor NEWAn integrated package configuration, build and deployment system for software
Home Page: https://rez.readthedocs.io
License: Apache License 2.0
An integrated package configuration, build and deployment system for software
Home Page: https://rez.readthedocs.io
License: Apache License 2.0
The inexact versioning mechanism means that the package descriptor 'foo-5' is supposed to map to foo-5[.x.x.x...]. However if an exact match ('foo-5') exists in the repository, rez will always pick that, and ignore other foo.5.x[.x.x...] packages.
There are a couple of different docstring formats being used in rez right now, we should decide on a convention and stick to it.
First, we need to decide what documentation generator we will use as that will dictate the available styles. I think that sphinx is the obvious choice, because it is the only actively maintained documentation generator for python out there. I've used epydoc, and it was easy to use, but it was already pretty outdated 5 years ago, so I would not bet on that horse now. Doxygen was designed for C++ and its python support is just a hack on top of that. IIRC, it could not do python introspection on live python objects like sphinx and epydoc, it could just parse python code.
So, if sphinx is the choice we have 3 styles to choose from: native, numpy, and google. here's a sphinx extension that adds supports for the latter 2: http://sphinx-doc.org/latest/ext/napoleon.html
Here they are for your simple viewing pleasure:
sphinx
def func(arg1, arg2):
"""Summary line.
Extended description of function.
:param arg1: Description of arg1
:type arg1: int
:param arg2: Description of arg2
:type arg2: str
:returns: Description of return value
:rtype: bool
"""
return True
numpy
def func(arg1, arg2):
"""Summary line.
Extended description of function.
Parameters
----------
arg1 : int
Description of arg1
arg2 : str
Description of arg2
Returns
-------
bool
Description of return value
"""
return True
def func(arg1, arg2):
"""Summary line.
Extended description of function.
Args:
arg1 (int): Description of arg1
arg2 (str): Description of arg2
Returns:
bool: Description of return value
"""
return True
A note on sphinx:
If you have not used sphinx before, but you have used other documentation generators like epydoc, it might be confusing at first. It is important to know that sphinx is designed in layers: the primary layer is a pure restructuredText parser, which translates an rst file into rich output like html, usually one html file per rst file. Hence, most of the sphinx documentation covers writing restructuredText documents from scratch and has nothing to say about generating API documentation from source code. That is handled by a set of extensions (namely, the autodoc extension) which generate rst documents from python modules and their docstrings, which are then converted into rich output as a second pass.
From Allan: For 2.0 I would also add: setup.py. I think this is achievable, and the installation code right now is ugly and non-standard.
It seems like the order you specify your packages to a rez-env or rez-config call affects the resolution. I experienced this where one of my requested packages was unversioned and the other package had a specific version.
Suppose my package "foo" has two versions -- 1.0 and 2.0. Now suppose I have another package "bar-1.0" that requires "foo-1.0". This call will succeed: "rez-env bar-1.0 foo" where I will have an environment with bar-1.0 and foo-1.0. However, if I switch the order and run "rez-env foo bar-1.0" I will get a conflict saying that foo-1.0 conflicts with foo-2.0.
The problem is that in the latter case where the unversioned package is specified first, rez will first resolve it to the latest version ("foo-2.0" in this case) and then try to resolve the other requests around that. But that seems to go against the general idea of rez where an unversioned request is supposed to be "any compatible version of foo."
This is problematic for end users if they need to specify packages in a particular order to get the right rez environment.
md yay, pdf boo.
The default build process isn't releasing from a fresh export of the repo, this should be changed. It will also have to create package-release.yaml and variant-release.yaml files, and this will take into account per-variant timestamps as well.
me:
detecting cpp compiler...
found cpp compiler: /usr/bin/gcc, id: 4.4.4
Couldn't detect compiler version, assuming 1.0.1...
cpp compiler version: 1.0.1
/usr/bin/gcc --version
gcc (GCC) 4.4.4 20100726 (Red Hat 4.4.4-13)
Sent at 8:34 PM on Tuesday
me: sorry, didn't mean to just dump this on you, but just so you're aware when you get to it yourself ;) I don't have time to fix the script, just hacked the version for now
Allan: cool thx. When I get a chance to look at this (next week) I might have to get you to try some code, as I probably won't be able to replicate the prob on a different system
Allan: oh can you try:
/usr/bin/gcc -dumpversion
maybe that doesn't work for gcc-4.4.4+... easy enough to fix, I'll make it try --version if that fails..
me: 11:04 jensj@teargas: ~/Documents 38 >/usr/bin/gcc -dumpversion
4.4.4
Allan: ah I see the prob... for some reason the 'compiler id' is 4.4.4, usually it's 'GNU'. Dunno why this behaviour
anyway should be a pretty simple fix
I would love to see the rez codebase made more PEP8 compliant: http://www.python.org/dev/peps/pep-0008/
current problems:
type(foo) == type(bar)
instead of isinstance
_g_variable
for global constants (_
prefix denotes private variables)foo=3
instead of foo = 3
and foo(1+2)
instead of foo(1 +2)
from module import *
import rez_config
instead of import rez.rez_config
.non-pep8 style issues:
rez_
prefix in front of some of the modules is redundant and non-standard. I get the impression that rez used to be a bunch of separate modules that were not under a package, but now that it is a package, there's no longer a need for the prefix.now is the time to fix these things since the code is in such flux that other changes are highly unlikely to merge anyway.
If you give me the go ahead, I can automatically fix all whitespace issues with autopep8
. I use these settings:
autopep8 -i -r -v --pep8-passes=300 --ignore=E501,E302
I'm getting errors when I'm running configure.sh, I've fixed it so I get some relevant info on a Mac. This is what with the fix:
detecting OS distribution...
OS distribution is: Apple
detecting OS distribution version...
OS distribution version is: 10.8.3
Patch:
diff --git a/configure.sh b/configure.sh
index 29d2148..c6f2e6c 100755
--- a/configure.sh
+++ b/configure.sh
@@ -246,6 +246,9 @@ if [ "$distro" == "" ]; then
distro=`lsb_release -i | awk '{print $NF}'`
if [ $? -ne 0 ]; then distro=; fi
fi
+ if [ "$osname" == "Darwin" ]; then
+ distro=Apple
+ fi
if [ "$distro" == "" ]; then
echo $echoerr"Could not identify OS distribution - $or_set""REZCONFIG_DISTRO" 1>&2
echo "Setting to DISTRO_UNKNOWN-0!"$echoreset 1>&2
@@ -263,6 +266,10 @@ if [ "$distro_version" == "" ]; then
distro_version=`lsb_release -r | awk '{print $NF}'`
if [ $? -ne 0 ]; then distro_version=; fi
fi
+ if [ "$osname" == "Darwin" ]; then
+ distro_version=`sw_vers -productVersion`
+ if [ $? -ne 0 ]; then distro_version=; fi
+ fi
if [ "$distro_version" == "" ]; then
echo $echoerr"Could not identify OS distribution version - $or_set""REZCONFIG_DISTRO_VERSION" 1>&2
echo "Setting to 0!"$echoreset 1>&2
eg patching an env containing 'foo-1.5|3' fails - looks like a simple string parsing issue.
If a package.yaml was modified after it was cached 'rez-env' will only return the (outdated) cached entry. 'rez-env --no-cache' returns the expected updates.
Workaround: restart memcached server
Hi,
I'd be interested in providing support for Windows.
Having briefly looked through your manual and source-dived part of your 2.0 release, I've gathered that much of Rez is built upon bash, which I can imagine being a challenge if ported to cmd.exe.
Is there anything else you think I should be aware of? Which part of your codebase would you recommend I get started with?
In an attempt to make this issue referable to newcomers and have them get up to speed more quickly, I'll collect some relevant links here.
Best,
Marcus
Another small one: ./examples/demo/run_demo display an error after an invocation of date. The date help is printed when this happens.
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Here's a patch that fixes it, with lengthy explanation of what goes wrong.
diff --git a/bin/rez-context-info b/bin/rez-context-info
index ccf2f59..53e13c5 100755
--- a/bin/rez-context-info
+++ b/bin/rez-context-info
@@ -17,7 +17,19 @@ if [ ! "$REZ_CONTEXT_FILE" ]; then
exit 1
fi
-readable_time=`date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"`
+echo REZ_REQUEST_TIME=$REZ_REQUEST_TIME
+
+# Pre python port
+# readable_time=`date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"`
+# $ date -d "1970-01-01 $REZ_REQUEST_TIME sec GMT"
+# Thu Aug 15 03:30:29 UTC 2013
+# $ python
+# >>> import datetime
+# >>> print datetime.datetime.fromtimestamp(1376537429).strftime('%a %b %d %H:%M:%S UTC %Y')
+# Thu Aug 15 03:30:29 UTC 2013
+#
+readable_time=`python -c 'import sys, datetime ; print datetime.datetime.fromtimestamp(float(sys.argv[1])).strftime("%a %b %d %H:%M:%S UTC %Y")' $REZ_REQUEST_TIME`
+
echo "requested packages (mode="$REZ_RESOLVE_MODE", time="$REZ_REQUEST_TIME: $readable_time"):"
echo $REZ_REQUEST | tr ' ' '\n'
echo
also ther may be toolchain-specific stuff in rez-run still? Remove if so
This is not a rez issue, but since there's not a mailing list that I know of yet, so go vote for this ticket: https://bitbucket.org/xi/pyyaml/issue/4/facilitate-modification-of-serialized-yaml
It would make life a lot easier if it was possible to run the tests without having to reinstall rez prior to each run. I tried to jump start this by first prefixing all of the test modules with test_
, since this is what unittest
uses for discovery by default.
then, from the src
directory, I ran python -m unittest2 discover -s rez/tests
i got a number of errors, but the ones that troubled me the most were these two:
======================================================================
ERROR: test_command (test_shells.TestShell)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_shells.py", line 52, in test_command
r = self._create_context(["hello_world"])
File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_shells.py", line 32, in _create_context
caching=False)
File "rez/resolved_context.py", line 131, in __init__
resolver.solve()
File "rez/resolver.py", line 48, in solve
verbose=self.verbose)
File "rez/solver.py", line 1277, in __init__
phase = _ResolvePhase(self.request_list.requirements, solver=self)
File "rez/solver.py", line 729, in __init__
scope = _PackageScope(package_request, solver=solver)
File "rez/solver.py", line 537, in __init__
package_request.range)
File "rez/solver.py", line 1537, in _get_variant_slice
slice = self.package_cache.get_variant_slice(package_name, range)
File "rez/solver.py", line 510, in get_variant_slice
building=self.building)
File "rez/solver.py", line 265, in __init__
"package family not found: %s" % package_name)
PackageFamilyNotFoundError: package family not found: hello_world
I believe that is using the hello_world
package creating during installation. all the tests should be completely self contained and should only use packages created inside the data
directory. several of the tests do this already.
This is the other issue that stuck out:
Traceback (most recent call last):
File "/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/test_build.py", line 28, in setUpClass
shutil.copytree(packages_path, cls.src_root)
File "/usr/local/python-2.7.4/lib/python2.7/shutil.py", line 171, in copytree
names = os.listdir(src)
OSError: [Errno 2] No such file or directory: '/Volumes/sv-dev01/devRepo/chad/shell/rez/src/rez/tests/data/build/packages'
It seems like there are packages missing for the build
test? perhaps these weren't committed?
So with the init.csh and the mktemp fix I'm a little further, but now that I'm doing real stuff it fails. I can now see the GNU sed errors, I'll try to see how to debug them, and see if something easy can be done to detect the distribution
panzani rez$ uname -a
Darwin panzani.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
panzani rez$ uname -a | awk '{ print $3 }'
12.3.0
Errors below.
[venv] panzani rez$ ./examples/demo/run_demo
INSTALLING REZ LOCALLY...
Detecting operating system...
Operating system is: Darwin
detecting shell...
Shell is: tcsh
detecting OS distribution...
Could not identify OS distribution - either specify manually in configure.sh, or set $REZCONFIG_DISTRO
Setting to DISTRO_UNKNOWN-0!
detecting cmake...
found cmake binary: /usr/bin/cmake
detecting cpp compiler...
found cpp compiler: /usr/bin/c++, id: GNU
cpp compiler version: 4.2.1
detecting python...
found python binary: /Users/bsergean/src/rez/venv/bin/python
python version: 2.7.2
detecting pyyaml...
found pyyaml at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages
detecting pydot...
found pydot at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages
detecting pyparsing...
found pyparsing at /Users/bsergean/src/rez/venv/lib/python2.7/site-packages
There were 1 issues, please review above. Note that installation has still succeeded.
rez.configured written.
Now run ./install.sh
rez 1.6.9 installed successfully.
READYING TEST PACKAGES...
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
sed: -e: No such file or directory
BUILDING TEST PACKAGES...
BUILDING: hello_world_py-1.0.0
rez-build: invoking rez-config with args:
--time=1376457195
requested packages: python-!PYVER!
package search paths: /Users/bsergean/src/rez/examples/demo/build/published_packages
Invalid version: Invalid version '!PYVER!'
rez-build failed - an environment failed to resolve.
We are starting to build packages with both ICC and GCC variants. As ICC requires GCC at run time (and I think build time), my icc
package looks something like this:
name: icc
version: 14.0.0
variants:
- [ CentOS-6.2, gcc-4.1.2 ]
- [ CentOS-6.2, gcc-4.4.6 ]
commands:
- CXX=!ROOT!/bin/icpc
- CC=!ROOT!/bin/icc
To build all ICC and GCC variants of a higher level package, we must do something like this:
name: foo
version: 1.0.0
variants:
- [ CentOS-6.2, gcc-4.1.2 ]
- [ CentOS-6.2, gcc-4.4.6 ]
- [ CentOS-6.2, icc-14.0.0, gcc-4.1.2 ]
- [ CentOS-6.2, icc-14.0.0, gcc-4.4.6 ]
This creates three problems:
icc
package redefines the CXX and CC environment variables, which are also defined in the gcc
package. Rez does not allow this (it's a conflict) and so temporarily I've commented this restriction out. I understand the restriction, but maybe we can loosen it a little?foo
package now has unbalanced variants. This doesn't appear to be a problem (it works) but is maybe not considered very neat. This would perhaps benefit from naming the variant folders as we discussed in another thread.bah
with the same variants as foo
we can build an environment using the gcc-4.4.6
variant of foo
and the icc-14.0.0, gcc-4.4.6
variant of bah
.Currently (I believe) a call rez-env foo bah
will pick the first variant that creates a resolution (both GCC), rez-env foo bah icc-14.0.0 gcc-4.4.6
will resolve for both packages to use the ICC variant. Is there currently some syntax I can use to be more specific, e.g. rez-env foo:gcc-4.4.6 bah:icc-14.0.0
?
Perhaps this is an extension of what was discussed in issue 21.
To extend point 1) a little further with another use case:
We want to define a PYTHON_EXE
environment variable which points to the current python interpreter for the resolved environment. For example:
name: maya
requires:
-python-2.6
commands:
- export PYTHON_EXE=mayapy
name: python
version: 2.6.6
commands:
- export PYTHON_EXE=python2.6
As both the python
and maya
package define PYTHON_EXE
we get a conflict. The way around this I see would be to define the python-2.6
requirement of maya
as a separate mayapy
package.
name: maya
requires:
-mayapy-2.6
name: mayapy
version: 2.6.6
commands:
- export PYTHON_EXE=mayapy
But our python applications have three variants for python 2.5, 2.6 and 2.7 and for applications which are pure python, they work in a normal python interpreter, as well as Maya/Nuke/Houdini's. To follow this approach we'd have to build many more variants of essentially the same code which seems wasteful:
name: foo
variants:
- [ python-2.5 ]
- [ python-2.6 ]
- [ python-2.7 ]
- [ mayapy-2.6 ]
- [ hython-2.6 ]
- [ nukepy-2.6 ]
I don't like this approach. Any suggestions?
I can't figure out that one ... I tried on a Linux box and it works perfectly; I installed a non-system cmake in /usr/local/bin, I've installed py-parsing + pydot to the system python. Still no luck, not sure what to try now, and I'm not familiar with cmake unfortunately.
INSTALLING REZ LOCALLY...
Detecting operating system...
Operating system is: Darwin
detecting shell...
Shell is: tcsh
detecting OS distribution...
OS distribution is: Apple
detecting OS distribution version...
OS distribution version is: 10.8.3
detecting cmake...
found cmake binary: /usr/local/bin/cmake
detecting cpp compiler...
found cpp compiler: /usr/bin/c++, id: Clang
Couldn't detect compiler version, assuming 1.0.1...
cpp compiler version: 1.0.1
detecting python...
found python binary: /usr/bin/python
python version: 2.7.2
detecting pyyaml...
found pyyaml at /Library/Python/2.7/site-packages
detecting pydot...
found pydot at /Library/Python/2.7/site-packages
detecting pyparsing...
found pyparsing at /Library/Python/2.7/site-packages
There were 1 issues, please review above. Note that installation has still succeeded.
rez.configured written.
Now run ./install.sh
rez 1.6.9 installed successfully.
READYING TEST PACKAGES...
BUILDING TEST PACKAGES...
BUILDING: hello_world_py-1.0.0
rez-build: invoking rez-config with args:
--time=1376545154
requested packages: python-2.7.2
package search paths: /Users/bsergean/src/rez/examples/demo/build/published_packages
rez-build: in new env:
running rez-config v1.6.9
REZ_REQUEST_TIME=1376545154
requested packages (mode=latest, time=1376545154: Wed Aug 14 22:39:14 UTC 2013):
Darwin
python-2.7.2
cmake-2.8.9
resolved packages:
Darwin /Users/bsergean/src/rez/examples/demo/build/published_packages/Darwin (local)
cmake-2.8.9 /Users/bsergean/src/rez/examples/demo/build/published_packages/cmake/2.8.9/Darwin (local)
python-2.7.2 /Users/bsergean/src/rez/examples/demo/build/published_packages/python/2.7.2/Darwin (local)
number of failed attempts: 0
context file:
/Users/bsergean/src/rez/examples/demo/build/projects/hello_world_py/1.0.0/build/build-env.context
rez-build: calling rez-cmake -d ../
rez-cmake: calling cmake with the following arguments: -DCMAKE_MODULE_PATH=/Users/bsergean/src/rez/examples/demo/build/rez_local_install/1.6.9/cmake;/Users/bsergean/src/rez/examples/demo/build/published_packages/Darwin/cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_SKIP_RPATH=1 ../
-- The C compiler identification is GNU 4.2.1
-- The CXX compiler identification is Clang 4.0.0
-- Could not determine Eclipse version, assuming at least 3.6 (Helios). Adjust CMAKE_ECLIPSE_VERSION if this is wrong.
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.27.1")
CMake Error at /Users/bsergean/src/rez/examples/demo/build/rez_local_install/1.6.9/cmake/RezInstallPython.cmake:31 (message):
a version of python must be listed as a requirement when using the
'rez_install_python' macro. Packages for this build are: python-2.7.2;
Call Stack (most recent call first):
CMakeLists.txt:7 (rez_install_python)
Last week I started experimenting with using rez on linux and osx, and quickly discovered that OS packages are not mutually exclusive (I think I may be the first person actually using rez on two operating systems at the same site). There is nothing stopping a resolve on osx from pulling in packages that require Linux. Making Linux and Darwin anti-packages of each other only protects against bad resolves by aborting the resolution.
My plan is to create an "os" package, with versions per os: os-darwin
, os-linux
, os-windows
. Versions are inherently mutually exclusive, so it will prevent os-darwin
and os-linux
from being loaded simultaneously, and thus allow packages that require or vary on these to resolve properly.
To achieve this, we will first need to implement a new type of Version
subclass that supports a string label.
name: os
version: linux
name: os
version: darwin
name: os
version: windows
similarly, architectures can be implemented with arch-i386
and arch-x86_64
packages.
name: arch
version: x86_64
name: arch
version: i386
lastly, specific versions of an operating system can be implemented as their own packages. e.g. osx-10.8.1
, fedora-19
, etc. Here are some complete examples from windows, with numeric versions taken from wikipedia.
name : windows
version: 7 # technically, this is version 6.1
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]
name : windows
version: 6.0
version_alias: Vista
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]
name : windows
version: 5.1
version_alias: XP
requires:
- os-windows
variants:
- [arch-x86_64]
- [arch-i386]
name : windows
version: 5.0
version_alias: 2000
requires:
- os-windows
- arch-i386
.metadata/info.txt
needs a major face lift and should probably happen now while we're introducing so many changes
info.txt
stores release info, but why is the release_time in a different file than the release info? Is release_time meaningfully different than ACTUAL_BUILD_TIME and BUILD_TIME from info.txt
?
It seems like this file predates use of yaml file format, so maybe now that we have yaml we can consider condensing this all into .metadata/release.yaml
or even just .release.yaml
.
Also, we need to address version-control-specific data, and no longer assume SVN.
It would be nice if rez's package files supported the concept of "optional requirements" (i'm sure we can come up with a less oxymoronic name). These packages would be loaded if found successfully, but otherwise would not result in an error.
my current scenario is that I'm using rez to build MtoA (the Maya to Arnold translator), which has an optional plugin for shave and a haircut. We have shave for maya 2012 installed, but we don't yet have it for maya 2014. it would be nice if in the mtoa package.yaml file I could effectively say "load shave if you can", then in the build script (in this case it's scons) handle the logic if it is or is not there.
This feature is a nice analog to cmake's find_package, without the REQUIRED flag.
..to list info on wrappers currently in the env
Currently I am maintaining a private branch of Rez with some studio specific changes.
An example of these changes are a different tag name (we use name-version instead of just version) and popping an event onto ActiveMQ in rez release
.
Currently this works ok, but I can envisage problems when it comes to merging Chad's pure python changes. And on an ongoing basis we'll have to do some merging/patching as Rez continues to evolve.
Would there be a benefit to considering plugins for the Rez architecture? The repository strategy is already a plugin of sorts with Git/Svn/Mercurial being chosen at runtime, but perhaps this idea could be extended? Some of the issues (such as tag naming) would be resolved with improved configuration, however sending events is very studio specific and most likely only configurable
in real python.
This would be easier to implement with Chad's pure python changes too. What do you think?
dynamic shells, caching, rez-egg-install etc
I've done a quick conversion of the README file and here's how it looks like.
https://github.com/bsergean/rez
Tell me what you think !
the merge_osx branch tries to solve some problems with regard to which browser is used to open the help url, but i think ultimately the changes are lipstick on a pig, as they say in the south (or so i'm lead to believe by television).
i think it is generally a bad idea to attempt to specify the application that should open the file / url within the package.yaml. using an environment variable for $BROWSER
is at least somewhat portable, but, even assuming everyone has this variable set, it still only solves the problem for webpages.
i think a better solution is to use the OS-specific command for opening a file: e.g. open
on osx, and xdg-open
on linux (gnome, kde, xfce). this behavior would occur if the help is in the form of an absolute path or url only (i.e. no proceeding application name).
I've just finished converting rez-cmake
to python, and the command strikes me as somewhat unnecessary. currently, rez-build
creates a bash script build-env.sh
which calls rez-cmake
which in turn calls cmake
.
here's what i don't like about rez-cmake
it doesn't do much
rez-cmake
essentially just makes it a little easier to pass arguments to cmake
, and thus doesn't do anything that rez-build
couldn't easily bake into build-env.sh
.
here's what it always does:
CMAKE_SKIP_RPATH
-DCMAKE_MODULE_PATH=CMAKE_MODULE_PATH
CMAKE_INITIAL_CACHE
if already sethere's what it optionally does:
CMakeLists.txt
)-DCENTRAL=1
)-DCOVERAGE=1
)it adds another layer of confusion
rez-build
simply passes all args after the first --
along to rez-cmake
untouched. and i assume it is by design that one could cram rez-cmake
, cmake
, and make
args this because rez-cmake
passes along all args after ---
to cmake
.
e.g.
rez-build -v 0 -- -b eclipse --- -DFOO=1 --
having 3 sets of arguments for rez-build
is tolerable. but when you throw in a 4th set of arguments for a cmake
wrapper, which might actually conflict with arguments passed directly to cmake
, then i think it's crossed a line of convolutedness.
how would you feel about moving the rez-cmake
arguments and behavior to rez-build
and call cmake
directly in build-env.sh
? since rez-cmake
has now been converted to python, it would be easy to create a function shared by both rez-build
and rez-cmake
for performing the tasks therein, and we could keep rez-cmake
around as a cli utility, but not actually have rez-build
use it.
rez already has plenty of utilities for reading and validating data from multiple sources in rez.resources
(see Issue #52 for a discussion on schemas). Instead of placing the schema definition within rezconfig
we should use our existing metadata validators to do this work for us. I don't love the idea of having configuration files with huge "DO NOT EDIT" blocks.
I've got a few improvements that I want to make to the variable expansion convention that I proposed earlier.
version.part(1) == 2
version.part(2) == 7
version.part(3) == 4
version.thru(1) == '2'
version.thru(2) == '2.7'
version.thru(3) == '2.7.4'
commands: |
PYTHON_MAJOR_VERSION = '{version.part(1)}'
PYTHON_MINOR_VERSION = '{version.part(2)}'
CMAKE_PREFIX_PATH.prepend('$PYTHON_DIR')
if machine.platform == 'Linux':
PYTHON_DIR = '/usr/local/python-{version}'
PATH.prepend('$PYTHON_DIR/bin')
elif machine.platform == 'Darwin':
PYTHON_DIR = '/usr/local/python-{version}'
PATH.prepend('$PYTHON_DIR/Python.framework/Versions/{version.thru(2)}/bin')
else:
PYTHON_DIR = 'C:/Python{version.part(1)}{version.part(2)}'
PATH.prepend('$PYTHON_DIR')
PATH.prepend('$PYTHON_DIR/Scripts')
I want to switch over to using "new style" string formatting, which allows custom types to define their own formatting via a __format__
method. It also provides a way to pass custom "format specifiers" to those types (the format specifier is everything after the :
in the following examples).
For example:
>>> '{:04d}'.format(1)
'0001'
>>> import datetime
>>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)
>>> '{:%Y-%m-%d %H:%M:%S}'.format(d)
'2010-07-04 12:15:58'
Python's new-style formatters support attribute access {foo.myattr}
and item access {foo[0]}
as well as chaining the two {foo.myattr[0]}
. However, unlike rez's current formatters, it does not support function execution {foo.myfunc()}
. Thus, I'd like to remove the thru()
and part()
functions, and provide simply a parts
attribute (0-indexed):
version.parts[0] == '2'
version.parts[1] == '7'
version.parts[2] == '4'
explicit formatting can be done like this:
"{version.parts[0]}.{version.parts[1]}.{version.parts[2]}" == '2.7.4'
but custom format specifiers for our ExactVersion
class will provide a more abbreviated option:
"{version:#.#.#}" == '2.7.4'
"{version:##}" == '27'
"{version:#.#v#}" == '2.7v4'
"{version:#0.#1.#2}" == '2.7.4'
"{version:#1.#2}" == '7.4'
"{version:#000.#01.#2}" == '002.07.4'
and the yaml example would look like this:
commands: |
PYTHON_MAJOR_VERSION = '{version.parts[0]}'
PYTHON_MINOR_VERSION = '{version.parts[1]}'
CMAKE_PREFIX_PATH.prepend('$PYTHON_DIR')
if machine.platform == 'Linux':
PYTHON_DIR = '/usr/local/python-{version}'
PATH.prepend('$PYTHON_DIR/bin')
elif machine.platform == 'Darwin':
PYTHON_DIR = '/usr/local/python-{version}'
PATH.prepend('$PYTHON_DIR/Python.framework/Versions/{version:#.#}/bin')
else:
PYTHON_DIR = 'C:/Python{version:##}'
PATH.prepend('$PYTHON_DIR')
PATH.prepend('$PYTHON_DIR/Scripts')
thoughts?
How should internal package variants be modelled in Rez?
The current variant system caters well for the cases where a variant is an external dependency (such as the gcc version, or the target operating system). But what about cases where the variant is internal to the package?
For example, the packages foo and boo both depend on the same version of bah. However they depend on different variants of bah; foo wants it compiled with option A, boo with option B. The options are mutually exclusive, and are build time options for bah.
I could make foo and bar variants of bah, allowing me to create the different builds, but this reverses the dependency which is incorrect.
I could make a meta-package for the two options so they can be listed as variants of bah, but this will become messy as the number of options increases.
Or I could make two packages; bah_A and bah_B. But this again will become messy as the number of options increases.
Is there a better way of modelling this in Rez?
From Chad:
should be achievable by both yaml or python, though python will obviously be more powerful.
the install() function should make use of rex, and should use the same idioms that we are designing for the commands section. e.g. testing if a package is set:
if pkgs.mypackage:
blah = pkgs.mypackage.root
new install-specific commands should also be provided through rex, such as patch()
we need to write pure python routines for finding libraries, include directories, etc. rez_install_cmake will no longer be necessary for this. finding libraries should support static vs dynamic vs no preference.
pkgs.mypackage.libraries.foo
pkgs.mypackage.static_libraries.foo
pkgs.mypackage.include_directories
the end goal is to remove cmake as a middle-man for building "external" package (e.g. packages which have their own build system which we did not write ourselves).
Trying to build a package with a version number of '0' breaks, .e.g, in package.yaml
version: 0
This will cause rez-build to fail.
Currently rez release
releases all variants in a single pass, which is generally a good thing.
However, we are finding that in some cases we have to go back and release an additional variant to an existing package (for example, an additional boost variant). We are unable to do this with the current implementation of rez-release.
One solution would be to increment the sub-patch version number (so 1.0.0
would become 1.0.0.1
) however this means releasing all variants again. For some tools this takes many hours. However for external packages (which we are currently releasing using the ExternalProjects
macro in CMake we want the version number to match the external number.
Another solution would be to add a -v
argument to rez release
which behaves in a similar way to rez build
. This creates problems with tagging and if misused could result in a package being partially released. That said this might become more necessary if we consider distributing variant builds across a CI system such as Jenkins.
Has anyone else encountered this problem? Does anyone have any suggestions?
Howdy,
Here's a small diff to make it so the init.csh that gets produced by install.sh is "sour cable". There was 2 straights if that should have been endif, one extra double quote in there; and the PATH settings is closer to the one for bash now. I'm still pretty rusty with git so here's a straight git diff on that file; I'll try to make a proper pull request.
diff --git a/init.csh b/init.csh
index 5a36c5d..fff5231 100644
--- a/init.csh
+++ b/init.csh
@@ -30,20 +30,18 @@ else
# where rez will publish local packages to (ie those installed with rez-build -- -- install)
if (! $?REZ_LOCAL_PACKAGES_PATH ) then
setenv REZ_LOCAL_PACKAGES_PATH !REZ_LOCAL_PKGS_PATH!
- fi
+ endif
# where rez-egg-install will install python egg packages to
- if (! $?REZ_EGG_PACKAGES_PATH" ) then
+ if (! $?REZ_EGG_PACKAGES_PATH ) then
setenv REZ_EGG_PACKAGES_PATH !REZ_PACKAGES_PATH!
- fi
+ endif
# expose rez binaries, replacing existing rez paths if they have been set already
set clean_path = `echo "$PATH" | /usr/bin/tr ':' '\n' | grep -v '^$' | grep -v '!REZ_BASE_PATH!' | /usr/bin/tr '\n' ':'`
- set rez_path = ":${REZ_PATH}/bin"
- set full_path = "$clean_path$rez_path"
- setenv PATH $full_path
+ setenv PATH ${clean_path}:${REZ_PATH}/bin
if (! $?REZ_RELEASE_EDITOR ) then
setenv REZ_RELEASE_EDITOR !REZ_RELEASE_EDITOR!
Now when I try to run the rez run hello_world, I get an error about the -p option of mkdir, which is probably only available with GNU mkdir and not the BSD one that I'm running on my Mac.
Cheers,
It would be nice if rez allowed 0-padding of version numbers. We name our maya installations after the Maya API version (e.g. 201217 becomes 2012.17), so for initial releases, that means our version is 2014.00 (201400 becomes 2014.00). This is somewhat of a special case, but in general, it would be nice if rez understood a few more common versioning conventions so that we could stick more closely with the actual version used by the vendor. Another example is the 7.0v2 convention used by The Foundry.
I want to be able to do the following:
rez-env '(foo ~~bar)'
where the package 'bar' will be added to the wrapper shell but some special syntax (~~ in this case) will prevent its tools to be exposed to the main / master shell.
Hum, not sure what to do about that one ...
bash-3.2$ file /usr/bin/rez
/usr/bin/rez: Mach-O executable i386
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/Rez.1.html
The -i is not supported on the BSD/MAC sed. So I'm just creating a temp file instead.
diff --git a/examples/demo/run_demo b/examples/demo/run_demo
index 9b5861a..704ede5 100755
--- a/examples/demo/run_demo
+++ b/examples/demo/run_demo
@@ -92,10 +92,11 @@ cp -rf ./projects ./build/projects
find ./build -type d -name '.svn' | xargs rm -rf
for yaml in `find ./build/projects -name 'package.yaml'`
do
- sed -i -e "s/!OS!/$osname/g" \
+ tmpf=`python -c 'import tempfile, sys ; print tempfile.mktemp()'`
+ sed -e "s/!OS!/$osname/g" \
-e "s/!CXX!/$_REZ_CPP_COMPILER_NAME-$_REZ_CPP_COMPILER_VER/g" \
- -e "s/!PYVER!/$_REZ_PYTHON_VER/g" \
- $yaml
+ -e "s/!PYVER!/$_REZ_PYTHON_VER/g" < $yaml > $tmpf
+ mv $tmpf $yaml
done
I'd like to look into using schema.py to define and validate our metadata schemas in rez.resources
. Its Schema
class is pretty similar to my MetadataSchema
class (formerly MetadataValidator
), but is more fully-featured. I think the additional features could come in handy in the future, and could also help consolidate some concepts within rez.resources
: namely the idea of 'path patterns'.
Currently we use schemas to describe the contents of files, and we use the
concept of a 'path pattern' to define where we will find that file in our directory tree. I would like to investigate treating the directory structure itself as a schema, with file metadata schemas nested within it. After reading over the docs, I think that schema.py will be flexible enough to do this. I think the hardest part will be modifying the ResourceIterator
to work based on Schema
instances.
Two other schema validation libraries I looked at are jsonschema and pykwalify. Despite its name, jsonschema is generic enough to validate yaml data, but might be a stretch to validate data coming from python files (callables, for example). The same goes for pykwalify, but to use it we'd also have to port it from python 3.x to 2.x.
schema.py is designed to validate python structures, so that applies equally to data from from json, yaml, or a python file. I think that makes it a good choice.
One feature that I'd really like to see in rez is the ability to manage external applications using a single multi-version package file.
A quick clarification of terms:
I've added this multi-version package feature in my own fork and used it for awhile, and it makes managing applications like maya and nuke much easier. Previously I had a directory structure like this:
maya/
├── 2012.17
│ │── package.yaml
│ │── platform-darwin
│ └── platform-linux
├── 2013.0
│ │── package.yaml
│ │── platform-darwin
│ └── platform-linux
├── 2014.0
│ │── package.yaml
│ └── platform-linux
├── 2014.2
│ │── package.yaml
│ └── platform-linux
├── 2014.4
│ │── package.yaml
│ └── platform-linux
├── 2014.50
│ │── package.yaml
│ └── platform-linux
├── 2014.51
│ │── package.yaml
│ └── platform-linux
└── 2014.59
│── package.yaml
└── platform-linux
(Note that all of those variant sub-directories are empty, they exist because rez errors if they don't. Also, I stopped using the ext
symlink + '!ROOT!'
because I found it hurt more than helped)
Now, with my fork, I have a single file called maya.yaml
that looks like this:
name : maya
config_version : 0
versions :
- "2012.17"
- "2014.0"
- "2014.2"
- "2014.4"
- "2014.50"
- "2014.51"
- "2014.59"
# -- global settings:
variants:
- [ platform-linux ]
help: google-chrome http://download.autodesk.com/global/docs/maya2014/en_us/index.html
commands: |
# TODO: support padding in rez
env.MAYA_VERSION_NUM = version.major
env.CMAKE_MODULE_PATH.append('{root}/cmake')
if machine.platform == 'linux':
env.MAYA_LOCATION = '/usr/autodesk/maya{version:#0.#01}-x64'
elif machine.platform == 'darwin':
env.MAYA_LOCATION = '/Applications/Autodesk/maya{version:#0.#01}/Maya.app/Contents'
elif machine.platform == 'windows':
env.MAYA_LOCATION = 'C:/Autodesk/maya{version.major}-x64'
env.PATH.prepend('$MAYA_LOCATION/bin')
env.MAYA_DEBUG_ENABLE_CRASH_REPORTING = 1
tools:
- maya
- mayapy
---
# -- version overrides:
version : 2012|2013
requires:
- python-2.6
# local variant override for 2012/2013:
# moving forward we will only support linux
variants:
- [ platform-linux ]
- [ platform-darwin ]
---
version : 2014
requires:
- python-2.7
The ---
separates yaml sub-documents. The first sub-doc provides the global values that apply to all versions, and the remaining sub-docs provide version-specific overrides.
On the one hand, I know that package files are meant to be immutable once released, and while I try to stick to that rule for built packages, for external packages I find it very difficult. Adjustments sometimes need to be made that affect the package.yaml of every version of a package, or, even worse in terms of automation, most of them but not all. This single file gives a comprehensible birds-eye-view of the differences between versions. I put these external package definitions under version control and manage new releases and modifications by editing and committing these files.
Another perk is that when editing many rez package files it is much easier to keep track of what package you're editing, since they're not all named package.yaml.
One issue that I still need to address is time-stamping, as I'm still a novice with rez's timestamp mechanics. I'm hoping that these can be handled either automatically in an external side-car file (e.g. .maya-release.yaml) or by manually, by editing a timestamps
metadata field in the package file. I'm open to suggestions.
To me this is a killer feature and I'd like for it to be included in rez, but I'd like to hear what people think.
And ensure that resetenv is working correctly wrt this. Relevant test cases should be added to the commands unittest.
It should write appropriate format matching given filename extension. Specifically, including .dot.
rez-build will not complain when version value is missing from package.yaml and install a local package in ~/packages/package_name/None/Linux/
Hey there,
There is a lot of excitement at my company with adopting rez-config for software configuration ; it feels like it's gonna fix a bunch of headaches for us here. Big thanks for making rez-config.
One thing that I'd love to have thought is not forcing a shell on users ; currently it looks like bash is the only option for working with rez. If you look at virtualenv for python, which also is a system that change the environment (to run a different version of python), it can happily work with a different shell (zsh or tcsh).
It looks like what you'd need is to move to subcommands; this is what git, mercurial, subversion etc ... do; Instead of typing git-add, you just go git add (git SPACE add), and here you go. So this is the only CLI user interface change that people would have to do. git would be rez in your case, and rez would be a python script. In order to not dump the whole source code in it you can still split the code base per sub-command and just import the subcommand.
So you'd go I took a quick glance at your source code and you guys are using optparse in python. If you were to switch to argparse (standard in python 2.7, but it can be backpatched to 2.5, I did that), you would have subcommand support for free.
=> Thoughts ? Would a small throw-away fork on github make sense to see what I have in mind ?
Thanks !
ps:
the problem is simple:
$ echo $MAYA_LOCATION
$ rez-env maya
> $ echo $MAYA_LOCATION
/usr/autodesk/maya2014.51-x64
> $ rez-env nuke
>> $ echo $MAYA_LOCATION
/usr/autodesk/maya2014.51-x64
rez makes no effort whatsoever to remove old variables.
This is a pretty big problem. what if you drop into a shell with a resolve that sets LD_LIBRARY_PATH
, then drop into another shell below that which does not. LD_LIBRARY_PATH
is still set, potentially along with dozens of other variables which are not relevant and potentially harmful to the current resolve.
I believe if we can keep track of which variables rez sets, then we can unset all of those before the next resolve. This is one of the reasons that I'm not a fan of the source
command, since it can wreak total havok on the environment in ways that rez cannot easily track (aside from the fact that it breaks promise of parity between various shells).
python -V reports 2.7 so when striping the last ., python version appears as 2 and the test version < 25 fails. That should be easy to fix. If rez does not support python 3 it would also be good to make a test to fail configure if python version > 3
Timestamps are currently associated with packages, not variants. This needs to be changed, otherwise we cannot facilitate adding of variants to a package after it's been installed - something we're going to need to do.
some still exist in docs, one_liners.txt at least
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.