Windows support about rez HOT 40 CLOSED

There is substantial work towards a pure python implementation if i am not mistaken. I would be interested in helping out towards windows too! I think the majority of python work is in the luma pictures fork branches but i am not really up to date on that.

Cheers,
Thorsten

from rez.

Hi,

We aren't ready for an official release yet but we have been working on a
Rez-2.0 branch which brings with it quite a lot of changes. These include
amongst other things cross- shell support - there is no longer a bash
dependency, instead support for new shells can be added as plugins (bash
and tcsh come as builtin plugins). Much of the work in making Rez cross
platform has also been done in this branch, although while Linux and Mac
are supported, work does still need to be done to add windows to that list.
But 2.0 has certainly done a lot of the groundwork here. And yes, it has
pure python support now also, meaning you can resolve new environments
directly inside python, which alters os.environ and sys.paths. It shouldn't
take much to get pure python resolves working in windows.

Even though it's not quite ready, the 2.0 branch is largely functional. If
you're interested in adding windows support, I would definitely start
there. Work you'd probably want to tackle first might include adding Cygwin
as a supported shell? As well as getting pure pytho resolves working, as I
mentioned earlier.

More news on 2.0 soon, stay tuned.

A

On Tuesday, May 6, 2014, Thorsten Kaufmann [email protected] wrote:

There is substantial work towards a pure python implementation if i am not
mistaken. I would be interested in helping out towards windows too! I think
the majority of python work is in the luma pictures fork branches but i am
not really up to date on that.

Cheers,
Thorsten

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42318853
.

from rez.

Cool, thanks for the heads up.

@instinct-vfx, how about we smash our heads together and give this a go?

I can imagine there is a few things to catch up on, especially since I have no experience with the library to begin with.

@nerdvegas where would be a good place for us, and potential others, to collect our thoughts? I'm thinking either continue posting in this thread, or if you had a forum/mailing-list setup? If not, I'm running a few over at Google Groups and would be happy to set one up for this.

from rez.

Hi Marcus,

yeah i'd happily join on in. I have no real experience with rez either (being a windows shop). I am not really a fan of cygwin. It kinda introduces a parallel world on a windows machine.

Cheers,
Thorsten

from rez.

The Rez google group is here:
https://groups.google.com/forum/#!forum/rez-config

It hasn't seen much recent action, but with 2.0 on the way that could
change. It would be good to start activity on it once more.

Thanks,
A

On Tue, May 6, 2014 at 1:58 PM, Marcus Ottosson [email protected]:

Cool, thanks for the heads up.

Thorsten, how about we smash our heads together and give this a go?

I can imagine there is a few things to catch up on, especially since I
have no experience with the library to begin with.

@nerdvegas https://github.com/nerdvegas where would be a good place for
us, and potential others, to collect our thoughts? I'm thinking either
continue posting in this thread, or if you had a forum/mailing-list setup?
If not, I'm running a few over at Google Groups and would be happy to set
one up for this.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42358003
.

from rez.

Hey guys,

I don't have much of an opinion RE cygwin as I haven't used Windows in a
long time. You will see an unfinished example of windows shell support in
the codebase (plugins/shells/windows.py), perhaps it would be good to take
a look at that.

Shell support aside, it's probably going to be easiest to get the python
API working first. Rez-2 has a setup.py-based install, so first thing to do
is see if there are any problems with the installation. After that, I'd
suggest poking into the API in this order:

• settings.py - see if the settings are loading properly. See if the
  "rez-settings" cli tool works.
• system.py - this is an abstraction layer to the system, and reports
  things such as platform ("windows"), arch, OS etc. Check that these are as
  expected.
• the version submodule. Note that this isn't quite in the 2.0 branch yet,
  but I will be merging it very soon, probably tomorrow. You will find it in
  contrib/version, it has a unit test and is well documented. Have a read and
  see that the unit test works.
• the solver (solver.py). Again this isn't merged, but it's in the same
  branch as the version submodule and will be in tomorrow. It also has a unit
  test, which you should be able to run using the "rez-test" cli tool.
• rex. Rex is the submodule that allows you to write python code that
  agnostically describes actions to apply to a shell. Shell plugins supply
  the means to translate this code into bash, tcsh etc. You don't need a
  windows plugin right away though - you can use Rex purely within python,
  there's a builtin rex interpreter called "Python" that does this. There is
  also a unit test (tests/rex.py) which doesn't use any shell plugins, and
  should work on windows. You can run this test via the rez-test cli tool.
• resolved_context.py. This is the entry point in the API for creating
  resolved environments. In 2.0 this is much improved, using this class is
  quite simple. You should be able to create a context, save it to disk, look
  at its graph etc, but you won't be able to spawn an interactive shell of
  course, because the windows shell support doesn't exist yet.

If you get all of the above working or show that it already does work,
that's a lot of progress already! I've tried to put this list in logical
order - the earlier items have less dependencies.

Good luck and let me know if you run into problems.

thx
A

On Tue, May 6, 2014 at 2:32 PM, Thorsten Kaufmann
[email protected]:

Hi Marcus,

yeah i'd happily join on in. I have no real experience with rez either
(being a windows shop). I am not really a fan of cygwin. It kinda
introduces a parallel world on a windows machine.

Cheers,
Thorsten

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42362021
.

from rez.

So i guess i will update my fork and see. Should we report/track windows issues in this project? Or better the fork?

For starters: os.symlink is not available on windows. Pathing that up with a custom replacement in _sys_setup.py lets me install rez via setup.py install. Next up some testing later on.

Cheers,
Thorsten

from rez.

Could you link to that fork?

from rez.

I re-forked to start from a clean slate. The fork is here:
https://github.com/instinct-vfx/rez

The fork has a new branch 2.0-windows which already contains said fix.

Cheers,
Thorsten

from rez.

Cool, let's await @nerdvegas reply on whether to track here or in the fork. My vote goes to the fork, I think it might help keep things focused.

from rez.

I agree on tracking in the fork, at this early stage at least.

Cheers
A

On Wednesday, May 7, 2014, Marcus Ottosson [email protected] wrote:

Cool, let's await @nerdvegas https://github.com/nerdvegas reply on
whether to track here or in the fork. My vote goes to the fork, I think it
might help keep things focused.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42415236
.

from rez.

Cool, thanks Allan.

@instinct-vfx if you could enable issue-tracking on your fork, we could get chattin' :)

from rez.

Done Marcus. Also added you as a contributor with full access there.

from rez.

Hey guys,

I've just merged the 'versions' branch with 2.0. It contains a lot of changes, including a couple that should help with the windows port a bit. Specifically it has a new suite of fairly extensive unit tests which will help validate that the port is working.

Cheers!
A

from rez.

Changes merged, thanks Allan.

from rez.

Hi folks,
I've been working on changes to the setup process, which I've yet to roll out. I'd be interested in any windows-specific changes to setup that need to be made. For example, setuptools provides an entry_points feature which automatically generates windows .exe's for the cli scripts, but we're not using it yet. Is this is a requirement for proper windows support?

from rez.

Not sure, Windows has (of late) been capable of running .py scripts as though they were binaries, thanks to PEP397 so it might not be a requirement.

I haven't looked into this feature of setup.py yet, but I'll let you know what comes out of it when I do. Maybe @instinct-vfx knows more.

As far as I know, the .exe's it "generates" are merely copies of a single binary that all run a file named after itself followed by "_script.py" and can be copied and renamed by hand, so it shouldn't be a problem even if setuptools fails us.

I'll keep an eye open for other Windows-specific parts.

from rez.

It's tough to define "proper windows support" in that case. pylauncher while nice and long overdue is only part of python as of 3.3 if i am not mistaken, which would make it an odd additional dependency. As in terms of "proper support" it is fair game to have to call rez via "python rez-..." i think. At least initially. I think we should concentrate on core functionality first and then go for "cosmetic" and convenience stuff. I will try to take a look at the new testing stuff tonight. Things are pretty busy here (yeah i know, it always is :( )

from rez.

How are the new tests supposed to be used? I remember reading about bez and a unittest2 backend, but am unsure how to give that a go.

Cheers,
Thorsten

from rez.

I think merely running those tests on a Windows station would reveal what works and not, then we can use the errors as starting points for figuring out what is missing.

from rez.

I hope so yes!

from rez.

There is a rez-test cli tool that runs the unit tests.

That might be a chicken and egg problem for you though. I'll have a look, I
think those tests should run directly (if in an env containing Rez as a
standard python module that is), in which case I'll add the main code
so you can run them with python directly.

On Monday, May 12, 2014, Thorsten Kaufmann [email protected] wrote:

I hope so yes!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42887671
.

from rez.

@instinct-vfx Did you mean how to physically make use of the test, as with the rez-test tool? Sorry, I thought you meant in practice, as in "how do use tests". I haven't got much experience with other peoples tests and I haven't had a chance to look at these.

How about we do a form of "teambuilding" to get to know each others strengths and weaknesses a bit better? :) Otherwise, we might end up talking past each other often, and then others could read up on it and join in in time. I'm thinking in the form of an "issue" in your fork, what do you say?

@nerdvegas Out of curiosity, what's the motivation for having a dedicated cli tool for tests? From what I've seen around, they're either test.py scripts meant to be run on their own, or something like nose which scans a directory tree for patterns of tests.

from rez.

On Monday, May 12, 2014, Marcus Ottosson [email protected] wrote:

@instinct-vfx https://github.com/instinct-vfx Did you mean how to
physically make use of the test, as with the rez-test tool? Sorry, I
thought you meant in practice, as in "how do use tests". I haven't got much
experience with other peoples tests and I haven't had a chance to look at
these.

How about we do a form of "teambuilding" to get to know each others
strengths and weaknesses a bit better? :) Otherwise, we might end up
talking past each other often, and then others could read up on it and join
in in time. I'm thinking in the form of an "issue" in your fork, what do
you say?

@nerdvegas https://github.com/nerdvegas Out of curiosity, what's the
motivation for having a dedicated cli tool for tests? From what I've seen
around, they're either test.py scripts meant to be run on their own, or
something like nose which scans a directory tree for patterns of tests.

Rez is different from most other python modules, because it is itself a
package manager. Often you are using Rez via cli tools, in an env not
configured for Rez the python module (the cli tools are especially patched
to be able to deal with this). So, when I want to run the unit tests, I
can't run them directly without fudging around with sys.path. Rez-test is
an easy way to guarantee that the tests are run in the correct environment.
Python version is an issue also, if you run the tests directly from a non
configured env, you could easily be using a python binary different to the
one Rez is using.

Having said all that, I don't think Rez-test and nose etc are necessarily
mutually exclusive, but for simply ease of use, I want Rez-test to stick
around. Especially considering the audience - Rez ends up getting used by
artists, tds etc in a studio. If someone calls me with a problem I like
that I can say, "run Rez-test and tell me if there are any problems".
That's about as simple as it gets.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-42921906
.

from rez.

Ah, that makes sense, very good. Thanks the thorough explanation.

from rez.

Hey @instinct-vfx and @mottosso,

Just wondering how you were getting on with your windows branch of Rez 2.0? Is it in a working state?

Mark.

from rez.

Hey Mark, sorry for the late reply. For some reason I wasn't notified.

In my case, as it turns out, Rez is solving problems we aren't having at
the moment so I will have to return once we're there. I'm still receiving
enquiries every so often from folks interested in a Windows build so the
demand is certainly there. I'm trying to stay up to date on developments on
the 2.0 front as well and it seems Windows is getting its fair share of
thought, that's great and I hope to see more of it.

Best,
Marcus

On 4 July 2014 03:30, Mark Streatfield [email protected] wrote:

Hey @instinct-vfx https://github.com/instinct-vfx and @mottosso
https://github.com/mottosso,

Just wondering how you were getting on with your windows branch of Rez
2.0? Is it in a working state?

Mark.

—
Reply to this email directly or view it on GitHub
#55 (comment).

Marcus Ottosson
[email protected]

from rez.

Hey Mark,

sorry too. I am still really interested, but ran out of time to really spend on rez. I will be out of country a bit and then on vacation next so will be pretty much unable to do anything for the next month at least. We are moving towards a bunch of bigger changes in our pipeline and at the moment i am torn if this is the perfect moment to add rez to the mix or if it will cannibalise the other efforts.

Sorry for being so unresponsive!

Cheers,
Thorsten

from rez.

Hey,

No worries, thanks for replying!

I would be curious to know if you got it running at all/spotted any obvious issues that you could maybe pass on some information about - or did you not get that far?

Mark.

from rez.

I have not really gotten to a point where i could say "it works" or "it does not" sadly

from rez.

I started looking into it and right off the bat there is an issue with trying to use os.getppid(). https://github.com/giampaolo/psutil could be included as a vendor package to find the parent process id on Windows. Unfortunately it's a compiled python package so it may make it a bit trickier to deploy.

Once I got past the ppid issue, most everything else "appeared" to work correctly. The support I was looking at adding was limited to the Git Bash terminal, not Windows cmd. I haven't had the time to dig too deeply yet but plan to in a month or two.

from rez.

I did want to utilise psutil, however I've decided not to include it in Rez because it's compiled as you mention.
What I think should happen is the code that detects the process chain should move into platform_, then a specific implementation can be written for Windows. Coupled with this, the shell detection code should be moved into the shell plugins, and this process chain would be passed to the plugin so it can use it to determine the shell type.

from rez.

Hi Windows peeps,

The thought has struck me that you may not realise that if you forgo the command-line tools, you should actually be able to use Rez on Windows without a hitch!

When you construct a ResolvedContext, that can be used to create an instance of a number of different shells (bash tcsh etc), and that's where work for Windows needs to be done. However - you don't need to create a shell if you don't want to. You can run the context in a mode that just constructs the target environ dict in python, and then calls a subprocess using this environ dict. So, if all your packages are just updating environment variables, and not doing anything funky like sourcing scripts or creating aliases etc, then this is all you need.

The function you want to use is ResolvedContext.execute_command(). Alternatively there is a get_environ() function, which will give you the target configured environment.

If you'd like to try it out, here's the code you need. The example assumes packages 'foo' and 'bah', whos commands would just be setting/appending/etc environment variables:

from rez.resoved_context import ResolvedContext
r = ResolvedContext(["foo", "bah"])
r.execute_command("which foo")

That's it! What I might do in light of this is add a flag to rez-env that would also let you skip the shell construction from the command line. This would allow you to run commands on windows via rez-env, like so:

]$ rez-env --shell=python foo bah -- which foo

thx
A

from rez.

Hello @nerdvegas, @mottosso, @instinct-vfx and @chadrik,

I got Windows and OSX support up and running at work on Friday and am now able to rez env on both operating systems.

For OSX there really wasn't much to do as it pretty much 'just worked' but I'll make a pull request with the changes next week. I can also rez build on OSX (but have not tested rez release or other features). Unit tests pass.

WIndows was a little tricker.

• There are changes to the install.py script which are fairly trivial and I'll make a pull request for these next week too - mainly some path handling changes for virtualenv.
• I created a cmd shell plugin to work with the Command Prompt and to replace the existing Windows plugin. Currently there is no shell detection so this is the default. I'll include this in the pull request. It doesn't include some of the more advanced features of Bash and SH plugins but gets us going.
• The API for shell plugins is really unclear. It wasn't obvious what methods the plugin should implement, and the inheritance between existing plugins is broken in some cases. UnixShell ended up being a better base class than the intended Shell. I think it would be good to make a pass at refactoring this before looking at other Windows shells such as PowerShell.
• Coloured output doesn't work - the control characters are printed. According to the colorise documentation we are doing right so not sure what is wrong there.
• There is a bug in the config module and the handling of ~ in config values related to Python's os.path.expanduser method (read the documentation on this method to see how it is implemented different across operating systems to better understand this problem). When getting the implicit packages from the config file (which are all weak), the ~ is being expanded, so the implicit package becomes C:\Users\platform-osx. This isn't a problem on Linux/OSX due to the different implementation of os.path.expanduser. My current thinking is to update the config module so only parts of the schema defined as a path (not just Str) are expanded. Thoughts?
• I haven't tried building/releasing yet. I expect there to be more problems.
• The unit tests don't pass. In part this is due to the rez.vendor.sh module which doesn't work on Windows, and various temp file issues. I'll try and tackle those.
• I need {system.home} as in issue #153 so will add this.

But I'll make a pull request with the changes I have so far as it's definitely a step forward and things 'work'. Hopefully this will enable someone else to pick up some of these other problems (such as the colour) as we are no longer starting from scratch.

It was actually pretty rewarding to get this up and running in such a short space of time (<4hrs) given I thought it's be pretty daunting. I haven't seen comments/commits saying others have already done this, so hopefully I'm not doubling up!

from rez.

nice work!  thanks, Mark.

-chad

—
Sent from Mailbox

On Fri, Feb 13, 2015 at 4:35 PM, Mark Streatfield
[email protected] wrote:

Hello @nerdvegas, @mottosso, @instinct-vfx and @chadrik,
I got Windows and OSX support up and running at work on Friday and am now able to rez env on both operating systems.
For OSX there really wasn't much to do as it pretty much 'just worked' but I'll make a pull request with the changes next week. I can also rez build on OSX (but have not tested rez release or other features). Unit tests pass.
WIndows was a little tricker.

• There are changes to the install.py script which are fairly trivial and I'll make a pull request for these next week too - mainly some path handling changes for virtualenv.
• I created a cmd shell plugin to work with the Command Prompt and to replace the existing Windows plugin. Currently there is no shell detection so this is the default. I'll include this in the pull request. It doesn't include some of the more advanced features of Bash and SH plugins but gets us going.
• The API for shell plugins is really unclear. It wasn't obvious what methods the plugin should implement, and the inheritance between existing plugins is broken in some cases. UnixShell ended up being a better base class than the intended Shell. I think it would be good to make a pass at refactoring this before looking at other Windows shells such as PowerShell.
• Coloured output doesn't work - the control characters are printed. According to the colorise documentation we are doing right so not sure what is wrong there.
• There is a bug in the config module and the handling of ~ in config values related to Python's os.path.expanduser method (read the documentation on this method to see how it is implemented different across operating systems to better understand this problem). When getting the implicit packages from the config file (which are all weak), the ~ is being expanded, so the implicit package becomes C:\Users\platform-osx. This isn't a problem on Linux/OSX due to the different implementation of os.path.expanduser. My current thinking is to update the config module so only parts of the schema defined as a path (not just Str) are expanded. Thoughts?
• I haven't tried building/releasing yet. I expect there to be more problems.
• The unit tests don't pass. In part this is due to the rez.vendor.sh module which doesn't work on Windows, and various temp file issues. I'll try and tackle those.

• I need {system.home} as in issue #153 so will add this.
  But I'll make a pull request with the changes I have so far as it's definitely a step forward and things 'work'. Hopefully this will enable someone else to pick up some of these other problems (such as the colour) as we are no longer starting from scratch.
  It was actually pretty rewarding to get this up and running in such a short space of time (<4hrs) given I thought it's be pretty daunting. I haven't seen comments/commits saying others have already done this, so hopefully I'm not doubling up!

  Reply to this email directly or view it on GitHub:
  #55 (comment)

from rez.

Mark, that's awesome news, thanks a lot.

On Fri, Feb 13, 2015 at 5:24 PM, Chad Dombrova [email protected]
wrote:

nice work! thanks, Mark.

-chad

—
Sent from Mailbox

On Fri, Feb 13, 2015 at 4:35 PM, Mark Streatfield
[email protected] wrote:

Hello @nerdvegas, @mottosso, @instinct-vfx and @chadrik,
I got Windows and OSX support up and running at work on Friday and am
now able to rez env on both operating systems.
For OSX there really wasn't much to do as it pretty much 'just worked'
but I'll make a pull request with the changes next week. I can also rez build on OSX (but have not tested rez release or other features). Unit
tests pass.
WIndows was a little tricker.

• There are changes to the install.py script which are fairly trivial
  and I'll make a pull request for these next week too - mainly some path
  handling changes for virtualenv.
• I created a cmd shell plugin to work with the Command Prompt and to
  replace the existing Windows plugin. Currently there is no shell
  detection so this is the default. I'll include this in the pull request. It
  doesn't include some of the more advanced features of Bash and SH plugins
  but gets us going.
• The API for shell plugins is really unclear. It wasn't obvious what
  methods the plugin should implement, and the inheritance between existing
  plugins is broken in some cases. UnixShell ended up being a better base
  class than the intended Shell. I think it would be good to make a pass at
  refactoring this before looking at other Windows shells such as PowerShell.
• Coloured output doesn't work - the control characters are printed.
  According to the colorise documentation we are doing right so not sure
  what is wrong there.
• There is a bug in the config module and the handling of ~ in
  config values related to Python's os.path.expanduser method (read the
  documentation on this method to see how it is implemented different across
  operating systems to better understand this problem). When getting the
  implicit packages from the config file (which are all weak), the ~ is
  being expanded, so the implicit package becomes C:\Users\platform-osx.
  This isn't a problem on Linux/OSX due to the different implementation of
  os.path.expanduser. My current thinking is to update the config module
  so only parts of the schema defined as a path (not just Str) are
  expanded. Thoughts?
• I haven't tried building/releasing yet. I expect there to be more
  problems.
• The unit tests don't pass. In part this is due to the rez.vendor.sh
  module which doesn't work on Windows, and various temp file issues. I'll
  try and tackle those.

• I need {system.home} as in issue #153 so will add this.
  But I'll make a pull request with the changes I have so far as it's
  definitely a step forward and things 'work'. Hopefully this will enable
  someone else to pick up some of these other problems (such as the colour)
  as we are no longer starting from scratch.
  It was actually pretty rewarding to get this up and running in such a
  short space of time (<4hrs) given I thought it's be pretty daunting. I
  haven't seen comments/commits saying others have already done this, so
  hopefully I'm not doubling up!

  Reply to this email directly or view it on GitHub:
  #55 (comment)

—
Reply to this email directly or view it on GitHub
#55 (comment).

from rez.

In an ideal world, I'd be able to use the same rezconfig file for both Linux, Windows and OSX installs.

However this doesn't quite work. For example, the prompt value is different between platforms - on Linux > is fine, but on Windows (at least for the Command Prompt) this must be $G as > is not directly supported. Other values such as tmpdir are affected in a similar way.

What do you think the best way of handling this is? I could:

1. add a section at the end of the rezconfig file for operating system overrides and update config.py to respect this, so:

   prompt: >
   windows:
   prompt: $G
   tmpdir: C:\temp

2. or make platform specific keys that would override the default ones, so:

   prompt: >
   prompt_windows: $G

Obviously I could just install rez separately for the different platforms, but that seems to be a missed opportunity.

from rez.

convert the rezconfig file to a python config file like celery's and sphinx's?

from rez.

I have created pull request 159. Take a look and let me know what you think.

Tests pass (ish - I've had to skip a few) and the Windows Command Prompt isn't 100% supported but it does work.

from rez.

We have Windows support!

from rez.