seattletestbed / buildscripts Goto Github PK

The initialize.py file does not recognize Github branches in the config_initialize.txt as valid index substitutes, and raises a repository not found error

Ex- https://github.com/SeattleTestbed/repy_v2/tree/branch_name ../DEPENDENCIES/repy_v2

initialize.py script has been modified to include the skip-mode functionality (i.e skipping the already exisiting directories inside DEPENDENCIES).
This change should be reflected in the build-scripts of all other repositories of SeattleTestbed. Hence, initialize.py should be updated in other repositories.

Reviewing @heatherm's PR SeattleTestbed/docs#39 shows that build.py and initialize.py treat their PWD and suggested target paths inconsistently. For example, running ../../repy_v2/build.py will use the correct paths to copy files from, though expecting its target dir inside the PWD at call time. OTOH, initialize.py requires its config file in the PWD.

We should clean this up and make initialize.py and build.py independent of the PWDand adopt a common understanding of where the target path is supposed to be located. This perhaps requires a doc change as well.

Proposed overall workflow:

$ cd COMPONENT
$ mkir TARGET_DIR
$ python scripts/initialize.py
$ python scripts/build.py TARGET_DIR

Now, build scripts only work on certain build config. However, installer-packaginghopes that build scripts can refer to sub-build-configs in dependencies repos that also get built in, so that we don’t need to sync dependency updates between the installer-packaging and other dependencies' build scripts. e.g. nodemanager, demokit, repy_v2, seash...

Thus, build scripts need to work on the build_config that we select. After adding this functionality, we also don't need to sync different repos updates.

I think we can add a dependencies (or similar) keyword that we can refer to sub-build-configs in dependencies repos and reuse these build config. For instance, putting these sub_build_configs to one build_config and remove duplicated lines.

If I correctly understand, then if a line in config_initialize.txt (the dependencies specification for a build) is intended as a comment, but the # is not the very first character (e.g. if the comment line is indented), it will not be properly recognized as a comment line and may be parsed incorrectly. It would fall through this check: https://github.com/SeattleTestbed/buildscripts/blob/master/initialize.py#L73
If we want to avoid that issue, we can simply do this instead of the existing line:

  if line.strip().startswith("#") or line.strip() == '':

Currently, initialize.py will fail (due to git clone's nonzero return value) if subdirs exist in DEPENDENCIES that would also be created from freshly checking out dependent repos.

I think we should add an option so that existing subdirs are skipped. This gives you a little more flexibility in the build process if you know what you do, e.g. copy over huge repos that you already have in order to save time / bandwidth, or build from repos you don't want to / cannot push to GitHub.

Ajay (@kellender) reports that editing an init config on a Windows machine breaks the build process. I can reproduce the problem by appending a \r (by typing ^V^M) on the end of a line of the init config. This causes the repo on that line to be git-cloned into a directory reponame\r; subsequently, the build script fails as the dirname it uses doesn't match the one we cloned into.

The issue is in initialize.py, which loops over the files of the init config here, but doesn't care to strip() off whitespace (which includes \r and \n!) when constructing the git command line here.

Apart from the obvious fix, we may consider dealing with line endings on the level of local Git configs.

Example

# My config_initialize.txt. Note the ``^M'' at the end of the line
https://github.com/SeattleTestbed/seattlelib_v2 ../DEPENDENCIES/seattlelib_v2^M

This clones seattlelib_v2 into ../DEPENDENCIES/seattlelib_v2^M. Then, firing up the build:

albert$ python build.py 
Building into /private/tmp/rv2/RUNNABLE
WARNING: File expression 'DEPENDENCIES/seattlelib_v2/*' does not match any files. Maybe the directory is empty, or the file / directory doesn't exist?
Done building!

History side-note

It has been brought to my attention that the fact that different operating systems historically use different character(s) to signal the end of a line of text should not be considered common knowledge.

Of relevance today are two different schools of line endings:

• Windows uses \r\n(often rendered in descriptive text as CR/LF, from old line-printer commands "carriage return" and "line feed"; sometimes labelled "DOS line breaks"), whereas
• all things Unixoid (Linux, Mac / the BSDs, ...) use \n.

This causes some cross-platform grief for Seattle time and again. See here for a list of open issues/PRs concerning Seattle and CR/LF. For more intricate detail, please refer to Wikipedia.