berrak / fabricate Goto Github PK

This is not quite as nice, but is much better as soon as you start having
escaping issues, or spaces in your file names, like Ben did just now. For
example, try calling `shell()` on a command whose path has spaces in it,
and with a filename argument with spaces in it -- I couldn't get it to work:

{{{
run(r'"c:\program files\catdoc\catdoc.exe" -w "word document.doc"")
}}}

It doesn't work, no matter what backslash and quote combos I tried. I'm
probably not trying hard enough.

All of which to say, I think we need to "do it properly", like Popen, or
similar:

{{{
run(r'c:\program files\catdoc\catdoc.exe', '-w', 'word document.doc')
}}}

Or we could make it so if you gave a list it'd do it Popen-style, otherwise
it'd consider it a single command line string as it does now. But is that
bad -- more than one way to do things? And then the "right" way takes an
extra set of brackets, like:

{{{
run([r'c:\program files\catdoc\catdoc.exe', '-w', 'word document.doc'])
}}}

Some things are just too hard to automatically handle. For these cases, allow 
specifying deps and outputs manually for running a command. This could happen, 
for example, via keyword arguments to the run command.

This could be implemented with a separate Runner, but I am unsure how to use 
multiple runners in the same Builder.

This should be an enhancement.

Hello,

I get the following traceback on a quite simple fabric file.
Strangely enough, on the corresponding strace file, I get an "open" line before 
the "execve" one…


Traceback (most recent call last):
  File "./fabric.py", line 22, in <module>
    main()
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 1402, in main
    this_status = eval(action, globals_dict)
  File "<string>", line 1, in <module>
  File "./fabric.py", line 8, in build
    gen_source(1, s)
  File "./fabric.py", line 17, in gen_source
    run(VALIDSH, str(chain), '--gen', scenario)
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 1233, in run
    return default_builder.run(*args, **kwargs)
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 998, in run
    deps, outputs = self.runner(*arglist, **kwargs)
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 689, in __call__
    return self._runner(*args, **kwargs)
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 644, in __call__
    status, deps, outputs = self._do_strace(args, kwargs, outfile, outname)
  File "/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/validation/tests/fabricate.py", line 585, in _do_strace
    cwd = processes[pid].cwd
KeyError: '15151'

Here is the relevant portion of the strace file (with first lines where 15151 
pid appears)

15138 vfork( <unfinished ...>
15151 open("/proc/self/fd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
15151 
execve("/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/v
alidation/bin/bpp.sh", ["bpp.sh", "/home/clabaut/c478/c478_traducte"..., 
"/tmp/File2606627264771462039.tmp"], [/* 67 vars */]) = -1 ENOENT (No such file 
or directory)
15151 
execve("/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/v
alidation/runtime/bpp/bpp.sh", ["bpp.sh", 
"/home/clabaut/c478/c478_traducte"..., "/tmp/File2606627264771462039.tmp"], [/* 
67 vars */]) = -1 ENOEXEC (Exec format error)
15151 execve("/bin/sh", ["/bin/sh", "/home/clabaut/c478/c478_traducte"..., 
"/home/clabaut/c478/c478_traducte"..., "/tmp/File2606627264771462039.tmp"], [/* 
67 vars */] <unfinished ...>
15138 <... vfork resumed> )             = 15151
15151 <... execve resumed> )            = 0
15151 open("/etc/ld.so.cache", O_RDONLY) = 3
15151 open("/lib/libreadline.so.6", O_RDONLY) = 3
15151 open("/lib/libncursesw.so.5", O_RDONLY) = 3
15151 open("/lib/libdl.so.2", O_RDONLY) = 3
15151 open("/lib/libc.so.6", O_RDONLY)  = 3
15138 clone(child_stack=0x7f10de33aff0, 
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLO
NE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, 
parent_tidptr=0x7f10de33b9d0, tls=0x7f10de33b700, child_tidptr=0x7f10de33b9d0) 
= 15152
15151 open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
15151 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
15151 open("/proc/meminfo", O_RDONLY)   = 3
15151 open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such 
file or directory)
15151 open("/usr/lib/gconv/gconv-modules", O_RDONLY) = 3
15151 
open("/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/val
idation/runtime/bpp/bpp.sh", O_RDONLY) = 3
15151 clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f8d2392c9d0) = 15153
15153 execve("/bin/uname", ["uname", "-s"], [/* 66 vars */]) = 0
15153 open("/etc/ld.so.cache", O_RDONLY) = 3
15153 open("/lib/libc.so.6", O_RDONLY)  = 3
15153 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
15153 exit_group(0)                     = ?
15151 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=15153, si_status=0, 
si_utime=0, si_stime=0} (Child exited) ---
15151 open("/tmp/ttt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
15151 clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f8d2392c9d0) = 15155
15151 clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f8d2392c9d0) = 15156
15151 clone(child_stack=0, 
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f8d2392c9d0) = 15157
15156 execve("/lib/cpp", ["/lib/cpp", "-traditional", "-P", "-C", "-undef"], 
[/* 66 vars */]) = 0
15156 open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
15155 
execve("/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/v
alidation/runtime/bpp/bpp", ["bpp", "/home/clabaut/c478/c478_traducte"...], [/* 
66 vars */] <unfinished ...>
15156 <... open resumed> )              = 3
15156 open("/lib/libc.so.6", O_RDONLY)  = 3
15155 <... execve resumed> )            = 0
15155 open("/etc/ld.so.cache", O_RDONLY <unfinished ...>
15157 open("/tmp/File2606627264771462039.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) 
= 3
) = 0
15157 execve("/bin/sed", ["sed", "-e", "s,\\\\#,#,g", "-e", 
"s,\\\\\\\\,\\\\,g"], [/* 66 vars */] <unfinished ...>
15155 open("/lib32/libc.so.6", O_RDONLY) = 3
15156 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
15157 <... execve resumed> )            = 0
15156 open("/usr/share/locale/locale.alias", O_RDONLY) = 3
15157 open("/etc/ld.so.cache", O_RDONLY) = 3
) = 0
15157 open("/lib/libacl.so.1", O_RDONLY) = 3
15156 open("/usr/share/locale/fr_FR.utf8/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
15156 open("/usr/share/locale/fr_FR/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 ENOENT 
(No such file or directory)
15156 open("/usr/share/locale/fr.utf8/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
15156 open("/usr/share/locale/fr/LC_MESSAGES/gcc.mo", O_RDONLY) = 3
15157 open("/lib/libc.so.6", O_RDONLY)  = 3
15155 
open("/home/clabaut/c478/c478_traducteur_java_hgsvn/traducteurBversCJavaTest/val
idation/tests/RT7_2/in/m1_i.imp", O_RDONLY <unfinished ...>
15157 open("/lib/libattr.so.1", O_RDONLY) = 3
15155 exit_group(0)                     = ?
15157 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
15157 open("/usr/lib/charset.alias", O_RDONLY) = -1 ENOENT (No such file or 
directory)
15157 open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such 
file or directory)
15157 open("/usr/lib/gconv/gconv-modules", O_RDONLY) = 3
15156 vfork( <unfinished ...>
15161 execve("/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.2/cc1", 
["/usr/lib/gcc/x86_64-unknown-linu"..., "-E", "-traditional-cpp", "-quiet", 
"-C", "-P", "-", "-mtune=generic", "-march=x86-64", "-undef"], [/* 69 vars */] 
<unfinished ...>
15156 <... vfork resumed> )             = 15161
15161 <... execve resumed> )            = 0
15161 open("/etc/ld.so.cache", O_RDONLY) = 3
15161 open("/usr/lib/libcloog-isl.so.2", O_RDONLY) = 3
15161 open("/usr/lib/libisl.so.7", O_RDONLY) = 3
15161 open("/usr/lib/libppl_c.so.4", O_RDONLY) = 3
15161 open("/usr/lib/libppl.so.9", O_RDONLY) = 3
15161 open("/usr/lib/libpwl.so.5", O_RDONLY) = 3
15161 open("/usr/lib/libgmpxx.so.4", O_RDONLY) = 3
15161 open("/usr/lib/libmpc.so.2", O_RDONLY) = 3
15161 open("/usr/lib/libmpfr.so.4", O_RDONLY) = 3
15161 open("/usr/lib/libgmp.so.10", O_RDONLY) = 3
15161 open("/lib/libdl.so.2", O_RDONLY) = 3
15161 open("/usr/lib/libz.so.1", O_RDONLY) = 3
15161 open("/lib/libc.so.6", O_RDONLY)  = 3
15161 open("/usr/lib/libstdc++.so.6", O_RDONLY) = 3
15161 open("/lib/libm.so.6", O_RDONLY)  = 3
15161 open("/usr/lib/libgcc_s.so.1", O_RDONLY) = 3
15161 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
15161 open("/usr/share/locale/locale.alias", O_RDONLY) = 3
15161 open("/usr/share/locale/fr_FR.utf8/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
15161 open("/usr/share/locale/fr_FR/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 ENOENT 
(No such file or directory)
15161 open("/usr/share/locale/fr.utf8/LC_MESSAGES/gcc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
15161 open("/usr/share/locale/fr/LC_MESSAGES/gcc.mo", O_RDONLY) = 3
15161 open("/proc/meminfo", O_RDONLY)   = 3
15161 open("/proc/meminfo", O_RDONLY)   = 3
15161 open("/proc/meminfo", O_RDONLY)   = 3
15161 open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such 
file or directory)
15161 open("/usr/lib/gconv/gconv-modules", O_RDONLY) = 3
15161 open("/usr/lib/gconv/ISO8859-1.so", O_RDONLY) = 3
15161 exit_group(0)                     = ?
15156 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=15161, si_status=0, 
si_utime=0, si_stime=0} (Child exited) ---
15156 exit_group(0)                     = ?
15157 exit_group(0)                     = ?
15151 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=15155, si_status=0, 
si_utime=0, si_stime=0} (Child exited) ---
15151 exit_group(0)                     = ?
15152 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=15151, si_status=0, 
si_utime=0, si_stime=0} (Child exited) ---

At the moment if you have a function/target in build.py like so:

{{{
def check():
    return int(outofdate(build))
}}}

fabricate.py's main() will raise an ExecutionError() which it'll catch and
then print a message and exit with status 0. Should it do this -- ask Ber?
See s2's build.py for an example.

AlwaysRunner is meant to be a fallback in case none of the dependency tracking 
methods can be used. However, what is not mentioned is that this fallback also 
means that autoclean() stops working entirely.

This means that if one wishes to write a project that would work also with 
AlwaysRunner, cleanup must be written manually.

With the atimes_runner(), if you are editing/creating some files in your
build directory while a build is taking place, you can modify the atimes of
files, including sources files (say main.c).

This isn't a big deal for building, because for building fabricate treats
outputs the same way as inputs (just checks whether they're there and
haven't changed), but for autoclean it's a problem, because you might end
up cleaning the source file you edited.

Till we come up with a better solution, Berwyn's suggestion is to mark
outputs as "output?" on the first build. If you autoclean now, it'll ask
you to confirm before it deletes files. Then on the second one if we get
the same outputs we can mark them as true outputs in the .deps file
("output"), and autoclean can delete them immediately, as it does now. This
would work because the chances of the same file being edited at the same
time during a build is almost zilch.

However, it'd be nice to come up with a better solution for atimes_runner
to autoclean safely, so if you have any ideas, let us know!

This is an issue Tim Docker brought up in the mailing list. Here's his 
message:

-----
I have had some troubles with an older version of fabricate (1.08)
relating to the way the StraceRunner was handling the current working
directory when subprocesses were involved. I see that an attempt has
been made to fix this in more recent versions, by keeping a stack of
working directories.

But is this stack based approach correct? Given that it's possible for
different subprocesses to be executing simultaneously (and hence have
interleaved file access output in strace), I would have though that a
dictionary of working directories should be kept, indexed by pid, with

 - new entries created by fork/clone matches
 - entries updated by chdir matches
 - entries removed by exit matches
-----

As per Michael Haggerty
http://groups.google.com/group/fabricate-users/browse_thread/thread/64670b732d07
1b75
-- the .deps file is both huge and contains lots of redundant info for
larger builds.

Consider compressing the redundant data somehow -- a simple gzip might be
the simplest way, and for a 17MB file this would actually speed things up,
because gzipping is faster than writing big files.

There are other, perhaps more code-complex ways to compress this data too.
But if gzip is chosen, it'll mean the .deps is no longer human-readable, so
have an no-zip override for debugging (but how often does one read their
.deps file?).

What steps will reproduce the problem?
python fabricate.py true

What is the expected output? What do you see instead?
Expected: execute the program
Seen: help message

What version of the product are you using? On what operating system?
r27


Please provide any additional information below.

The problem is, OptionParser strip the first argument (the command name)
from the CLI arguments. See attached patch.

What steps will reproduce the problem?
When running commands involving changing current directory, which generates 
chdir line(s) in strace output.

What is the expected output? What do you see instead?
The pid variable should be obtained from the chdir line, but instead it's not 
assigned when a chdir line is mathced, so its previous value is retained. This 
may not always cause a problem as its previous value has certain probability to 
be the same as the current value (when the previous line of strace output is 
from the same process). When the previous line of strace output is from a 
different process, the problem occurs and fabricate cannot keep track of 
correct current directory, eventually giving incorrect file list in the .deps 
file.

What version of the product are you using? On what operating system?
The problem appears in revision 157 from trunk of svn repository. It 
occasionally shows up in one of my Ubuntu machines.

Please provide any additional information below.
The problem can be fixed by the attached patch, which simply adds a line to 
assign to pid the value from the matched chdir line.

If --quiet is passed no output messages are printed for commands to run
and files to delete.

Bry and I were wondering whether prioritising atime check over mtime check
in atimes_runner() would make a difference for the problem of external
processes modifying/creating files. Investigate.

As per Michael Haggerty's tests, on big projects fabricate is slower than
it needs to be (especially for do-nothing builds), in part because it MD5s
the same dependencies over and over for different commands. Think about how
to speed this up (without breaking anything).

Also investigate why strace_runner + mtime_hasher REbuilt both times for
Michael.

See Michael's fabricate-users message:
http://groups.google.com/group/fabricate-users/browse_thread/thread/64670b732d07
1b75

Using more descriptive names improves the code readability.

What steps will reproduce the problem?
1. attempt to import or run fabricate with python 3+

I wasn't necessarily expecting Python3 (python 3, py3k, whatever) to be 
supported, but am working on making a codebase that *uses* fabricate 
Python3-compatible.

I'm using fabricate 1.25 on both OSX and Linux with Python 3.3.0.

Attached is a patch that works for my uses but is probably not exhaustive.

At least in windows `subprocess.Popen('mycommand', shell=True)` is about 4
times as slow as without the shell, like `Popen('mycommand')`.

And our fabricate.shell() function always calls Popen with shell=True,
meaning it's a lot slower than it needs to be.

Bry and I discussed it, and we don't see any real problem with shell=False
being the default. If you really want bash/cmd.exe syntax, you can specify
`run('mycommand', shell=True)` manually. Or, if you want shell-style
globbing in Linux, you can use `glob.glob()` directly -- then your build
script would work the same in Windows as in Linux, too.

Any comments, Ber?

What steps will reproduce the problem?
1. Set 'parallel_ok=True' with any fabricate script.

What is the expected output? What do you see instead?
1. Parallel goodness.

What version of the product are you using? On what operating system?
1. Tip of tree.

Please provide any additional information below.
main(parallel_ok=True, jobs=1) begins creating the same command over-and-over, 
never stopping.

It would be great to have WAF automatically execute the steps required when 
files change.

This may allow something like SASS --watch which saves a lot of time.

The watch should allow to specify a target target so that only that is built 
automatically.

What steps will reproduce the problem?

I run 'fabricate.py tar czvf foo.tar.gz a.c b.c', but foo.tar.gz was not traced.
A simple patch is attached here,

Regards,

What steps will reproduce the problem?

Try to run non-existent command on OS X.


What is the expected output? What do you see instead?

Expected a sensible error. Got instead a NameError while trying to catch a 
WindowsError, which is undefined on OS X.


What version of the product are you using? On what operating system?

Fabricate 1.16, on OS X 10.6.7.


Please provide any additional information below.

Here's a simple, easy fix:


$ svn diff
Index: fabricate.py
===================================================================
--- fabricate.py    (revision 108)
+++ fabricate.py    (working copy)
@@ -165,7 +165,7 @@
     try:
         proc = subprocess.Popen(command, stdin=stdin, stdout=stdout,
                                 stderr=subprocess.STDOUT, shell=shell)
-    except WindowsError, e:
+    except OSError, e:
         # Work around the problem that windows Popen doesn't say what file it couldn't find
         if e.errno==2 and e.filename==None:
             e.filename = arglist[0]

What steps will reproduce the problem?
python fabricate.py -c

What is the expected output? What do you see instead?
Expected: run the autoclean() function
Seen: Usage message

What version of the product are you using? On what operating system?
1.02 (r25)

Patch attached.

I'm using the new (and pretty cool) TrackerRunner to compile and link a 
project. It works fine, even with parallel building turned on.

However, if there's an error with one of the files in the compile stage, 
fabricate will continue and run the link stage anyway.

See the attached zip file with tiny build script for repro.

For example, here's what the attached build script shows with parallel building 
turned off (or with AtimesRunner):

-----
gcc -c main.c -o main.o
main.c: In function 'main':
main.c:7:30: error: expected ')' before ';' token
main.c:9:1: error: expected ';' before '}' token
fabricate: 'gcc' exited with status 1
-----

Note that it stops immediately, and doesn't go on to the after() link stage.

But here's what it shows with TrackerRunner and parallel building turned on:

-----
gcc -c main.c -o main.o
gcc -c one.c -o one.o
gcc -c two.c -o two.o
main.c: In function 'main':
main.c:7:30: error: expected ')' before ';' token
main.c:9:1: error: expected ';' before '}' token
gcc -o main main.o one.o two.o
-----

There's the same error in the compile stage, but fabricate runs the link stage 
anyway.

Note that multiprocessing.cpu_count() returns 4 on my machine.

What steps will reproduce the problem?
1.pip search fabricate

What is the expected output? What do you see instead?

  I expect to see fabricate listed :)  Instead... I don't.

What version of the product are you using? On what operating system?

  Any version up to May 3rd, 2013

Please provide any additional information below.

  IWBNI projects that I work on could set up fabricate as a build-depends much like other python test tools and etc.

On the examples wiki pages, there's this annoying "flags or []" situation:

def link(build_dir='build', flags=None):
    objects = [oname(build_dir, s+'.o') for s in sources]
    run('gcc', objects, '-o', oname(build_dir, target), flags or [])

If shell() removed args that were None, it'd simplify this case. In Python,
it's usually bad to define eg "def link(flags=[]): ..." because the default
of [] is only evaluated once.

What steps will reproduce the problem?
1. ls ${TMPDIR:-/tmp} | wc -l
2. Run a build script (like the example provided)
3. ls ${TMPDIR:-/tmp} | wc -l

What is the expected output? What do you see instead?

The expected result is that the number of temporary files is similar before
and after doing the build. Instead, many new temp files are created. I
found around ten thousand files (strace output) produced by fabricate in my
/tmp directory.

What version of the product are you using? On what operating system?

fabricate version 1.13 on Ubuntu Linux 9.10 (karmic) with Python 2.6.4
(r264:75706, Dec  7 2009, 18:45:15).

Please provide any additional information below.

Here's a basic sample script I used for testing. It produces 100 new
temporary files when it is run.

#!/usr/bin/env python

from fabricate import run

for i in range(100):
    run(['dd', 'if=/dev/zero', 'count=1', 'of=foo%d' % i])

This is good for consistency, so users using the Builder directly don't
have to catch the run() exception by hand.

What steps will reproduce the problem?
1. Run attached script


What is the expected output? What do you see instead?

Case 1: If neither foo nor .deps exists when the script is run, then fabricate 
creates a .deps file in which "touch foo" has no dependencies.

Case 2: If foo exists but .deps does not when the script is run, then fabricate 
creates a .deps file in which "touch foo" depends on foo as an output.

I expected the .deps file in Case 2 to be the same as in Case 1.


What version of the product are you using? On what operating system?

fabricate 1.22
Python 2.6.5
Ubuntu 10.04, kernel 2.6.32-34-generic


Please provide any additional information below.

I believe this happens because a test for ignore(name) is missing. 
Specifically, lines 420-421 of fabricate.py are

if afters[name][1]-mtime_resolution/2 > befores[name][1]:
    outputs.append(name)

but I believe they should instead be

if afters[name][1]-mtime_resolution/2 > befores[name][1]:
    if not self.ignore(name):
        outputs.append(name)

What steps will reproduce the problem?

parse_option() (and main()) always parse sys.argv[1:], which may reduce 
usability of this function.

for example,

# foo.py
if __name__ == '__main__':
    import sys

    # setup parameter file is given as sys.argv[1]
    # it may include the config for fabricate

    env = load_config(sys.argv[1])

    # setup & run fabricate

    fab_env = pick_fab_env(env)

    import fabricate
    def myfab():
        fabricate.run('command');

    fabricate.main(default='myfab',**fab_env)
# end of foo.py

in this code, user run the command with './foo.py config_file', but fabricate 
tries to call 'config_file()' and fails.

What is the expected output? What do you see instead?

In order to solve it temporary, cmdline option was introduced to main() and 
parse_options() functions. When cmdline is explicitly given, parse_options 
works on it (cmdline=[] for nothing) instead of sys.argv[1:]

Ber copied a source tree (including his .deps) and it didn't rebuild, which
he spent quite a while tracking down. Change the .deps paths to relative
paths would fix this -- but would that introduce other gotchas?

Also, if you run your build.py script outside of the directory, it'll fail
because one usually doesn't specify absolute paths on the gcc command line.
Consider switching to the build.py directory before a build. (It'll also
write out the .deps file into the current directory, not the build.py dir.)

When sub-classing the Builder sometimes is useful to be able to override
the output of the programs when executing a command or removing a file.

This patch add 2 new methods: print_command() and print_delete() and
factor out some useful code for making filenames shorter when possible to
a new function shrink_path().

The patch depends on the patch on Issue 8 to be applied.

If a common build step is required by many targets, normally this build step 
gets run only once as fabricate notices that the sources and targets have not 
changed.

However, when running the same script with parallel execution enabled, the same 
build step gets run many times, in parallel, with potentially nasty results.

Using simple assignments is easier and more maintainable since there is no
need to update the "wrapper" if the original function changes.

Note that the patch attached depends on the patch in Issue 9 to apply cleanly.

What steps will reproduce the problem?
1. Make a build.py that uses fabricate to make a python virtualenv and forces 
use of atime_runner
2. be root
3. do 'python build.py build clean' and watch your 
/usr/lib/python2.7/config/Makefile (among other things) get deleted by the 
'clean'

What is the expected output? What do you see instead?

It shouldn't be deleted as it's not created.


What version of the product are you using? On what operating system?

latest stock fabricate.py

Please provide any additional information below.

The issue is that virtualenv makes a symlink to /usr/lib/python2.7/config/, 
which is this (incorrectly!) traversed by _file_times() in AtimeRunner.
The following patch fixes it:

diff --git a/fabricate.py b/fabricate.py
index ed8235b..e9c353e 100644
--- a/fabricate.py
+++ b/fabricate.py
@@ -353,11 +353,11 @@ class AtimesRunner(Runner):
                 fullname = name
             else:
                 fullname = os.path.join(path, name)
-            st = os.stat(fullname)
+            st = os.lstat(fullname)
             if stat.S_ISDIR(st.st_mode):
                 if depth > 1:
                     times.update(self._file_times(fullname, depth-1))
-            elif stat.S_ISREG(st.st_mode):
+            elif stat.S_ISREG(st.st_mode) or stat.S_ISLNK(st.st_mode):
                 times[fullname] = st.st_atime, st.st_mtime
         return times

Note that this is especially problematic on MacOS where the user running 
fabricate also quite likely is the owner of the lib/python../config/Makefile ; 
average linux users are protected by the fact that they don't own that file and 
so can't delete it.

What steps will reproduce the problem?
Under parallel_ok, and after() is called with some arguments, FAB.py script 
fails with exception. see attached files.
- FAB.py
- auto.sh (to run ./FAB.py in some conditions)
- log-1.24.txt (output of auto.sh)

What is the expected output? What do you see instead?
In parallel mode, run() registers groupname(and some information) in _group, 
and after() function waits the jobs listed in _groups. However, if a 
commandline is skipped due to cmdline_outofdate() or so, the group name is not 
registered to _groups, which causes key error exception. 

What version of the product are you using? On what operating system?
fabricate 1.24
Python 2.6.6
Linux (SL 5.8)

Please provide any additional information below.
I would like to suggest some modification to add groupname even if the 
commandline is skipped (_Groups.add_fake() method to register groupname 
anyway). I am not sure about this problem, so I am pleased with more better 
solutions.

While OSX has dtruss which has similar output to strace, it (and anything else 
using DTrace) requires root access making it impractical.  Here is an 
alternative implementation using DYLD_INSERT_LIBRARIES, similar to LD_PRELOAD 
on Linux, to detect what files are being accessed.  This requires bundling C 
code into fabricate.py, but seems to work correctly and quickly (probably 
faster than strace as it doesn't need to use regular expressions).

Now that run() takes a list of args, memoize() is defined as
run(*command.split()), which is a bit nasty. Need proper escape/quote
handling here -- i.e., a cmdline2list() type function.

What steps will reproduce the problem?

1. run('gcc', '-c', 'test.c', cwd='other')

What is the expected output? What do you see instead?

I would expect it to work. Instead, ".deps" gets created in the subdirectory 
and the paths there are wrong.

What version of the product are you using? On what operating system?

1.25 on Linux

Please provide any additional information below.

It seems that fabricate has no support for the "cwd" keyword argument. Adding 
this is fairly trivial, however. The "cwd" keyword argument needs to be passed 
onwards as the first process cwd, instead of the hardcoded '.' that is there 
now.

What steps will reproduce the problem?
1.pip search fabricate

What is the expected output? What do you see instead?

  I expect to see fabricate listed :)  Instead... I don't.

What version of the product are you using? On what operating system?

  Any version up to May 3rd, 2013

Please provide any additional information below.

  IWBNI projects that I work on could set up fabricate as a build-depends much like other python test tools and etc.

On the some servers strace shows stat calls as "stat(...)" instead of the 
usual "stat64(...)". I don't know why the difference. The strace version is 
on there is 4.5.13. So we'll have to figure out a portable way to handle 
this or fix it some other way.

As Berwyn said in an email (9 Sep 2009):

"Good find. Looks like we may need to make it more robust.

"Stat -e trace=file covers all the file operations (which also covers lstat 
which the man page says it would bet we'll forget).  And trace=process for 
all process operations, so presumably we'd want to do 
trace=file,process,chdir,mkdir.  trace=process may also cover exit_group 
which is used to get the exit code: I don't know.

"And then we have to write parsers for all of them, of course.  Including 
lstat and fstat (see their man page entries)."

What steps will reproduce the problem?

$ cat > foo.py
from fabricate import *

def build():
        run('touch', 'bar')

main()
$ stat foo.py
234881026 59660628 -rw-r--r-- 1 ljosa broad 0 67 "Sep 10 13:40:08 2009" "Sep 10 
13:40:06 
2009" "Sep 10 13:40:06 2009" "Sep 10 13:40:04 2009" 4096 8 0 foo.py
$ python foo.py
touch bar
$ stat foo.py
234881026 59660628 -rw-r--r-- 1 ljosa broad 0 67 "Sep 10 13:40:13 2009" "Sep 10 
13:40:06 
2009" "Sep 10 13:40:13 2009" "Sep 10 13:40:04 2009" 4096 8 0 foo.py


What is the expected output? What do you see instead?

I expected ctime to stay the same.  Instead it has been updated to the current 
time.


What version of the product are you using? On what operating system?

Fabricate 1.08 on Mac OS X 10.5.8.


Please provide any additional information below.

This is a problem mainly because emacs will complain that "foo.py changed on 
disk; really edit 
the buffer?" the next time I try to edit the already open buffer.

There are other programs like strace for other OSs. A good source for them
is Wikipedia: http://en.wikipedia.org/wiki/Strace#Other_Tools

Here is the current list presented there:
    * Linux distributions after 2006 have SystemTap
    * Solaris has Truss and DTrace
    * FreeBSD provides the Truss command, ktrace and DTrace
    * Mac OS X provides ktrace (10.4 and earlier) and DTrace (from Solaris)
      in 10.5 and later.
    * MS Windows has a similar tool called StraceNT created by Pankaj Garg.

Maybe they can be used to increase portability while keeping a good
performance.

What steps will reproduce the problem?
In the code shown in issue ID 30, build3 in FAB.py still causes exception.
# ./FAB.py -j 3 build3
Error: exception <type 'exceptions.AttributeError'> at line 824
Error: unexpected results handler exit

What version of the product are you using? On what operating system?
fabricate 1.24
Python 2.6.6
Linux (SL 5.8)

Please provide any additional information below.

It seems that this is due to the error of instance check around line 824
I would like to suggest a patch (including patch for issue ID 30) to correct it.

In spite of this modification, there is an error remains.
Please consider the following code, for example
  run('job1.sh', group='job1') # assigned to 'job1' group immediately
  run('job2.sh', after='job1') # assigned to False group firstly
                               # and then re-assigned to True group
                               #   after job1.sh finished
  after()                      # wait 'job1' and False groups only
                               # (True group is not listed yet)
In this case, after() cannot catch the finish of job2.sh.
This issue causes incomplete output of .deps file, etc.

memoize.py has a -d option to specify relevant dirs on the command line.
Also a -t option to use modtimes instead of md5sum.

These could be useful in Fabricate, particularly to make it function as a
drop-in replacement

When a program failed, the status code wasn't propagated. That tend to
break a lot of scripting that depends on knowing if the command was
successfully executed and what was the cause of the failure.

Plus, the user won't get an ugly Python traceback anymore when the command
fails =)

The idea of the "shell=true" argument is that the entire command line is 
invoked via the shell. However, when StraceRunner is used, the independent 
command-line arguments are merely appended to the strace line, converted to a 
command-line. This means that a command like "touch foo.txt; touch bar.txt" 
will result in strace invocation "strace -fo /tmp/tmp6VNsRF -e 
trace=open,stat,stat64,lstat,lstat64,execve,exit_group,chdir,mkdir,rename,clone,
vfork,fork,symlink,creat touch foo.txt ; touch other.txt", which will mean that 
"bar.txt" does not get marked as an output of the command: "touch foo.txt ; 
touch bar.txt": { "foo.txt": "output-d41d8cd98f00b204e9800998ecf8427e" }

The real fix is to instead translate such a line to "strace -fo /tmp/tmp6VNsRF 
-e 
trace=open,stat,stat64,lstat,lstat64,execve,exit_group,chdir,mkdir,rename,clone,
vfork,fork,symlink,creat /bin/sh -c 'touch foo.txt ; touch other.txt'", and not 
pass the shell=True to the underlying subprocess.Popen.

A command might do "gcc -c *.c". If a new file is added to that directory, it 
should cause re-running the gcc command.

For the strace runner, this could be achieved by tracking 'getdents' and 
similar syscalls, and recording in the dependencies that the command depends on 
the directory entries of the directory.

However, implementing this is probably somewhat difficult, and might cause a 
lot of spurious dependencies.

The system call rename is not detected when using strace, and because of
that the classic trick to output to a temporary file and then switch to
the final file using rename() to keep the file consistent is not detected.
GCC is an (important) example of this, object files were detected (as
inputs) by chance, just because they were stat64()ed.

This patch adds rename() detection, adding the destination file as an
output, which seems the most sensible thing to do.

The attached patch depends on the patch in Issue 12 to be applied.

What steps will reproduce the problem?
1. run the test/creat test via 'python build.py' in that dir
2. echo "// changes" >> test/creat/a.c
3. re-run 'python build.py'

What is the expected output? What do you see instead?

  I expect foo.tar.gz to get rebuilt ; it doesn't.  

What version of the product are you using? On what operating system?

  f10c4063ab4bc1f0ccb7a8bac3d9c7a485918652 on ubuntu 12.04

Please provide any additional information below.

  Looking in .deps, there's only an entry for foo.tar.gz as an output, no listed inputs.
  I did an strace and see not the expected open()s of a.c and b.c, but instead: openat(AT_FDCWD, "b.c", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) 

Looks like solving this will, as a by-product, solve the fchdir() problem.

What steps will reproduce the problem?
1. Run incremental fabricate.py build.

What is the expected output? What do you see instead?
Minimal md5 hashing, ie, only rehash those files that have changed, not every 
file that is dependent.

What version of the product are you using? On what operating system?
ToT; Windows 8 x64; CentOS 6.3 x64.

Please provide any additional information below.

By adding a (preferably python-only) daemon that runs "on the side" using the 
inotify/FindFirstChangeNotification APIs, it is possible to detect the set of 
files that are written to. These files can have their md5 update automagically 
"on the side". This will preclude fabricate from having to run the md5 hash 
during incremental builds.

What steps will reproduce the problem?
1. Apply patch from issue 30
2. Apply patch from issue 31
3. Down the FAB.py and auto.sh test program from issue 30
4. Run FAB.py

What is the expected output? What do you see instead?

I expect to see the list of commands printed instead I occasionally get the 
following error (among others):

Error: exception <type 'exceptions.AttributeError'> at line 836

In my code base this corresponds to the "while not _stop_results.isSet():" 
statement in the _results_handler function.  The two other errors were with the 
"_groups.set_ok(False)" and "if isinstance(a.do, tuple):" lines in that same 
function.

What version of the product are you using? On what operating system?

32bit
OpenSUSE 11.1
Python 2.6

Please provide any additional information below.

I have attached a path fixing these bugs that prevented me from using the 
parallel building mode.  The first issue was fixed by explicitly waiting for 
the _results_handler thread to stop when exiting.  The other two were single 
line fixes.