Hello Franta!
Automatic isolation (from unity) of files that are worked on looks like a great feature. Unfortunately, as it is now, it is tied to how Perforce marks those files.

I would like to add support for other VCS (git in particular) for determining if file should be isolated, and I see 2 main ways to do this:

1. Integrate support into FASTbuild:

• Add String .UnityInputIsolationMethod to UnityNode, which would determine how isolated files are found out. It would accept values like 'none', 'auto' (automatic method detection), 'perforce' (current behaviour), 'git', etc.
• Rename .UnityInputIsolateWritableFilesLimit to .UnityInputIsolatedFilesLimit, to clarify the name (but accept old name to maintain backwards compatibility).
• Deprecate .UnityInputIsolateWritableFiles and treat setting it to true as setting .UnityInputIsolationMethod to 'perforce'.

Pros:

• Support for other VCS will work out of the box.

Cons:

• Too much changes to FASTbuild code base considering that this isn't a core feature. (Git support for example would require launching git itself or linking to libgit2.)
• Support for some VCS may require additional configuration which will require additional variables for UnityNode or Settings.
• There are many VCS and nobody is familiar with all of them, so maintaining support for them in one place would be difficult.

2. Add plugin support, similarily to .CachePlugin (maybe call it .VCSPlugin ?)
   Pros:

   • Minimal changes to FASTbuild.
   • Plugin implementation wouldn't be bounded by FASTbuild restrictions (mentioned in #22). For example Mercurial plugin would probably be python module with C interface.

   Cons:

   • Users will need to find and get those plugins manually.

Please let me know which approach (if any) do you like more.

The Test() function has the required parameter .TestOutput which, as far as I can tell, can only be set to a filepath on disk. I would like to see other options here, such as redirecting it directly to stdout.

I use FASTBuild from within Sublime Text which captures stdout and presents that live to the user. Seeing the output of my unit tests from within the editor, especially when they fail, would be very useful.

Or is this possible already and I missed something in the docs? Maybe with a magic string that couldn't be a path, such as .TestOutput = ':stdout'?

the website of fastbuild Quick Start Guide just for windows?There isnot for iMac?Thanks !

Hi,

I wonder if it's possible to override settings an a per file basis?

Say that you have something like this

.CompilerInputFiles  = { "file0.cpp", "file1.cpp " }
.Section 
{
   // These files needs to have opts disabled.
  .CompilerFlags = "-O0"
  .CompilerInputFiles  = { "file2.cpp", "file3.cpp " }
}

Is that possible?

Caching fails on objects which follow a name.cpp.obj format. This makes caching incompatible with things such as Unreal Engine 4 which uses this format.

The responsible bit of code is ObjectNode.cpp:389:

// use preprocesed output path as input
Node * sourceFile = GetSourceFile();
fullArgs.Replace( sourceFile->GetName().Get(), tmpFileName.Get() );

This ends up replacing the output file name as well, which causes the object to just get created in my temp directory, leading to the build eventually failing when it can't find the object.

My dirty not-so-nice solution was to look for a character (' ' or '"') at the end of the source file name.

//Replace Instances followed by space
{
    AString safeFileName;
    safeFileName = GetSourceFile()->GetName();
    safeFileName += " ";
    AString safeTempName;
    safeTempName = tmpFileName;
    safeTempName += " ";
    fullArgs.Replace(safeFileName.Get(), safeTempName.Get());
}
//Replace Instances followed by "
{
    AString safeFileName;
    safeFileName = GetSourceFile()->GetName();
    safeFileName += "\"";
    AString safeTempName;
    safeTempName = tmpFileName;
    safeTempName += "\"";
    fullArgs.Replace(safeFileName.Get(), safeTempName.Get());
}       

There's definitely better ways, this was just the fastest fix.

FBuildWorker creates a copy of itself in exe's directory and names itself "FBuildWorker.exe.copy".
This is so that updates via source control etc are not blocked by the application being open.

The current solution means that the FBuildWorker.exe cannot be in a network drive, and many workers referencing the same location. Because when a new worker spawns, it finds it cannot create the .copy file.

My suggestion would be to turn this into:
FBuildWorker.exe.hostname

Or to always copy the FBuildWorker into the local TMP files location on the local machine.

This would mean that the brokerage location can house the FBuildWorker, and when a new FBuildWorker is deployed to that directory, all workers will update.

Perhaps there are reasons to avoid this though?

BFFParser::ParseIncludeDirective always append the path of the currrent file, assuming that given string is a relative path.

Ex :

#include "c:\test.bff"

c:\test.bff(1,11): FASTBuild Error #1032 - Unable to open include 'c:\c:\test.bff'.

It will be really nice to have somekind of fbuild --list-build-targets
command switch that will enumerate all build targets one can build

The currently distributed binary only supports Linux platforms with GCC 4.9 (CXXABI_1.3.8 required).

This excludes Centos7 which has GCC 4.8.

Is there a way the distributed binary can target a lower CXXABI ?

If this is bothersome for you, I could otherwise write CMakeLists for the purpose of bootstrapping to a lower CXXABI.

Thanks

One useful feature I would like to see in FASTBuild is retrieving values from registry, that would be useful for example for getting paths and then copy executable after build, it's possible now if we set environment variable and use #import or with Exec and custom program. It's windows-only though. What do you think about it ?

In the example below FileA.hpp is not added as a dependency and modifications to FileA.hpp don't trigger a rebuild of main.cpp.

main.cpp

    #include "FileA.hpp"    // Missed since line starts with whitespace
#  include "FileB.hpp"      // Correctly added as a dependency
#include "FileC.hpp"        // Correctly added as a dependency

This is especially problematic for us as we include Bullet in our project and it uses tabs before #include all over the place.

P.S. Thank you for making fastbuild, its awesome!!

Hi again,

So, after my pull request about unity time stamp "fix" (#18) I continued to investigate my false-nothing-to-build-state while using unities.

I am French and my compiler does not output :

Note: including file: X:\A\B\C

But :

Remarque : inclusion du fichier : X:\A\B\C

Which is not recognized by the parser, resulting in ObjectNodes without dynamic dependencies.
The problem is coming from CIncludeParser::ParseMSCL_Preprocessed(), which does not handle compiler message localization :

In the end It was not just the unities, but all ObjectNodes that were missing their includes.
Your code is handling all the dependencies nicely, It just needs a better include parser ;)

Place source code of fastbuild in a folder that contains spaces in its name.
In my case it was c:\Users\User Name\Projects\fastbuild

Run fbuild on the code. the Windows res compilation will fail because command line for res compiler was malformed.

I guess this is by design, but I was wondering what the reasoning is behind the design decision to store the "timestamp of last built" and the dependencies found for each node into the serialized depgraph, instead of in a different file. If kept in a different file, I think you would no longer be in the situation that any .bff file change always causes a rebuild of everything

For security reasons we have FBuildWorker running under a special domain user, and removed this users read permissions for the D: drive. This results in HasEnoughDiskSpace Worker.cpp failing, as GetDiskFreeSpaceExA(...) returns false when access is denied.

My solution to this was to simply remove the second part of that if, but a more detailed way to check if it's something actually wrong with the drive might be needed.

We have a custom compiler/linker that links in clang which we use to generate reflection info for our engine and games.

The compiler emits a reflection "obj" file for each source file as well as some ancillary generated source.
The "linker" concats all the "obj" files into a big reflection db.

The kicker here is that FBuild doesn't appear to have a path for generating response files for clang and even if it did, we'd still have a bit of trouble with FBuild auto-detecting what to do based on tool file names.

Would it be reasonable to add an optional var to Library/Linker nodes which allows you to force response files?

Library('my-lib')
{
.UseResponseFiles = true/false/default
}

Hi, i wrote a small bash completion script for fbuild (including flags and targets completion)
its not exactly belongs to the code repository, whats the preferred way to distribute it?

Is there a plan to support modules?
It seems to be working in visual studio (I don't know about other platforms).

https://blogs.msdn.microsoft.com/vcblog/2015/12/03/c-modules-in-vs-2015-update-1/

Hi,

I need to call the preprocessor in my build chain. For that I use an ObjectList() with my compiler and just add '/P /Fi "%2"' to .CompilerOptions, which causes cl.exe to write preprocessor output to %2.

This fits in nicely in theory, but I got an access violation when running it. With a debugger I found the crash in ObjectNode::ProcessIncludesMSCL(), dereferencing output which was null. This is only asserted and not handled :

// FBuild\FBuildCore\Graph\ObjectNode.cpp(437)
// ProcessIncludesMSCL
//------------------------------------------------------------------------------
bool ObjectNode::ProcessIncludesMSCL( const char * output, uint32_t outputSize )
{
    Timer t;

    {
        ASSERT( output && outputSize );

        CIncludeParser parser;
        bool result = parser.ParseMSCL_Output( output, outputSize );
        if ( result == false )
        {
            FLOG_ERROR( "Failed to process includes for '%s'", GetName().Get() );
            return false;
        }

Called from here :

    // FBuild\FBuildCore\Graph\ObjectNode.cpp(247)
    // compiled ok, try to extract includes
    if ( ProcessIncludesMSCL( ch.GetOut().Get(), ch.GetOutSize() ) == false )
    {
        return NODE_RESULT_FAILED; // ProcessIncludesMSCL will have emitted an error
    }

The problem seems to come from the /ShowIncludes flag of cl.exe. On the MSDN we can read :

The /showIncludes option emits to stderr, not stdout.

For some obscure reason this is not the case when calling cl.exe with "/c %1 /Fo %2" or even by adding "/E" or "/EP", in those cases -and in fact most of the time- the output is written on stdout.
But when I add "/P /Fi" then /ShowIncludes will write on stderr, stdout will be empty and ch.GetOutput() will return nullptr.

I am building UE4 using the article from the wiki here. Sometimes when using remote workers the build fails with the following error:

cl: Command line error D8049 : cannot execute 'C:\tmp[...]\c1xx.dll': command line is too long to fit in debug record

I have observed the following:

• it happens only when cl.exe process is spawned on remote workers, when the file is compiled on the same machine where the build was initiated it's ok
• it happens when the arguments count sent to cl.exe exceeds around 20k characters
• when I execute CreateProcess in my own test aplication with the same compilation command it fails with the same error (which is very odd, because as I write above, it works fine from Fastbuild)
• using a response file doesn't help

I'm attaching the command I used to reproduce it in my test application. It's actually saved from Process.cpp (fullArgs parameter) in Fastbuild.

I'm on Win7, using VS2015 professional.

Command.txt

Hi,

The following code is not accepted by BFF parser :

.Count = 0
.Count + 1
Print( .Count )

Output :

fbuild.bff(2,8): FASTBuild Error #1027 - Cannot concatenate <Int> to <Any> for Variable '.Count'.
.Count + 1
       ^
       \--here

The code of BFFParser::ParseVariableDeclaration() checks that :

else if ( ( openToken >= '0' ) && ( openToken <= '9' ) )
{
    if ( concatenation )
    {
        Error::Error_1027_CannotConcatenate( operatorIter, varName, BFFVariable::VAR_ANY, BFFVariable::VAR_INT );
        return false;
    }

Is there a particular reason to forbid integer concatenation ?

I was expecting -clean to do this.

This flag should remove all object, executable, libraries and fdb files

Hi Franta,

first of all, I'm so glad you're working to further improve FASTBuild, which is already so brilliant, and that you found the time to host the sources on github. Seems it will pay off, you've already got forkers to contribute :-).

Just wanted to point at the links to fastbuild.org on your releases page here on github, those contained in "See fastbuild.org for:", seems they're broken, because they contain a backslash.

Other than that, keep up the good work, and, btw, I already succeeded in seducing few colleagues at work into using FASTBuild at least for inofficial, development builds of a large project 👍 .

Best regards,
Lucian

Hi !

I'm helping a bit on the CMake generator for Fsatbuild, and I've run into a problem, where very long command lines fail with Fastbuild.

When linking a large library, the command line can easily exceed 32K (I have 57K right here, and it's not the largest lib in the project) as CMake likes absolute paths better. And even if it used relative paths, this particular command line would be around 36K long.

I googled a bit and it seems there is a way around this:
http://blogs.msdn.com/b/oldnewthing/archive/2003/12/11/56043.aspx

I guess this would be a better way of passing information to the created process.

Best

For example like this one: https://github.com/fastbuild/fastbuild/blob/master/Code/Core/Core.bff

With this code (missing ' at the end of something)

.Test =
[
    .Foo = 'something
    .Bar = 'other'
]

One gets this error

fbuild.bff(30,1): FASTBuild Error #1002 - Matching closing token ] not found.

Which is a bit misleading as it's a ' that is missing and not a ]

The entire Data directory appears to be missing, resulting in tests failing. Example missing files:
Data/TestIncludeParser/fbuildcore.gcc.ii
Data/TestBuildAndLinkLibrary/fbuild.bff
Etc.

Trying to compile a hello world with 3000 threads takes forever to finish. It works "fine" for 1000, so i guess that's an issue spawning more threads than the ones needed to build the solution.

The dev branch version of Fastbuild outputs this in my french VS2015 :

1>  LINK : .netmodule ou module MSIL compilÚ avec /GL trouvÚá; redÚmarrage de l'Údition de liens avec /LTCGá; ajoutez /LTCG Ó la ligne de commande de l'Údition de liens pour amÚliorer les performances de l'╔diteur de liens

I did not check the code, but could this be related to recent changes ?

Hi, I'm currently evaluating fastbuild to replace our game engine build system. I've got everything setup and building ok but for one area.

We support runtime compiling of a game dll module which can be hot loaded while the game is being debugged in visual studio. To do this I need to be able to change the name of the game.pdb file on each dll compilation as visual studio will lock the current one.

I've got this working using and environment import to randomise the name, but doing this causes a reparsing of the .bff to occur and all dependencies to also be rebuilt. I thought this would be ok with a special dll declaration with dependencies removed for the rtc, but it also triggers a full game rebuild when you next come to build and run through visual studio (which is not going to fly)

I can't see a solution to this at the moment other than the extra .bff dependency work mentioned in #55. Any ideas?

In case a source code generator produces more than one output from the given inputs, the Exec function cannot be used safely, as it does not allow registering all the outputs of the task.

Use .o on unix

Sadly it doesn't provide much information as to why other than "This website contains elements classified as malware"

http://trafficlight.bitdefender.com/info?url=http%3A//fastbuild.org/&language=en_US

When a Visual Studio solution is generated, the generated GUIDs in the solution and project files do not match. When opening the generated .sln file, Visual Studio is forced to immediately update the GUID mismatch, resulting in a change in the opened file. This causes problems with version control.

There is a fix for this issue in pull request #52.

Showinclude write a human readable output, you should not parse it.
You can use tracker.exe instead.
It is used by MSBUILD to identify the input and output files of a process.
It will allow you to identify every files used in a build step and to know what is generated by this build step.
tracker.exe works with every kind of process: cl, lib, link, mt, midl... any.exe.

I'm not sure if I'm doing this wrong or if this is a bug but following yields an error.

.ParentScopeArray = {}
{
    .LocalVariableString    = 'hello'
    ^ParentScopeArray   + { .LocalVariableString }
}

Error:
FASTBuild Error #1026 - Variable '.LocalVariableString' not found for concatention.
        ^ParentScopeArray       + { .LocalVariableString }
                                ^
                                \--here

Using #import, if the given environment variable doesn't exist, the parsing of the .bff fails. this makes #import less useful than it could have been since if I use it anywhere, I now have to make sure this variable is always defined and I can't even do a simple ./FBuild in the .bff folder.
A reasonable solution is to add a second space delimited, optional argument to #import that would be the default value, in case the given name doesn't exist. this is similar to python's dict.get()
pro: gives maximum flexibility
con: some might find it a weird syntax
Another alternative is that #import just sets the variable to an empty string,
pro: no syntax change, simpler concept
con: less powerful
con: break backward compatibility as far as error handling is concerned.
let me know what you prefer and I'll code it.

Hi,
I have a main fbuild.bff that includes other ones:

include "lib1.bff"

include "lib2.bff"

...

include "exe.bff"

All is good but there are linker errors, so I modify linker options in "exe.bff" but it tells me "BFF file 'exe.bff' has changed (reparsing will occur)" and rebuilds everything including all libs.
I use '-cache' on the commandline but it doesn't seem to have any impact.
Thx

Found an issue with -config option :

FBuild.exe -config test.bff

This will use 'test.bff' and write the dependency graph in 'fbuild.fdb' (which is static).
Then if the user change the configuration file path :

FBuild.exe -config another_test.bff

The code will check first 'fbuild.fdb', which is valid since the last execution, and skip parsing even if -config changed.

I thing that ideally the path to the fdb file should be the same as the configuration file, except for extension.

I can't seem to figure out how to get fast build to not eat compiler warnings. You get to see them if there is an error but not after a successful compile.

Am I missing something?

Environment used: Windows 7, Visual Studio 2013

The problem appears with long argument lists, that are somewhat near the maximum command size (32767) boundary. The problem is that the total command length exceeds the boundary in certain cases, when the need for a response file is deducted wrongly.

Need for response file is deduced by the length of arguments provided for the linker. This deduction should also take into account the length of linker-command itself. In addition there are two slashes and a white space added in Process::Spawn, which adds further three characters to total length.

Following snippet demonstrates and fixes the immediate problem with linking process, but it seems that there are other call sites that use the same logic (LibraryNode, CSNode, ObjectNode). Additionally this fix is only for Windows, so probably a more general fix needs to be implemented (maybe let Process::Spawn() handle the response file logic?).

Problematic deduction:
Code/Tools/FBuild/FBuildCore/Graph/LinkerNode.cpp

Snippet:

const uint32_t MAX_COMMANDLINE_CHARS = 32767;
// Magic corresponds to number of characters added in Process::Spawn(), when __WINDOWS__ is defined.
const uint32_t MAGIC = 3;
const uint32_t totalCommandLength = m_Linker.GetLength() + fullArgs.GetLength() + MAGIC;

const bool useResponseFile = (totalCommandLength > MAX_COMMANDLINE_CHARS) && (GetFlag(LINK_FLAG_MSVC) || GetFlag(LINK_FLAG_GCC) || GetFlag(LINK_FLAG_SNC) || GetFlag(LINK_FLAG_ORBIS_LD) || GetFlag(LINK_FLAG_GREENHILLS_ELXR) || GetFlag(LINK_FLAG_CODEWARRIOR_LD));

Hello,

I have found a weird behavior of Alias() with the following code :

Alias( 'all' )
{
    .Targets = { 'to_be_defined' }
}
Exec( 'to_be_defined' )
{
    .ExecOutput = 'output.txt'
    .ExecInput = 'input.bff'
    .ExecExecutable = 'cmd.exe'
}

Which generates this confusing error :

fbuild.bff(5,1): FASTBuild Error #1100 - Exec() - Target 'to_be_defined' already defined.
Exec( 'to_be_defined' )
^
\--here

I would have thought Alias() was only referencing targets, not defining them.
The documentation seems to imply that targets must be already defined though :

.Targets = { 'Library-Alias'                         // A previously defined Alias()
             'tmp/Libraries/X64/Release/Core.dll'    // A previously defined DLL()
             'External/SDK/VS2012/libs/libcmt.lib' } // An external DLL import library

Is this behavior normal ?

Currently the only useful #defines are __WINDOWS__, __LINUX__, __OSX__ and anything that derives directly from those. It is not possible to add additional custom conditional compilation.
some use cases:

• "RELEASE_BUILD" - if I don't want to create separate structs for debug and release to avoid complexity
• "USE_SIMD_INSTRUCTIONS" - enables just a single compilation switch
• "WITH_GLES2_CG_SUPPORT" - add a "-DGLES2_CG_SUPPORT" flag to all compilation

The only way to do these things today is to add more and more configurations in an ever expanding matrix of options
I propose adding a command line option "-define NAME" that would add an implicit #define to the .bff parsing.
Attached is a patch that does this.

define_from_cmd.diff.txt

Hi,
I tried that with no success:
.UnityInputExcludePath = ' $ProjectPath$/_NotUsed1'
.UnityInputExcludePath + ' $ProjectPath$/_NotUsed2'
One line is Ok, but once I add the second line then none of them are taken into account.
thx,

I came across an error with this snippet :

.UntypedArray = {}
.Struct = []
.ArrayOfStructs = { .Struct }
.UntypedArray + .ArrayOfStructs

Which generates :
fbuild.bff(4,10): FASTBuild Error #1034 - Operation not supported: 'ArrayOfStrings' + 'ArrayOfStructs'.

Empty arrays seem to be typed by default as arrays of string when declared with {}, which later causes a typing error when I try to concatenate it with an array of structs.

The DirectoryListNode.DoBuild() does get called, and it actually searches the filesystem, however the ExeNode does not realize that the there's one fewer ObjectNode to link, and thinks everything is just dandy, instead of relinking without the objectfile belonging to the removed cpp file.

The CMake tests try to pass lots of strange characters by define, and the following fails:

-DTARGETSTRING=""hello`~!@^$*)(+-=}{][:^'.?/ ,&|^^<>world;""

Notice that there are two spaces after the forward slash.
Only the first space in a sequence is kept in this situation.

This could be simplified to:
-DFOO="" ""

In the resulting source file, FOO would be " " instead of " ".

It seems the fastbuild tokenizer does not recognize the escape double quote and simplifies the whitespace when it should not.

Thanks

Attempting to add a Dll to the .Libraries on and Executable fails. It seems as though the Node is an Exe rather than a Dll. I assume this means the dynlib just isn't supported yet?

FASTBuild Error #1005 - Executable() - Unsupported node type in 'Libraries'. (Node: '/Users/kylerocha/dev/Bump/Project/master_rengine/bin/OSX-Debug/dll/rEngineDLL.dll', Type: 'Exe')
Executable( '$ProjectName$-Exe-$Platform$-$Config$' )
^
--here

We have some use cases where it would be handy to extract different parts of a file path to compose a new path and or make things relative. Something to the effect of...

.ProjectSourceRoot = "C:\my_project\source"
.InputFilePath = "C:\my_project\source\game\main.cpp

.InputFileDir = "$InputFilePath:directory$"  
-- ouput: "C:\my_project\source\game\"

.InputFileName = "$InputFilePath:file$"
-- ouput: "main"

.InputFileExtension = "$InputFilePath:extension$"
-- ouput: "cpp"

.InputRelativePath = "$InputFilePath:relative_to($ProjectSourceRoot$)$"
-- output: "game\main.cpp"