Compiler Chain separated from main executable about crown HOT 22 OPEN

This causes issues such as not being able to run different compiler chains at the same time [...]

Can you be more specific about that?

from crown.

Absolutely! As I wrote - this is under the assumption that you run a separate version of the main executable but with the compiler parts actually compiled into it. The assumption is that this separate version (which obviously can't be used as release-executable) hogs a specific port for network communication. If you plan on having several projects running at once - how will you be able to identify that the correct toolchain connects to the correct compiler using the correct port?

That is a very big problem with the engine which you are building everything upon.

A much better solution for the engine and also future use of such an engine is a separated compiler chain. Either you can use libraries from the engine to build the compilers, but that also opens possibility to use multiple machines for compiling resources (compile resource type A on machine a and resource type B on machine b etc) much more easily. Not to mention how much more easy it becomes for those using the engine that might want to add parts to the compiler chain. They can just add the compiler to a compiler-chain configuration rather than having to modify the source-code and struggle with code merging and most of all - changes in the engine.

from crown.

Another good example of the issues that the "main supercompiler which is the same as the engine"-causes:

If you have a target platform that you're going to compile for that isn't supported by LuaJIT and you use LuaJIT for all your other platforms (either with or without the jitting, but instead actually using the interpreter), you'll have to build a somehow complete separate version for compiling lua-code to that platform and incorporate that specific resource for that specific platform to be compiled on that very specific supercompiler. Instead it would be very easy to just build two very small separate compilers. One for the platforms using the luajit bytecode and one for using the normal lua bytecode. The option would in this case be to use the luajit bytecode on most platforms and pure text on other platforms which is quite sub-optimal. But maybe you see the issue - you can't link in two different versions of one library depending on the target platform.

from crown.

I can see the benefits of decoupling the engine executable from the compiler chain.
It feels like a much more flexible architecture and definitely a feature one would like to have.

On the other hand, I can imagine some valid points in having the two components combined as well, but I never had the chance to work with Bitsquid and so I could be wrong.

Honestly, I have not yet thought much about the compiler architecture and I'm open to suggestions/ideas, in fact, it would be very helpful if you had anything more to share about this topic.

from crown.

Well the benefit of having the two components combined is if you manually implement the serialization on a per-compile-type setting you will have the serialization code at the same place as your data structures. Like, you have your serialize() function for the "physics" part of your engine that does the serialization to/from ingame data for you, you'll probably implement it together with the rest of your physics-code.

A much more modern and flexible architecture would have a data-format that serializes your data for you. That way you can just define your data-structures either in the language of your choice and parse that for using it in other languages, or you could just stick to one language.

Then you can build your compilers specifically for compiling data which also opens up to a more flexible compiler chain down the line, such as having different compilers depending on which platform you are compiling for. If you go so far as to actually define an interface that all of your compilers are built upon you can choose to make one supercompiler (with all your compilers built into one executable) for one specific platform or have many multiple compilers (as long as you stick to the same language) which opens up for better adapting to really complex compiling that might have to be distributed to several computers or maybe even cloud-compiling.

from crown.

Hey again - yet another reason why to separate data-compilers from the main exe is because you might need different compilers for different libraries for different platforms. A good example of this is trying to build a tool-chain for xbox 360 AND xbox one at the same time. The first requires vs2010-compiler at latest (can't remember the version of msvc for that) and the latter requires vs2012. This makes it impossible to use one unified compiler for both platforms.

If you'd like some ideas / help regarding how to implement a separate toolchain I do have some examples of build systems that are free / open source and lua-based, if writing your own is not interesting.

from crown.

Data compilers are only built for development platforms (linux/win/osx) so I do not see the problem there. Am I missing something?

Anyway, it would be interesting to have a look at those build systems you mentioned. :)

from crown.

Ah yeah, the problem is that you'll most likely want a destination platform. Say that you want your engine to run on PS4 - you'll have to build the executable for the destination platform. What if you want to deploy on multiple platforms? What libraries will you link against if there should be different libraries for different target platforms?

from crown.

Makes sense although I fail to imagine a real-world scenario other than the lua/luajit example above. Most of resources are plain custom binary formats with no need for external libraries.

from crown.

It's a greater issue on libraries which have dropped some platform support on some versions or just simply collide when you try to link multiple platforms (symbols etc). I'll try to give you a run-through of a build-system which (I think) is more flexible tomorrow :)

from crown.

Cool! 👍

from crown.

Ok so here's the breakdown:

Most of the time I do things like this by using a 3rd party buildsystem. There's a variety of them out there: Bam (http://matricks.github.io/bam/), tundra (https://github.com/deplinenoise/tundra), Rake, AAP, Tup, etc etc. Some of them integrate nicely with continuous integration tools - other don't. Some of the continuous integration tools doesn't really care as long as it returns whatever has gone wrong or right.

Now I prefer something that is simple and easy to configure and use the way I want so I prefer to stick with something that has Lua (which I would recommend for crown since it uses lua as ingame-script as well) and just uses some sort of interface for you to work with.

The way that you should set up your buildsystem is to include a dependency graph (which most buildsystem include) of what things that a resource-type is dependent on. I know right now that probably all of your resources doesn't need multiple build-steps - but there might be platforms where you want to have multiple build-steps for resources and on other platforms you don't. Lua is a great example here where you might want to precompile the lua-code before it runs through your Lua to bytecode compiler. Or you might in the future need compilers that are dependent on say your physics-resources and your unit-resources in order to generate vehicle resources. Something like that.

So other than that you simply want to define a compiler for each and every resource. In the first step this will be easy since currently it's only your main executable that actually compiles resources. I would strongly recommend to separate the compiler-code to compilers rather than separating them at when you compile your engine executable so that you get smaller code units and all of that benefits. Not only that - some compilers you might want to write in a script language such as lua or python or whatever. Or maybe someone wants to use a proprietary compiler for something that they can't get the source-code for.

I would recommend that you use the same built system (preferably) for building your engine/tools/compilers rather than having multiple, but that's definetly not a requirement. Decent buildsystem can use other buildsystems to compile stuff.

Now - here's the tricky part. If you have multiple compilers then you're thinking that the serialization code has to be duplicated since you're going to use something that writes your format and something that reads it. I would consider this a none-issue if you start using a decent reflection-library. https://github.com/wc-duck/datalibrary is one that I use and I think that it works awesome. That also makes your resource-reading in runtime really really fast since you don't actually have to "parse" anything, it just patches up some pointers and you're good to go.

Honestly I'm a little hung over and I've probably rambled more than I should have. But from the different game engines that I've seen over the years this is definetly the most appealing solution that I've come across. Now this starts to pull into that you might want to share code between your game engine and your compilers which is where it might get interesting. Then you pretty much want to build your compilers and your game executable at the same time (so that the lib-files are up to date) which is something that I also strongly recommend :)

from crown.

If you would like some sort of example, I'd be happy to throw something together for you that would be a practical example.

from crown.

Sorry, I didn't have time to fully think about your approach and check out the suggested tools, yet. An example would be great! :)

from crown.

Hi just wanted to update you that I haven't forgotten this but I'm very slow and haven't really had the time to get back to you about this yet. I'm going to try to make an example this weekend :)

from crown.

Np, zero time here too, I place my hope in the weekend.

from crown.

Ok so this is a quick example of what I've used. It uses a build system called bam which uses lua. It's not really super well documented, but it's really simple to use and really powerful (and open source).

https://github.com/lundmark/example_content_compiler

It's really short and should be quite self-explanatory but basically the thing that I uses from BAM is the dependency-system and job-system (adddependency/addjob etc) so that any future issues regarding what is depending on what will be resolved by the build system so that I don't have to worry.

Naturally you could just implement all of this yourself in just a simple lua-script (which was my plan) but I didn't have the time for this unfortunately. The build system also detects when things has to be rebuilt so that I don't have to worry about that.

If you would like to do something like this for crown, I would suggest to start compiling content this way and just feeding your engine (which is your current compiler) with files by some sort of communication rather than the way it is now. And then in the future you should probably work towards separating out the compiler steps from the engine into compilers. If you want help I could probably try to put in some time into it but I doubt that I'll have much time the comming months.

from crown.

Just checked it out, got the idea, like it.
I am not too fond of using external stuff unless I have to; writing a custom dep/job system is probably a good idea.
I am in the middle of a huge refactor which involves a lot of systems and resource formats. I think I'll start preparing for compiler surgery just after that. If you want to contribute in any way you are welcome. :)

from crown.

Would you mind if it was implemented in Lua (using LFS for file-system stuff)?

from crown.

It's fine!

from crown.

I haven't forgotten about this, I just don't have that much time to put into it. I have the basic interface done, just need to work on the actual functionality - shouldn't be too hard once I get around to it. Just keeping you in the loop.

from crown.

👍

from crown.