Compiling without linking (and linking afterwards) about shaderc HOT 5 OPEN

Just as background, what I'm trying to achieve is functionality similar to what object files and binaries are to C. I want to compile each GLSL source separately and then link them all together into a single binary. Does this actually exist with SPIR-V? I'm not even sure if SPIR-V defines the concept of "object files", but what I wanted to do is kind of similar to that.

I know you don't yet support linking multiple files to one binary yet, but is this what you plan to implement? A fictional example:

glslc -std=450 -c sum.glsl -o sum.spv
glslc -std=450 -c mult.glsl -o mult.spv
glslc -std=450 -c compute_main.glsl -o compute_main.spv
glslc -std=450 -c vertex_main.glsl -o vertex_main.spv
glslc -std=450 -fshader-stage=compute sum.spv mult.spv compute_main.spv -o compute.spv
glslc -std=450 -fshader-satge=vertex sum.spv mult.spv vertex_main.spv -o vertex.spv

I want to be able to pre-compile each shader file separately without linking and use them in multiple shader stages.

from shaderc.

In other words, I want a way to cache compilation to reuse them between shaders and not have to recompile everything every time. Is there a way to do that? Could glslc/glslangValidator support a way to have multiple compilation units before joining everything into a single .spv?

from shaderc.

This is a very reasonable request. Yes, our plan is to add features to allow you to combine modules in the same way a C or C++ compiler would. That's the usual concept of linking.

But the word "link" is also used to describe the combination of shader stages into a single graphics pipeline. For example, that is where the compiler matches the output interface of one stage with the input interface of the next stage. Glslang's linker does that, as well as other less obvious work such as:

• check for an entry point in a stage. That's the error you got.
• resolve sizes for implicitly-sized arrays
• assign bindings (IO mapping)

Shaderc relies on Glslang for the core compilation functionality: parsing into an AST intermediate representation, and translation from the AST into SPIR-V. Glslang currently requires the graphics-style "link" phase be run before generating SPIR-V from the AST. In theory Glslang could be refactored to remove that assumption, and then Shaderc could use the refactored functionality.

A "module combiner" is useful for more than just GLSL, so it would be developed in SPIRV-Tools. Then Shaderc could use it.

A formal problem is that Vulkan SPIR-V modules require at an entry point. So separate compilation requires breaking the rules a bit to allow the Linkage capability at least temporarily, until module-combination is finalized.

A caution: the meaning of GLSL code can change depending on what stage it's compiled with. For example, default precision changes between a vertex shader and a fragment shader. In the separate compilation scenario, the .spv file already has floating point type finalized after the initial compilation step.

from shaderc.

I was reading the SPIR-V specification, and it looks like it was not developed with supporting multiple compilation units in mind. Requiring an entry point is a consequence of that. They even mention that this would be done in a higher level step, before generating SPIR-V.

It looks like it might not be that easy to support this, it will probably need another "object" SPIR-V format or at least an extension to SPIR-V, otherwise it would not be valid SPIR-V, like you said.

By the way, this needs to be done with caution. It is very much possible that different compilers in the next few years develop incompatible ways to do this, and the supposedly standard world we have with SPIR-V would turn into a mess.

For now, as we can include different files and compile them all together, the caution you mentioned is not an issue. However, it will be a problem that must be solved before this is implemented. SPIR-V was really not created to support this, this will have to be solved in a higher-level intermediate format.

from shaderc.

Two levels of specification are involved.

SPIR-V is the base level. It does support separate compilation, when the Linkage capability is used. OpenCL has separate compilation and uses SPIR-V.

Vulkan requires that a SPIR-V module be fully linked, and so does not support modules with the Linkage capability. The work of separate compilation is pushed out of scope of Vulkan itself. The ecosystem just needs an agreement about what such a module should look like. It could be as simple as "in all respects like a Vulkan module, but allowing Linkage capability".

Possibly the only trick is to have the validator check all rules for Vulkan except to also allow the Linkage capability. This could be handled by adding a "SPV_EXT_linkage" extension, and agreeing that it means "allows Linkage capability". Then tools could process modules with that, but still fully link the module, and strip the extension and Linkage capability, before presenting it to a Vulkan implementation.

from shaderc.