It would be nice if we could have a diagram in the docs showing the client/worker setup.

E.g. com.graphicsfuzz.thirdparty.thrift-js
And the script that checks for dependency information should ignore com.graphicsfuzz dependencies, unless they are third party.

This:

{
  int x = ...; // Does not shadow any existing variables
  ...
}

Could be reduced to:

int x = ...;
...

I guess the path to delete must be incorrect.

We disabled it a long time ago as some drivers had a lot of problems with it; we should consider enabling it now.

This is due to a quirk of Ant (which we use from Maven for custom build tasks); it is not good at maintaining executable permissions. We relocate the binaries from their original location in their respective zip files into assembly-binaries-1.0.zip and this is when they lose their executable permission (we don't relocate going from assembly-binaries-1.0.zip to graphicsfuzz-1.0.zip).

We should try the walkthrough on a windows laptop and fix all the bumps we encounter.

This:

if ( _GLF_BLAH((X)) ) {}

may lead to this at the very end of reduction, when GLF macros are replaced by their expension:

if ((X)) {}

But this double brackets could be reduced to:

if (X) {}

Currently the UI reports "no results" for reference if the reference has crashed.

When running glsl-generate, if the output directory does not exist currently one gets uninformative messages. We should change this so that either a better message is given or the directory is automatically created.

With daily snapshot of logs and long experiments, the current log can reach 10MB, which makes the home page too long to load. A solution would be to only load last 1000 lines of the log for instance.

Test GenerateShaderFamilyTest::testGenerateSmallWebGL1ShaderFamilyMultiPass fails due to this; it is currently marked as @ignore.

We should set the build outputs to have good names.

The image histogram and PSNR metrics that GraphicsFuzz uses do not consider the alpha channel. They thus miss shader compiler bugs that lead to incorrect alpha computation.

We should consider ways to adjust these metrics so that they do consider alpha (or look at different metrics).

This:

for (int i = 0; i < int(1.0); i++) {}

Could be reduced to:

for (int i = 0; i < 1; i++) {}

The documentation states that runnable code should be available on the releases page, however it is not.

This isn't super important but I thought I would download it just to try it out.

We should probably exclude .pdb files (debugging info files) from the Windows binaries; they are large and you can download them manually if needed.

Ran into the following error running mvn package:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (1-check-style) on project parent-checkstyle: Failed during checkstyle configuration: Cannot set property 'cacheFile' to '<root path>/graphicsfuzz/parent-checkstyle/target/checkstyle-cache.txt' in module Checker: InvocationTargetException: javax/xml/bind/DatatypeConverter: javax.xml.bind.DatatypeConverter -> [Help 1]

This occurred on my Macbook.

This is not hard, but we have previously assumed that this type of thing would be removed. We should start trying to handle shaders that have not yet been pre-processed.

Running:

shader_translator exhausts_shader_translator_memory.frag

on this file, which must be renamed from .txt to .frag

leads to:

ERROR: 0:2823: '+' : memory exhausted

I have added a workaround to say that a shader is valid (according to shader_translator) when we get this error, otherwise we chuck away a lot of shaders and cannot reliably test whether we are generating valid shaders.

Let's check whether this reproduces on recent builds of shader_translator, and if so see whether it is intended behaviour.

The image borders in the WebUI indicate whether a result is likely a bug (red) or probably just round-off error (blue).

We should make sure the documents properly explain this, to avoid situations where folks run a reduction on a blue-border image.

At present if a worker with name foo has previously connected to the server, if foo connects again with different platform info, it is rejected. That's good as we want to avoid confusion.

But there are cases where someone might want to use a changed worker with the same name, reusing old test results, if they know what they're doing.

For this, we could provide a mechanism for the check to be over-ridden and the worker's new info to override the old info. I guess having an extra field in the thrift message a worker sends to present itself to the server would do the trick.

to check for issues before the CI.

• Is it a regex or just a string? (WebUI says regex; command-line option says string.)

• Do you need to provide quotes when passing in from the WebUI? (Whatever the answer, update the documentation.)

• Can characters such as "(" appear verbatim, or do they need to be escaped?

When lots of results are available, the WebUI can be slow to load. We should think about ways of improving this. Also it can be nice to see results trickle in without having to reload the page, so it might be good to have a facility to avoid refreshes.

The current support for builtin functions is a bit ad hoc: we simply supported a bunch of functions we knew mattered, but have not gone through this systematically per shading language version.

It would be great to get full builtin support for GLSL ES 1.00 and GLSL ES 3.00, for Android and WebGL 1/2 testing.

getBuiltins in TyperHelper.java is the right place to look for this.

I guess it only deletes the _exp folder, not the _inv

...that can be executed by the CI.

This is probably best for usability.

When performing metamorphic testing for wrong code bugs, it is essential that transformations are semantics-preserving.

However, when looking for crashes or sanitizer bugs, there is no need to have families of equivalent shaders. Rather, we want many interesting shaders.

With this in mind, it could be good to have a generator option, --preseve-semantics, that is true by default but can be set to false. If true, we get the current behaviour. If false, we get (a) generalized versions of some existing transformations that take less care to preserve semantics, and (b) new transformations that deliberately change semantics.

It would make sense to start with (a).

Here are some examples of where semantics preservation is enforced and could be relaxed:

In DonateLiveCode.java, the prepareStatementToDonate method makes sure that immediate break and continues from the donated code get removed, and that return and discard statements get removed. With --preserve-semantics=false, this care could be dropped. There would still be a requirement to remove a return if its return type would be incompatible with the function being injected into, but otherwise these possibly semantics-changing control flow constructs could be kept.

In OpaqueExpressionGenerator, which is used for application of identity functions to expressions, the AbstractIdentityTransformation has an "exprMustBeSideEffectFree" field. This is used to ensure preservation of semantics. Its use could be relaxed.

The AddWrappingConditionalStmts class wraps a statement S in if(true) { S }, or if (false) { } else { S }, or for(i = 0; i < 1; i++) { S }. If we don't need to preserve semantics, we could do stranger things here.

VectorizeStatements takes a lot of care to preserve semantics; this care could be dropped if --preserve-semantics=false.

This is an easy fix

This pollutes the processing/ directory. Have the server logs being zipped to an archive less often?

If a donor shader has "out" variables, these may be donated to a reference, and currently the qualifier, and associated "location" info, will not be removed. This can make the resulting variant invalid.

References can be reduced manually, but the WebUi won't use the result when showing the diff.

• Kick off reference reduction when starting a variant reduction from WebUi.
• In the diff of a reduced variant, use the reduced reference result.

We have Java code that already does much of this logic (especially after the refactor).

And try different versions of JDK

When used with glsl-fuzz, the reducer needs to be applied to a .json file representing a shader job. The shader job file, as well as identifying the other shaders, holds all the info about uniforms, etc.

For the stand-alone reducer, it seems unnatural to require users to apply it to a JSON file that just contains {}. We might like to let them apply it straight to a .frag file. But the interestingness test gets a .json file by default so we'd have to think about whether to change that.

Logging as an issue here so we do not forget about it.

We should probably not set a "default value" for --seed or we should hide the "default value" from the help message.