Haskell files being touched causes unecessary recompilation about proto-lens HOT 2 CLOSED

Hi Niklas,

It appears that your second option is really the same as the first one just implemented further upstream. I suspect protoc maintainers will be hesitant to make such a change as their primary use case is building into a clean tree (that's the way bazel works). So, realistically doing an external workaround is the easier way to go about it.

For my edification, which scenario do you have in mind, ghc --make? Or when running it from stack/cabal? In my dream world we would be pushing fixes where they belong: the tools that rely on mtime shouldn't really do that.

from proto-lens.

Hi Greg,

So, realistically doing an external workaround is the easier way to go about it.

OK, I guessed right about that then. I have implemented the first approach in PR #177.

For my edification, which scenario do you have in mind, ghc --make? Or when running it from stack/cabal?

Both. Since cabal-the-library uses ghc --make, and both cabal-install and stack use cabal-the-library, in all possible scenarios does ghc --make get invoked. As a result, in my project where I use proto-lens-protoc, the .hs modules that are generated by a proto are touched on every build, and thus recompiled on every incremental build (that is, if you change a bit of code and run e.g. stack build again).

This problem gets intensified by "the [TH] check problem", as I call it: Right now in GHC, if a module A gets changed (in this case our proto-generated module), and a module B improts it and uses TemplateHaskell splices, then B needs to get recompiled (with [TH] shown as the reason in ghc --make), because values in A might have changed that might get reified in TH splices in B. Further, if a module C imports B and uses TH, then C also gets recompiled for the same reason, and so on. In projects where TH is used a lot (e.g. for logging in every module), this means a single file change (even due to touch) can result in the recompilation of large parts of the hole projects. A mini example of that can be seen here.

(I'm working on making this problem less severe but it will take a lot of effort to be fully fixed, and until then not touching files unnecessarily is a very effective cure.)

In my dream world we would be pushing fixes where they belong: the tools that rely on mtime shouldn't really do that.

I agree in part: Yes, any build system should check for content changes instead of relying only on mtime changes, and GHC should be improved to do so.

However, an mtime check before a contents check (as shake does) is a very effective optimisation: In the good (file unchanged) case, it reduces the cost, from reading the file to a single stat(), making no-op/incremental builds O(number of files in the project) vs O(size of source code in the project), and touching cached FS meta information only.

As a result, I think that wherever possible, file generators should not update the mtime of files when they can, and that pushing this decision to happen as early as possible is the most effective way to get fast incremental builds.

Let me know if this makes sense!

from proto-lens.