Concurrency safe `FileImporter` about go-jsonnet HOT 6 OPEN

@sbarzowski would know better but I think we didn't make it thread safe at all. It might be possible to put locks throughout the VM. I think our reasoning was that the language itself was single threaded so there was no point allowing a VM to process two executions at once. What if you wrapped the whole thing in a single global lock, or just serialised your interactions with it?

from go-jsonnet.

That would mean we undo any gain we get from concurrently generating, which is very significant.

from go-jsonnet.

If the gains you got were from cacheing then you'd still get those gains on account of the cache being maintained between runs

from go-jsonnet.

That's separate, we both run generation per "stack" concurrently, and when we did that we saw an additional improvement when using the same runtime. We could write our own importer, but if the rest of the runtime is not concurrency safe then it's of little use.

from go-jsonnet.

I see.

Any chance you can look at go race detector to see if it's only a small number of places that need protecting?

Another option might be to create n VMs but have them all accessing the same import cache, which is a smaller unit of code to protect.

from go-jsonnet.

I can't believe I haven't thought of that, agreed, go race detector should be able to catch this with sufficient amounts of iterations. Will try!

from go-jsonnet.