feat: multi-thread rendering about jekyll HOT 10 OPEN

So it turns out it is Liquid::Template#render that is not thread safe. That's why the rendered results are still incorrect even if each renderer uses a different payload. I think I may need to submit an issue on Shopify/liquid (done now: Shopify/liquid#1754) to ask about whether there is a way to let Liquid render in a thread-safe way.

For now, a workaround can be disabling the caching in Jekyll::LiquidRenderer::File#parse here:

I now basically can let Jekyll render in a multi-thread way. See the plugin on my blog. It seems like a good-enough workaround.

Even though workaround is possible, this issue should remain open. I think multi-thread rendering should be a basic feature that users can enable without using a plugin.

from jekyll.

Any pointers on how to make it work? Did you have any similar issue while developing it?

Please take any issues with a particular plugin to that plugin's repo. Thanks! ❤

from jekyll.

Please take any issues with a particular plugin to that plugin's repo. Thanks! ❤

Yes, I do not want to deviate the discussion in this issue either, but that plugin does not have its own repo. I created UlyssesZh/UlyssesZh.github.io#95 for discussions specifically for that plugin. @rouralberto Please continue there.

For other people watching this issue, if you want to talk about the workaround plugin I wrote and linked, please go to UlyssesZh/UlyssesZh.github.io#95.

from jekyll.

We can possibly use a front-matter entry to indicate that.

Maybe priorities, like hook plugins have?

from jekyll.

However, because the current implementation of rendering in Jekyll is not thread-safe, the rendered pages are wrong (one page has the content of another page). These could be fixed after some refactoring.

This is because the site payload used on Site#render_regenerated is shared, and Renderer#assign_pages! and others modify it. I got better results by duping the payload on your plugin, but it's probably not the best way to do it.

from jekyll.

We can possibly use a front-matter entry to indicate that.

Maybe priorities, like hook plugins have?

I am thinking about something called rendering_priority (not just priority because maybe we will have other features that need specifying priorities in the future):

---
rendering_priority: low
---

The rendering threads are grouped according to the priorities. Threads with lower priority are newed after the group with higher priority finished each &:joining.

I got better results by duping the payload on your plugin, but it's probably not the best way to do it.

They are better, but docs can still have contents of other docs in the same collection. I do not know whether there is a simple way to resolve this.

from jekyll.

Thanks! I'll try to run it over some big sites we have soon :)

from jekyll.

So it turns out it is Liquid::Template#render that is not thread safe. That's why the rendered results are still incorrect even if each renderer uses a different payload. I think I may need to submit an issue on Shopify/liquid (done now: Shopify/liquid#1754) to ask about whether there is a way to let Liquid render in a thread-safe way.

For now, a workaround can be disabling the caching in Jekyll::LiquidRenderer::File#parse here:

jekyll/lib/jekyll/liquid_renderer/file.rb

Lines 11 to 18 in 3f3a283

	def parse(content)
	measure_time do
	@renderer.cache[@filename] ||= Liquid::Template.parse(content, :line_numbers => true)
	end
	@template = @renderer.cache[@filename]
	self
	end

I now basically can let Jekyll render in a multi-thread way. See the plugin on my blog. It seems like a good-enough workaround.

Even though workaround is possible, this issue should remain open. I think multi-thread rendering should be a basic feature that users can enable without using a plugin.

I tried to use your plugin in my site but it gets stuck in Generating....

--

I'm using Jekyll 4.3.2. Any pointers on how to make it work? Did you have any similar issue while developing it?

from jekyll.

@mattr- any plans for this? maybe #9156 ?

from jekyll.

Maybe. I'm not super focused on Jekyll 5 at the moment, but rather a 4.4 release. Since this issue is already linked there (along with a comment on that issue also requesting multi-core support), I think that will cover everything.

from jekyll.