git-sync not working with Hugo. about git-sync HOT 17 CLOSED

In my view it shouldn't be used to update applications in production (maybe in dev), but to update data used by an application and separating concerns (ssh + git) into their own container.

People can push markdown in the git repo and the site will get regenerated on our side too. It's just another way to do it. Wasn't pushing for it being the better solution, just a workaround until this gets cleared up.

from git-sync.

you should be able to inotify on the symlink itself - hugo might want to learn how to do that.

from git-sync.

As a sidenote:
We are building the page via ci from each commit and then push the static version into a new repo $repo_static. This static repo will get pulled in by git-sync and works like a charm.
Positive things: hugo only needed during build, static repo only contains the page, less moving parts in production.

from git-sync.

That's fine, but as far as I understood, that's the whole idea of a git-sync sidecar container. In this case/demo was to prove you can update an application without redeploying.

So that people can just push markdown in the git repo, and the site will get regenerated.

from git-sync.

I agree that it's not something you want to do in prod. :)

Anyway, my feeling is that we move away from the subject which is this specific issue.

I set up a watch with inotify-tools on the the synced directory. This are the logs with what inotify sees that happened on the FS after updating the git repo (I just added a word in one of the files):

# inotifywait -m -r -e modify,attrib,close_write,move,create,delete /src/git_files
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/src/git_files/site/content/ DELETE index.md
/src/git_files/site/content/post/ DELETE hugoisforlovers.md
/src/git_files/site/content/post/ DELETE creating-a-new-theme.md
/src/git_files/site/content/post/ DELETE migrate-from-jekyll.md
/src/git_files/site/content/post/ DELETE goisforlovers.md
/src/git_files/site/content/ DELETE,ISDIR post
/src/git_files/site/ DELETE,ISDIR content
/src/git_files/site/ DELETE config.toml
/src/git_files/site/static/ DELETE .gitkeep
/src/git_files/site/ DELETE,ISDIR static
/src/git_files/site/ DELETE themes
/src/git_files/ DELETE,ISDIR site
/src/git_files/ DELETE .gitignore
/src/git_files/ DELETE .git

So basically this means that what Hugo sees is accurate. I'll try to look some more into it any maybe come up with a PR.

from git-sync.

Maybe it would be a better idea to use rsync between the checked out directory and clean directory instead of symlinks ?

from git-sync.

That would introduce rsync as additional dependency. Not sure, if that's worth it. Also it does not give you an atomic switching operation. Currently the files are pulled and the symlink is used as atomic operation so inconsistent state is prevented. Rsync wouldn't have that atomic operation.

from git-sync.

Well, the symlink doesn't work for this specific case. And I expect it to be the same for other cases.

Got any suggestions for a fix, @stp-ip ?

from git-sync.

Do you mind sharing a redacted yml/json file for the specific use case? Symlinks in gerenal should be supported by hugo, when they are on the root level. gohugoio/hugo#1857

One way to "solve" symlinking issues would be to add a "GIT_SYNC_ATOMIC" false option, which disables the symlinking behaviour and instead pulls in new changes perhaps. That will lead to other issues with inconsistent state, therefore I am not seeing it as the best way to go forward.

from git-sync.

As for a specific use case, just try launching what's in the demo folder and try to update the repository to see the changes on the (newly) generated website. It won't work.

As for the logs from different places, I added them to the issue, including what the inotify-tools sees on the filesystem if I put a watch on that directory. So it's not just a specific hugo problem.

from git-sync.

My vague assumption (don't know most of the internals of inotify) is, that inotify follows the symlink and basically adds the watch on the worktree, which is removed during cleanup. Therefore the filesystem actually changes atomically, but inotify (which hugo is also using) is only seeing the DELETE as it watches the rev-* directory instead of just the symlink.

from git-sync.

@stp-ip Yes, I think you're right about the problems but nothing pops in my head on how to fix it nicely.

I did a rough implementation that adds rsync functionality on top of the symlinks here: ursuad/git-sync/tree/feature. There are a few issues with it, but it got the job done and it can be used as a starting point for implementing something better.
• If GIT_SYNC_ADD_NONATOMIC is true (it's false by default), it will add rsync functionality on top of the symlink stuff. (it was faster add something than changing existing behaviour based on conditions )
• Rsync will put it's files in GIT_SYNC_NONATOMIC_DEST, which is mandatory to define if you enable first option
• You can specify GIT_SYNC_NONATOMIC_EXCLUDE - a pattern if you don't want to sync some files/directories from git
I added the last option because rsync is running with --delete and I wanted to be able to exclude the themes directory (or other directories which might be created by other containers) . In this case the themes directory was created by the Hugo container.

Also, I agree with the idea that the rsync operation is nonatomic and it might cause some problems but just like symlinks, it will work for some. :)

Again, this is just a draft implementation.
For Hugo, it worked. You can see the logs from Hugo during the rsync operation below, so you can see the transitional files created:

Change detected, rebuilding site
2016-10-27 16:16 +0000
Source changed /src/site/content/.index.md.KKpChN
Source changed /src/site/content/post/.My-New-Post.md.kjrq26
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.My-New-Post.md.kjrq26
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.My-New-Post.md.kjrq26 ; open /src/site/content/post/.My-New-Post.md.kjrq26: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.My-New-Post.md.kjrq26
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.My-New-Post.md.kjrq26 ; open /src/site/content/post/.My-New-Post.md.kjrq26: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/My-New-Post.md
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/hive_blog.md
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/.my-personal-test.md.xS56xK
ERROR: 2016/10/27 16:16:06 site.go:583: Error reading file /src/site/content/post/.my-personal-test.md.xS56xK ; open /src/site/content/post/.my-personal-test.md.xS56xK: no such file or directory
INFO: 2016/10/27 16:16:06 site.go:1096: rereading /src/site/content/post/my-personal-test.md
INFO: 2016/10/27 16:16:06 site.go:1489: found taxonomies: map[string]string{"tag":"tags", "category":"categories"}
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/test/page/1" translated to "tags/test/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/heating-patterns/page/1" translated to "tags/heating-patterns/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/hive/page/1" translated to "tags/hive/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/new-things/page/1" translated to "categories/new-things/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/tags/personal/page/1" translated to "tags/personal/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/another-category/page/1" translated to "categories/another-category/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/categories/haething-patterns/page/1" translated to "categories/haething-patterns/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/post/page/1" translated to "post/page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
INFO: 2016/10/27 16:16:06 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
WARN: 2016/10/27 16:16:06 site.go:2373: "404.html" is rendered empty
Built site for language en:
2 of 2 drafts rendered
0 future content
0 expired content
4 pages created
0 non-page files copied
9 paginator pages created
4 tags created
3 categories created
total in 54 ms

from git-sync.

I don't know what Hugo is doing, but in this case I don't really see any other way to update files atomically. This is how git-sync works. I really don't like the idea of having two very different operating modes - this is already too tricky. Can't we just fix hugo?

from git-sync.

@thockin it's not just hugo. As said above it seems that it is the used inotify, which follows symlinks and watches the dereferenced files/dirs instead of the symlinks.

@ursuad why use rsync and not just copy? Would at least prevent the additional dependency.

Overall I am still not convinced it's the best course of action. It complicates the side car container and basically hacks around a problem, we should first dive deeper, if there is a fix possible. I will do my best to dive deeper on my side next week or during the weekend.

Thanks @ursuad for the PoC with rsync. Always appreciate such work.

from git-sync.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

from git-sync.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

from git-sync.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

from git-sync.