Comments (16)
Which tool do you intend to use to to the sync ? rsync
?
I tried this approach not with OSXFS but with NFS in the dinghy project, see codekitchen/dinghy#197. The main issue I got was the lack of support of delete
events in the project fsevents_to_vm
, using OSXFS mount would indeed help.
from docker-sync.
I would say, this concept is to achieve true transperency for 2 way sync for now - it yet not really change too much for rsync and, since it is docker for mac only, i would not like to switch the current strategies to it, rather add this on as an extra.
I therefor tend to use unison as the local sync. So wie have
Host --- OSXFS ---> sync-container /tmp[fswatch-trigger]--- local unison sync ---> sync-container /app/code[fswatch-trigger] ----- named volume mount -> app container
from docker-sync.
After @mickaelperrin implemented unison-dualside we talked to let it "walk" for a while and then try this concept.
A correction, OSXFS shall be replaced by "any native sync" and the limitation on docker-for-mac is also invalid, iw would work with all of those
from docker-sync.
In fact, this concept only works with filemounts in docker that supports file events forwarding from the host to the mounted folder in the container. OSXFS supports this, but others solutions like Boot2docker don't, even dinghy
has only a partial support with fsevents_to_vm
(missing delete events).
from docker-sync.
I switched this week-end from dinghy
to Docker for mac
.
Despite the hard work done on unison:dualside
I tested a different approach as you spotted here and as I did previously with lsyncd.
I tested several approaches (fswatch / unison, unison only...). Currently, the setup that performs the better is also the simpler with a pure unison
implementation, without fswatch
. So far, this is the only solution, that supports (and with very little CPU usage) a full Drupal bootstrap from composer in real-time. However, I encountered some missing delete
events from the host to the container. I will investigate this further.
I released my experiments in a separate project docker magic sync
that relies on a simple additional container.
Feel free to give your impressions and how it could be used with/in docker-sync
.
This additional work with fswatch and unison make me understand how the implementation of unison:dualside
could be simplified a lot, also.
from docker-sync.
Well what shall i say, thats the concept outlined above, so there is nothing to comment more from me. As stated on the project page, i asked people to not re-do projects again and again, i guess, thats what we have this now.
The auto-discovery of mounts is a point is a clever thing, but probably can be done differently.
To use this in docker-sync i would need to understand:
- what do you start on the host? ( nothing i guess, due to osxfs usage )
- you do start one container and map all syncs in there. Does this work for several sync endpoints?
It is pretty easy to use with docker-sync:
- create a new strategy, lets call it magic.
- start your container there
- let the people configure compose the way you outlined
- as watcher strategy set 'disabled'
And thats it.
What is your plan in general, you want to keep docker-magic-sync as a new project or do you want to incorporate the changes / concept into docker-sync?
from docker-sync.
To be clear, I think some people will be interested from this "only docker needed" approach, so it could be maintained as a project.
But I would like docker-sync
to help it to be even more 'magic' by reducing even more the configuration setup and making it simpler to use for docker newcomers. So yes, I would like to see it implemented in docker-sync
as long as it is more simpler to use / configure than what I have done so far.
As you may have noticed, I don't use external volumes. This is something I found useful as there is no additionnal step to clean up docker when removing / restarting a container. However, currently I don't see how to implement this with docker-sync
. Maybe docker-sync-stack
could inject the container in the compose file dynamically ? Should it be done by hacking the compose
file or is there a better way with docker
commands ?
To answer your questions:
- yes, there is nothing to be run on the host
- yes it works with any sync endpoints as soon as they are not mapped to the same path even if they are in different containers.
Before starting a ruby implementation, I will test it for a few days to see if the core concepts are working very well or at least with limited drawbacks.
from docker-sync.
Simplifying docker-sync configuration: full opt in. Not only for magic, in general. But you know what that means, we will try to guess / automate assumptions leading to new issues when we are "wrong". If you also opt in helping with those, lets do it. Because auto-configration is mostly effort for the docker-sync developers and effort in maintainance.
Great, ok so battle plan:
- With "magic" we do not need external volumes by design, with the default strategys we need those. So we need to make those optional or strategy aware to so this is not needed. This should be already the case - external volumes are not used by docker-sync in general, its just a docker-compose thing and we document it. If we work out magic to be the new default, we swap the examples to be simple ( magic ) and add documentation for the others.
- docker sync stack:
docker-compose can ready the compose file into yml, and the write it ( talked with the developer about the write, should be easy. So we could read the yml file, modify it and then add it.
But i do not really see a reason for that - we could just have this .developer.yml coping with that "non automatically" or at least start with that, add more automation later
So working together with docker-compose.rb we can get this done "automatically" but i would suggest to start with .development.yml first, then add automatiation.
Does that sound reasonable?
from docker-sync.
@mickaelperrin probably join https://gitter.im/EugenMayer/docker-sync?utm_source=share-link&utm_medium=link&utm_campaign=share-link to further discuss it
from docker-sync.
Mmm, bad news here. It looks like there is an issue with OSXFS. As soon as the files increases, inotify
events are no more triggered in the container.
Some other users reports a similar issue on Docker forums. I will investigate this further and try to do a proper issue on Docker forums.
:(
from docker-sync.
Shall we delay this implementation until we know if it is "fixable" or shall we go forward with this one? Architectural / Conceptual wise things are set, i do not expect any big hurdles
from docker-sync.
On my side I won't work on the implementation of this in Docker-sync
until the core
concept works.
Currently, I will see if the knowledge gained while working on this could benefit to the dualside
implementation.
from docker-sync.
Some updates here.
I created the issue on the Docker forums, hope we will have some update soon.
I did also some new tests and what I can say it that as exepected the sync with magic
is far slower and heavy in CPU than with unox
. I think this is not a bid deal when doing simple file editing but when bootstrapping a big project you can feel the difference.
from docker-sync.
Just did some tests with the latest release of Docker with magic-sync.
Things got improved, basic sync is performed.
But the composer install
test (run on the host and on the container) doesn't perform well, only a few part of files got synced.
from docker-sync.
ending this discussion for now, should i? I would stall this ticket and remove it if we move on, or do you have other ideas @mickaelperrin
from docker-sync.
You can close it I think this strategy will work when OSXFS will improve performance and so... When we won't need syncing :)
from docker-sync.
Related Issues (20)
- Feature Requestion: add update command HOT 1
- Default strategy for new Virtualization framework for Mac HOT 2
- Looking for changes - huge node_modules HOT 1
- File deleted on host still exists in app container HOT 11
- External network access and permission problems
- Enforcing one-way sync HOT 3
- `docker-sync -n` should create dummy volumes for non-started syncs
- Cannot start docker-sync with macOS Monterey - Apple M1 Chip HOT 2
- Bug: Wrong Unison image is pulled on M1 Mac HOT 4
- Docker sync daemon is crashing HOT 5
- `initialize': uninitialized constant DockerSync::DockerComposeSession HOT 4
- docker-sync start success, but I can't use docker-sync logs HOT 3
- Fatal error with Unison because of wrong default exclude definitions HOT 3
- Error in RSync strategy using the `sync_excludes` setting: undefined local variable or method `excludes_list' HOT 1
- Sync issue on MacOS 13.0 HOT 26
- `expand_ignore_strings`: undefined method `append` for Array HOT 3
- What's the "extra care" for using unison on OSX with sleep/hibernate. HOT 3
- Add support for passing --env-file to docker-sync start (using ENV file with/from different name or location)
- docker-sync start failed - /tmp/supervisor.sock no such file
- /docker-sync-1.0.5/tasks/sync/sync.thor:69:in `kill': Operation not permitted (Errno::EPERM) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-sync.