Comments (6)
@fezzzza it sounds like everything is working with watchman
? I'm tempted to rename this issue to "delete the non-watchman code and only support watchman"; and then also close #21 and #22 as dups of "also quirks from the java watcher impl". Does that sound good?
from mirror.
Pre-emptively closing with the plan to "fix" via #25.
from mirror.
mtime
is what is used for comparison, and if mtime
is different, the latest mtime
wins, and then the opposite side has the file written + mtime
set to match, exactly as you're reasoning about:
https://github.com/stephenh/mirror/blob/master/src/main/java/mirror/SaveToLocal.java#L103
https://github.com/stephenh/mirror/blob/master/src/main/java/mirror/NativeFileAccessUtils.java#L21
Modify: 2018-10-27 21:38:38.372526215 +0100
Modify: 2018-10-27 21:38:38.372000000 +0100
I did have a bug where Java (or maybe even POSIX I forget) could only set mtime
with a resolution of milliseconds or what not. So the two sides would always look out of sync, because the "set mtime to X" was always nanoseconds/whatever off.
Here is the commit that fixed that:
I wonder if you're seeing some variation of that.
mirror client
has a --debug-prefixes
option that if you include one of the file paths that is doing this, it should output some hopefully useful information about why it was decided it needs synced.
from mirror.
Here's the relevant portion of the client log just for that file:
2018-10-29 22:46:37 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
2018-10-29 22:46:37 INFO edge/i/flags/Finland.png isRemoteNewer
2018-10-29 22:46:37 INFO l: null
2018-10-29 22:46:37 INFO r: modTime: 1540672718372 data: "\357\277\275PNG\r\n\032\n\000\000\000\rIHDR\000\000\000\024\000\000\000\f\b\002\000\000\000\357\277\275n\n\357\277\275\000\000\000\tpHYs\000\000-\357\277\275\000\000..."
2018-10-29 22:46:37 INFO Remote update edge/.../Finland.png
and on the server:
2018-10-29 22:46:37 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
2018-10-29 22:46:37 INFO edge/i/flags/Finland.png isLocalNewer
2018-10-29 22:46:37 INFO l: modTime: 1540672718372 local: true
2018-10-29 22:46:37 INFO r: null
...
2018-10-29 22:46:37 INFO Sending edge/.../Finland.png
then I installed watchman on the client and updated the client to v1.2.1 and:
2018-10-29 23:13:26 INFO Queueing: path: "edge/i/flags/Finland.png" modTime: 1540672718372 local: true
--
2018-10-29 23:13:29 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
--
2018-10-29 23:13:51 INFO Queueing: path: "edge/i/flags/Finland.png" modTime: 1540672718372 local: true
--
2018-10-29 23:13:53 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
--
2018-10-29 23:14:01 INFO edge/i/flags/Finland.png isLocalNewer
2018-10-29 23:14:01 INFO l: modTime: 1540672718372 local: true
2018-10-29 23:14:01 INFO r: null
--
2018-10-29 23:14:28 INFO Sending edge/.../Finland.png
--
still the file is transferred
then installing watchman and updating the executable on the server to v1.2.1:
server side with 1.2.1 + watchman:
--
2018-10-30 00:51:41 INFO Queueing: path: "edge/i/flags/Finland.png" modTime: 1540672718372 local: true
--
2018-10-30 00:51:43 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
client side:
--
2018-10-30 00:51:11 INFO Queueing: path: "edge/i/flags/Finland.png" modTime: 1540672718372 local: true
--
2018-10-30 00:51:13 INFO edge/i/flags/Finland.png gitIgnored=false, extraIncluded=false, extraExcluded=false
--
So now nothing seems to be transferred.
I think that's a pretty compelling argument for installing watchman!
from mirror.
Oddly, one file bounced quickly back-and-forth between the servers when I changed the file contents and saved it:
client side:
2018-10-30 01:24:58 INFO Client has 2314 paths
2018-10-30 01:25:15 INFO Server has 2315 paths
2018-10-30 01:25:15 INFO Tree populated
2018-10-30 01:26:33 INFO Queueing: path: "dev/panic_fw.php" modTime: 1540862793272 local: true
2018-10-30 01:26:33 INFO dev/panic_framework.php isLocalNewer
2018-10-30 01:26:33 INFO l: modTime: 1540862793272 local: true
2018-10-30 01:26:33 INFO r: modTime: 1540862407222 data: "initialSyncMarker" local: true
2018-10-30 01:26:33 INFO Sending dev/panic_fw.php
2018-10-30 01:26:36 INFO dev/panic_framework.php isRemoteNewer
2018-10-30 01:26:36 INFO l: modTime: 1540862793272 local: true
2018-10-30 01:26:36 INFO r: modTime: 1540862795427 data: "<?php\n/**\n ... * \n..."
2018-10-30 01:26:36 INFO Remote update dev/panic_fw.php
2018-10-30 01:26:36 INFO Queueing: path: "dev/panic_fw.php" modTime: 1540862795427 local: true
server side:
2018-10-30 01:25:14 INFO Server has 2315 paths
2018-10-30 01:25:14 INFO Client has 2314 paths
2018-10-30 01:25:14 INFO Tree populated
2018-10-30 01:25:15 INFO Sending dev/scripts
2018-10-30 01:26:34 INFO Remote update dev/panic_fw.php
2018-10-30 01:26:36 INFO Sending dev/panic_fw.php
2018-10-30 01:26:38 INFO Remote update dev/panic_fw.php
...though this has only happened once and it has not recurred during my very short test. Much seems to be working as it should.
from mirror.
2018-10-29 23:14:01 INFO r: null
Those r: null
s insinuates the client (l
) thought the server (r
) didn't have the file, so that is why it sent it...
Maybe this ties in with your other bug where, for some reason the Java/non-watchman codepath is just blithely ignoring/not finding files, so then they either don't sync or resync or what not.
Sounds like I should just hard-delete the Java watcher. I've only used the watchman-based codepaths for ~years now so really don't know how well the Java-only one does/does not work...
from mirror.
Related Issues (20)
- Do not respect .gitignore content HOT 7
- "OutOfMemoryError: Java heap space" error while running mirror HOT 4
- Support very large files HOT 1
- Deleting a directory hierarchy leaves empty directory
- ignoring directories with many files HOT 2
- About package name HOT 3
- Can we use the diff port instead of '49172'? HOT 5
- How to keep the file permissions? HOT 1
- CentOS docker image HOT 2
- Question: What causes lag between watchman and mirror sending a file? HOT 4
- java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.facebook.watchman.WatchmanException: unknown command watch-project HOT 1
- logs are not rolling. HOT 2
- exclude file pattern issue HOT 3
- Deadline exceeded after 3 minutes HOT 8
- Issue with bulk file uploads HOT 1
- restricting server to only one interface HOT 7
- Output per-file status info to systemd/syslog HOT 1
- Specify log file path? HOT 3
- Mirroring directories on same server (Docker) HOT 1
- How do I debug mirror? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mirror.