Comments (4)
another example is on startup if sync on start is enabled
from node-kubycat.
Kubycat has to push file synchronously to maintain the order of operations that were performed. If a delete and a create happened one after the other, you would have to ensure that they were processed in that order, making parallelization at the file event level challenging.
I think where we could make it parallel is pushing files to different pods. For each file change, we could push it to each of the pods being synced to at the same time. If your slowdown is coming from having many pods, that ought to help.
That's the next feature I'll work on implementing, though at the moment I'm a bit pressed for time so PRs would be welcome.
from node-kubycat.
The slow down is unfortunatly coming from mass updates of an entire folder
examples:
- the result of a build step, e.g. compiling the components of a website with
parcel
) - on startup sync
I guess the startup sync does not have the synchronization issues, so this could also be parallelized easily? That would already help a lot I think.
Ideally of course it would be great if kubycat could face the challenge and parallelize per file.
from node-kubycat.
Yeah, that makes sense. I think a couple of things could be done to address this. The first would be to change from individual item processing to batch processing, and scanning each batch for possible conflicts. (creates/deletes to the same files for example).
The offending items could be moved to the next batch such that all items in any given batch are safe to be processed in parallel.
I'll have to think about the issue with build folders. For folders like that which receive mass updates in short periods of times, I wonder if it would make sense to sync them less frequently and only at the entire folder level. Basically, it would wait till there hadn't been any changes for X period of time, then sync the entire folder instead of all of the individually changed files..
Sync on startup already syncs only the top-level item which in the case of folders, syncs all of the contents within too.
from node-kubycat.
Related Issues (17)
- Will this copy entire directory tree at start if not on pod? HOT 2
- Syncing multiple pods with the same settings fails HOT 1
- Linux support? HOT 4
- Problems with sync-on-start when using build-cache-on-start HOT 1
- Problems with sync-on-start when using build-cache-on-start HOT 2
- How to use not specify a namespace - like --all-namspaces HOT 1
- Support commandline arguments
- Support multiple pod-labels HOT 1
- syncronization issues HOT 5
- Both ways sync HOT 2
- filenames with spaces lead to error: source and destination are required HOT 1
- post-remote is not running on remote pods
- Support Node 20
- Support Node 20
- Support Node 20 HOT 2
- website https://www.kubycat.info/ is down
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-kubycat.