Comments (8)
I can look into this one. Just to be sure, you have each node using a different Mongo collection?
from skyring.
Yes, each one is using a different Mongo collection
from skyring.
If this.disconnect(cb)
gets commented out the node won't exit but the timers never drop. I believe a race condition is occurring here and the node is disconnecting too soon. I'm looking more into it
from skyring.
Ya. Sounds like a race with the last write ack from mongo
from skyring.
Ya. Sounds like a race with the last write ack from mongo
from skyring.
I believe the issue is with nats not being flushed out completely. #25 resolves this bug.
Update: Bug is only present in the Kubernetes chart now so I believe it's the way I handle the load balancer. It's resolved in just docker images
from skyring.
I agree that there still is a race condition that exists with the mongo storage. I'm not sure how to find it however. I can only reproduce this bug in Kubernetes using a large amount of requests and killing the node during this chain. Sometimes I will get +1 extra timer which I assume is from timers firing as they are trying to transfer them off (more acceptable explanation). Other times I will lose a large amount of timers from killing a node with no clear explanation as to why. And even rarer I will receive the following error messages stating that a timer with an id already exists. I think the post you made in the PR about the driver could be promising and I will try to follow up on that.
2018-11-30T00:52:15.418Z skyring:error Error: Timer with id 231fcc23-3057-476a-9d17-7499e6e057dd already exists 2018-11-30T00:52:15.419Z skyring:error Error: Timer with id 226d3015-4a3b-465d-a58c-2c74de221b42 already exists
from skyring.
Timers firing during a re-balance likely, and fixable. Right now a rebalance is a pretty anonymous operation. Transports don't know a rebalance is happening & vice versa. But That can be fixed. Sounds like it needs to be.
Can you give some specifics around "large amount". I'll investigate.
You only see this problem while running on kubernetes? only with a mongo backend?
from skyring.
Related Issues (20)
- Error while trying to execute skyring single node HOT 4
- The automated release is failing 🚨
- The automated release is failing 🚨 HOT 3
- The automated release is failing 🚨 HOT 1
- Document minimum/maximum allowed `timeout` value HOT 3
- NatsError when connecting to Nats instances in Kubernetes HOT 5
- Timers getting reset on double recovery HOT 3
- Node 10 support
- Class Based transports
- migrate tcp transport to class HOT 1
- migrate zmq transport to class HOT 1
- Timers executed twice during rebalance
- Replace request package
- Replace debug with pino
- Can not start the cluster using docker compose HOT 4
- Use firebase as backend HOT 2
- how to run this in kubernetes? HOT 2
- implement semantic release
- Run tests in parallel
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from skyring.