I ran some tests yesterday and it seemed like when a timer is rebalanced to a new node

this was found + fixed in <a class="issue-link js-issue-link" data-error-text="Failed

yes, this is being handled here <div c

Steps to reproduce: Run the dev Docker Compose file with <a hr

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Timer Rebalance Reset about skyring HOT 10 CLOSED

esatterwhite commented on June 7, 2024

Timer Rebalance Reset

from skyring.

Comments (10)

esatterwhite commented on June 7, 2024 1

this was found + fixed in #23.
released in v7.0.1

from skyring.

esatterwhite commented on June 7, 2024

yes, this is being handled here

skyring/lib/timer.js

Lines 168 to 180 in ac5362d

 const now = Date.now(); 

 const created = payload.created || now; 

 const elapsed = now - created; 

 if( now > created + payload.timeout ){ 

 debug('executing stale timer'); 

 setImmediate( 

 transport 

 , payload.callback.method 

 , payload.callback.uri 

 , payload.data 

 , id 

 , this 

 );

can you give me a more info?
skyring version?
Number of nodes?
curl command use to set the timer?

from skyring.

TheConnMan commented on June 7, 2024

Steps to reproduce:

Run the dev Docker Compose file with https://github.com/esatterwhite/skyring/blob/master/README.md#run-via-docker-compose (so 5 nodes, 3 nats)
Open https://webhook.site and use it as a callback URL
Run https://github.com/esatterwhite/skyring/blob/master/README.md#post-timer with 60000 ms and the link from above
Watch the logs to see which node gets the timer
Wait 30 sec from timer setting
docker stop <node-with-timer>
See the timer get distributed to a new node
Wait 30 seconds, nothing

However, I just ran through these steps and the timer correctly fired 60 seconds after creation, even after moving nodes. Things got weird once I repeated steps and went from 3 to 2 nodes. I remember reading you recommend having 3 nodes minimum, so I could have been seeing a Ringpop cluster breakdown.

from skyring.

esatterwhite commented on June 7, 2024

@TheConnMan Weird in what way?
I'll do some investigating

from skyring.

TheConnMan commented on June 7, 2024

The order I took nodes down this morning was 2, 3, then 5. When I took down 5 the timer got rebalanced, but never executed. Also, when I posted new timers I got two timeouts (only seen in the logs) before the timer was accepted. Would timeouts have occurred because node 2 is a seed node?

from skyring.

esatterwhite commented on June 7, 2024

Hmm ya, i was going through your steps w/ http:webhook.site and I say some "weirdness" but I'm pretty sure the webhook site was being a little wonky.

There a number of occasions where nothing happend - was getting some 504 gateway Timeout errors several minutes later. So what I suspect was the request just never finished. It took a while, but it happend. And at the same time the UI became non-responsive. It felt like it was running on a heroku Dyno that was restarting or "waking up"...

But I did find a bug there that can cause skyring to crash if there is not an error from the client connection. I'll get a fix for that up, And dig a little deeper tonight or tomorrow.

node-5_1  | 2018-02-06T14:19:15.826Z skyring:transport:http timer fail 504 <html>
node-5_1  | <head><title>504 Gateway Time-out</title></head>
node-5_1  | <body bgcolor="white">
node-5_1  | <center><h1>504 Gateway Time-out</h1></center>
node-5_1  | <hr><center>nginx/1.10.3</center>
node-5_1  | </body>
node-5_1  | </html>
node-5_1  |
node-5_1  | { Error: Gateway Timeout
node-5_1  |     at Request.request.(anonymous function) [as _callback] (/opt/skyring/lib/transports/http.js:55:18)
node-5_1  |     at Request.self.callback (/opt/skyring/node_modules/request/request.js:188:22)
node-5_1  |     at emitTwo (events.js:125:13)
node-5_1  |     at Request.emit (events.js:213:7)
node-5_1  |     at Request.<anonymous> (/opt/skyring/node_modules/request/request.js:1171:10)
node-5_1  |     at emitOne (events.js:115:13)
node-5_1  |     at Request.emit (events.js:210:7)
node-5_1  |     at IncomingMessage.<anonymous> (/opt/skyring/node_modules/request/request.js:1091:12)
node-5_1  |     at Object.onceWrapper (events.js:314:30)
node-5_1  |     at emitNone (events.js:110:20) code: 504 } '<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.10.3</center>\r\n</body>\r\n</html>\r\n'
node-5_1  | 2018-02-06T14:19:15.829Z skyring:timer timer cleared d490c02c-afaa-4b9a-aa23-dbe1328fdfb7
node-5_1  | /opt/skyring/lib/timer.js:265
node-5_1  |       , message: error.message
node-5_1  |                        ^
node-5_1  |
node-5_1  | TypeError: Cannot read property 'message' of null
node-5_1  |     at Immediate.(anonymous function) [as _onImmediate] (/opt/skyring/lib/timer.js:265:24)
node-5_1  |     at runCallback (timers.js:785:20)
node-5_1  |     at tryOnImmediate (timers.js:747:5)
node-5_1  |     at processImmediate [as _immediateCallback] (timers.js:718:5)
nats-c_1  | [1] 2018/02/06 14:19:15.892920 [DBG] 172.19.0.9:50420 - cid:8 - Client connection closed

from skyring.

esatterwhite commented on June 7, 2024

Thanks for the report!

from skyring.

TheConnMan commented on June 7, 2024

Of course, thanks for digging in!

from skyring.

esatterwhite commented on June 7, 2024

I'm going to close this one. It doesn't seem to be on skyring's side as far as I can tell.
I have plans to implement configurable retry for transports, Or maybe just make a separate transport http-retry. I'm not a fan of including it in core as it introduces the chance of duplicate execution. But I can see the use cases for it.

I'm also planing on adding building some kind of monitoring tool I added monitoring event streams a while back with that intention, but haven't gotten to it as of yet.

from skyring.

TheConnMan commented on June 7, 2024

@esatterwhite Thanks again for looking into this.

from skyring.

Timer Rebalance Reset about skyring HOT 10 CLOSED

Comments (10)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	const now = Date.now();
	const created = payload.created \|\| now;
	const elapsed = now - created;
	if( now > created + payload.timeout ){
	debug('executing stale timer');
	setImmediate(
	transport
	, payload.callback.method
	, payload.callback.uri
	, payload.data
	, id
	, this
	);