Proposal As a(n independent) follow-up to <a class="issue-link js-

Are you willing to open a PR with this recover <a class="user-mention notranslate" dat

Increase operator resiliency to unexpected scaler failures about keda HOT 5 OPEN

cyrilico commented on July 17, 2024 3

Increase operator resiliency to unexpected scaler failures

from keda.

Comments (5)

cyrilico commented on July 17, 2024 3

I'll gladly take a shot at it whenever I get some time 🙏

from keda.

zroubalik commented on July 17, 2024 1

Yeah, we should avoid panics, I agree.

from keda.

JorTurFer commented on July 17, 2024

This is an interesting point. We aim to prevent all the panics by code instead of just recovering them, but maybe we could recover panics on scaler metric requests:

keda/pkg/scaling/cache/scalers_cache.go

Lines 125 to 142 in 1e1cfb1

 func (c *ScalersCache) GetMetricsAndActivityForScaler(ctx context.Context, index int, metricName string) ([]external_metrics.ExternalMetricValue, bool, int64, error) { 

 if index < 0 || index >= len(c.Scalers) { 

 return nil, false, -1, fmt.Errorf("scaler with id %d not found. Len = %d", index, len(c.Scalers)) 

 } 

 startTime := time.Now() 

 metric, activity, err := c.Scalers[index].Scaler.GetMetricsAndActivity(ctx, metricName) 

 if err == nil { 

 return metric, activity, time.Since(startTime).Milliseconds(), nil 

 } 

 ns, err := c.refreshScaler(ctx, index) 

 if err != nil { 

 return nil, false, -1, err 

 } 

 startTime = time.Now() 

 metric, activity, err = ns.GetMetricsAndActivity(ctx, metricName) 

 return metric, activity, time.Since(startTime).Milliseconds(), err 

 }

As scalers are the place where more contributions are made, they're also the place with more unexpected problems and although I think that we should avoid panics, maybe in this case it can make sense.

In the other hand, we have a really few panics because we try to cover all the cases and we almost achieve it.
WDYT @zroubalik @dttung2905 ?

from keda.

JorTurFer commented on July 17, 2024

Are you willing to open a PR with this recover @cyrilico ?

from keda.

stale commented on July 17, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

from keda.

Recommend Projects

Increase operator resiliency to unexpected scaler failures about keda HOT 5 OPEN

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	func (c *ScalersCache) GetMetricsAndActivityForScaler(ctx context.Context, index int, metricName string) ([]external_metrics.ExternalMetricValue, bool, int64, error) {
	if index < 0 \|\| index >= len(c.Scalers) {
	return nil, false, -1, fmt.Errorf("scaler with id %d not found. Len = %d", index, len(c.Scalers))
	}
	startTime := time.Now()
	metric, activity, err := c.Scalers[index].Scaler.GetMetricsAndActivity(ctx, metricName)
	if err == nil {
	return metric, activity, time.Since(startTime).Milliseconds(), nil
	}

	ns, err := c.refreshScaler(ctx, index)
	if err != nil {
	return nil, false, -1, err
	}
	startTime = time.Now()
	metric, activity, err = ns.GetMetricsAndActivity(ctx, metricName)
	return metric, activity, time.Since(startTime).Milliseconds(), err
	}