Currently, ResourcePolicy is supposed not to be changed.

For VPA, we don't need to change anything.

If HPA is created by tortoise, we need to update HPA.
If HPA is given from users, we don't need to do anything with HPA, but users need to update their HPA to have the metric for a new container.

https://github.com/mercari/tortoise/blob/main/docs/horizontal.md#type-external-support-will-be-removed

use retry.RetryOnConflict

Tortoise has the default values on several fields.

After the emergency mode, we'd like to prevent to scale down too rapidly. Instead, it's better to scale down gradually, and Tortoise has got TortoiseBackToNormal phase during that period.

I think by default VPA should patch the deployment.spec for requests instead of pod.spec

This ensures that any other pod from the replicaset can also handle if uneven spike happens on it. Also, replicasets are recreated, so these new VPA recommended values should persist across pod creation

This is highly depended on controller type though, for eg. daemonsets and statefulsets should not apply the same changes across all pods

For the safety.

so that we can test the tortoise in the real cluster.

Currently, the tortoise doesn't say anything to users. It only logs something and it's very hard to know what's happening via tortoises.

When creating new tortoises, it should be either Auto or Off. Emergency mode in newly created tortoise cannot be handled by the controller.

Currently, the huge functions mostly have enough UTs.
But, we don't have the integration tests from the controller package.

This issue means the integration test (not e2e test), we don't need to run up the clusters (kind, minikube etc) as they're too mendokusai to wait for them to start. Let's just use envtest.
https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest

• [single container] test for the reconcile with the tortoise TortoisePhaseWorking #22
• [single container] test for the reconcile with the tortoise TortoisePhaseEmergency #57
• [multi container] test for the reconcile with the tortoise TortoisePhaseWorking #70
• [multi container] test for the reconcile with the tortoise TortoisePhaseEmergency #70
• test for the webhook

• proposed_hpa_utilization_target (Histogram): HPA utilization target that tortoise recommends
  • label tortoise_name: tortoise name
  • label namespace: tortoise namespace
  • label index: the metric index. (0 index-ed)
  • label hpa_name: hpa name
• proposed_hpa_minreplica (Histogram): HPA minReplica that tortoise recommends
  • label tortoise_name
  • label namespace
  • label hpa_name: hpa name
• proposed_hpa_maxreplica (Histogram): HPA maxReplica that tortoise recommends
  • label tortoise_name
  • label namespace
  • label hpa_name: hpa name
• proposed_cpu_request (Histogram): CPU request that tortoise recommends
  • label tortoise_name
  • label namespace
  • label container_name
• proposed_memory_request (Histogram): Memory request that tortoise recommends
  • label tortoise_name
  • label namespace
  • label container_name

Tortoise can work on the scaling of Pods under deployments.
https://github.com/mercari/tortoise/blob/main/api/v1alpha1/tortoise_types.go#L121

But, we want to use it for Pods managed by other resources as well, which implements /scale subresource

We need to push the image somewhere so that we can pull the image in the cluster
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry

Currently, tortoise needs to have at least one Horizontal.
But, people may want to have tortoise with only Vertical for both CPU/memory.

Let's say

• the Pod has two containers (app and istio-proxy)
• the both container's CPU scale via HPA. (both targets 75%)
• the app container usually uses around 75%, whereas the istio-proxy container usually uses around 50%.
  • It could happen because the app container always kicks HPA to scale out while the istio-proxy doesn't need to scale up.

In this case, some CPU given to istio-proxy is always wasted. And, we should change the CPU request of istio-proxy to be smaller or the CPU request of the app container to be bigger.

This issue is to automate such optimization in the tortoise controller

Tortoise generates the recommendations from past data.
And, at least, users need to run Tortoise as dry-run so that the Tortoise can generate the accurate recommendation.

During such period, the controller marks the tartoise as GatheringData phase. And, it'll mark it as Working once enough time passes.

• tortoise_count: (Gauge) – the current number of controlled tortoise object
  • label update_mode with values: off, emergency, auto
  • label phase with values: initializing, gathering_data, working, emergency, back_to_normal
• reconciliations_total: (Counter) -- the number of reconciliations split by result.
  • label error with values: none, internal?
  • EDIT: controller_runtime_reconcile_total created by kubebuilder already does this.
• etc

We need to fix the recommendation recorded in tortoise.

Currently, all of the places are considering that the deployment has multiple containers and thus HPA has external metrics or type: ContainerResource metrics.
But, users may want to use tortoises with single container deployment and HPA with type: Resource metrics

Add .status.condition.tortoiseConditions so that people can easily understand why the current recommendation is generated.

create the config file to configure tortoise controller, instead of flags.
https://github.com/mercari/tortoise/blob/main/docs/flag-configuration.md

For now, it's enough with flags, but it should be the file given we want to include something complicated in the future.

Currently, tortoise controller checkes the HPA and VPA periodically and modify based on the calculated recommendations.
But, people may apply something to those HPA and VPA by themselves.

Here, let's create the mutating webhook against HPA and VPA object so that the recommendations are always applied to them.

Tortoise can work for HPA which is created by users.
And we need to give "tortoises.autoscaling.mercari.com/tortoise-name" annotation on it as well.

https://zenn.dev/zoetro/books/testing-kubernetes-operator/viewer/tips#%E3%83%86%E3%82%B9%E3%83%88%E5%A4%B1%E6%95%97%E6%99%82%E3%81%AE%E3%83%87%E3%83%90%E3%83%83%E3%82%B0%E6%89%8B%E6%B3%95

If emergency mode is enabled, we only increase the replica number, but we may want to give additional buffer on resource under Vertical autoscaling policy.

Tortoise can work for HPA which is created by users. We need to make sure an attached HPA is valid or not.

https://github.com/mercari/tortoise/blob/main/main.go#L81-L105

The totorial guide for people to understand how tortoise looks like quickly

Spot Nodes are very cheap but they have high risk of getting evicted at very short notice.

In order to use Spot VMs efficiently we can always have much higher replica count compared to running on OnDemand Nodes, this will solve 2 problems -

• Spread pods on more nodes so less risk of eviction of 100% of replicas at same time
• The eviction period is quite short 120seconds at best, but new pods take long time to become ready (sometimes more than 5minutes), so existing replicas should be able to handle traffic even when lot of replicas are down. Meaning CPU utilization per pod will be lower than that of OnDemand Nodes

So suggestion is to run 25% (or even 50%) more replicas and ideal scenario if Deployment is running completely on Spot Nodes

• tortoise should have all container's policy
• tortoise shouldn't have the container's policy which doesn't actually exist on deployment

If the current HPA target resource utilization is 50% and tortoise suggests 80%, it might be dangerous to suddenly change like that. So, we'd like to change it gradually like 50 → 55 → 60 ... 80%.

Sometimes we can predict the increase of the resource consumption before it actually happens. (like TV, push notification on app, etc. Or load testing in a upstream service in dev.)
This feature allows people to schedule scaling up before it actually happens.
They will configure it with "when scaling up" and "how long scaling up" so that it can be back to normal afterward.

Maybe this feature will eventually replace resourcePolicy in Tortoise. (plus, Emergency mode as well?)

There are many places which has the comment of // TODO: make them configurable via flag.
As commented, we'd like to make them confiugrable as flags.

So far, @sanposhiho is the only person who knows how the tortoise is implemented.
Here, I'd like to add the documentation for the developers to know the internal detail.

https://github.com/mercari/tortoise/blob/main/docs/vertical.md

We have FeagureGates field in tortoise spec.
https://github.com/mercari/tortoise/blob/main/api/v1alpha1/tortoise_types.go#L59

It's intended to be used to control which tortoise uses which alpha feature.
This project is now very early stage, and we want to confirm each feature step by step via this field. Let's implement FeatureGates for each feature.

Tortoise needs to work with HPA and VPAs.
Both of them should be appropriately created before Tortoise starts to work.

Let's implement Initializing phase and initialize HPA and VPAs during that phase.

https://github.com/mercari/tortoise/pull/57/files#r1162381665

• delete VPAs managed by tortoises when a tortoise gets deleted.
  • we can use owner references, or delete them in the reconcilation.
• delete HPA managed by tortoises if HPA is created by the controller.
  • If .spec.targetRefs.HorizontalPodAutoscalerName is not nil, we shouldn't delete HPA as that's created by users.

enable the GitHub Actions to run lint and test in this repo

Currently, metrics have labels of containerName. But, it's difficult to identify the owner of those containers. Also, multiple deployments may have container with the same name, and they are mixed up in the same containerName label.

When #160, we want to change the phase to GatheringData. But we still want other resource to get scaled.

The current domain is sanposhiho.com. It should be renamed to another domain related to mercari, not me.

https://github.com/mercari/tortoise/blob/main/PROJECT#L11

For the safety. We may again change it back to DeleteAll once we're confident about tortoise's impl.