Git Product home page Git Product logo

common-instancetypes's Introduction

kubevirt/common-instancetypes

A set of instance types and preferences to help create KubeVirt VirtualMachines.

Overview

Structure

The available instance types can be divided into two categories:

  1. Workload agnostic - or general purpose
  2. Workload specific

Workload agnostic instance types are a good starting point to run your workload. Once you know more about the requirements of your workload, you can start choosing a workload specific instance type.

The following diagram summarizes the available instance types and their use-cases:

graph TD

classDef grp fill:white,stroke:lightgray,color:gray
classDef series fill:lightyellow,stroke:lightgray
classDef instancetype fill:

wrklda(Workload agnostic)
wrklds(Workload specific)
class wrklda grp

wrklda:::grp --> Universal:::series
Universal([Universal]):::series --> u1:::instancetype

wrklda:::grp --> Overcommitted:::series
Overcommitted([Overcommitted]):::series --> o1:::instancetype

wrklds:::grp --> Computeexclusive:::series
Computeexclusive([Compute Exclusive]):::series --> cx1:::instancetype

wrklds:::grp --> GPUNVIDIA:::series
GPUNVIDIA([GPU NVIDIA]):::series --> gn1:::instancetype

wrklds:::grp --> Memory:::series
Memory([Memory]):::series --> m1:::instancetype

wrklds:::grp --> Network:::series
Network([Network]):::series --> n1:::instancetype

Schema

Click in order to view the instance type names schema
instanceTypeName = seriesName , "." , size;

seriesName = ( class | vendorClass ) , version;

class = "u" | "o" | "cx" | "g" | "m" | "n";
vendorClass = "g" , vendorHint;
vendorHint = "n" | "i" | "a";
version = "1";

size = "small" | "medium" | "large" | [( "2" | "4" | "8" )] , "xlarge";

Series

. U O CX GN M N
Has GPUs
Hugepages
Overcommitted Memory
Dedicated CPU
Burstable CPU performance
Isolated emulator threads
vNUMA
vCPU-To-Memory Ratio 1:4 1:4 1:2 1:4 1:8 1:2

U Series

The U Series is quite neutral and provides resources for general purpose applications.

U is the abbreviation for "Universal", hinting at the universal attitude towards workloads.

VMs of instance types will share physical CPU cores on a time-slice basis with other VMs.

U Series Characteristics

Specific characteristics of this series are:

  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

O Series

The O Series is based on the U Series, with the only difference being that memory is overcommitted.

O is the abbreviation for "Overcommitted".

UO Series Characteristics

Specific characteristics of this series are:

  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • Overcommitted Memory - Memory is over-committed in order to achieve a higher workload density.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

CX Series

The CX Series provides exclusive compute resources for compute intensive applications.

CX is the abbreviation of "Compute Exclusive".

The exclusive resources are given to the compute threads of the VM. In order to ensure this, some additional cores (depending on the number of disks and NICs) will be requested to offload the IO threading from cores dedicated to the workload. In addition, in this series, the NUMA topology of the used cores is provided to the VM.

CX Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
  • Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
  • vNUMA - Physical NUMA topology is reflected in the guest in order to optimize guest sided cache utilization.
  • vCPU-To-Memory Ratio (1:2) - A vCPU-to-Memory ratio of 1:2.

GN Series

The GN Series provides instances types intended for VMs with NVIDIA GPU resources attached.

GN is the abbreviation of "GPU NVIDIA".

This series is intended to be used with VMs consuming GPUs provided by the NVIDIA GPU Operator which can be installed on Kubernetes and also is made available on OpenShift via OperatorHub.

GN Series Characteristics

Specific characteristics of this series are:

  • Has GPUs - Has GPUs predefined.
  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • vCPU-To-Memory Ratio (1:4) - A vCPU-to-Memory ratio of 1:4, for less noise per node.

M Series

The M Series provides resources for memory intensive applications.

M is the abbreviation of "Memory".

M Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Burstable CPU performance - The workload has a baseline compute performance but is permitted to burst beyond this baseline, if excess compute resources are available.
  • vCPU-To-Memory Ratio (1:8) - A vCPU-to-Memory ratio of 1:8, for much less noise per node.

N Series

The N Series provides resources for network intensive DPDK applications, like VNFs.

N is the abbreviation for "Network".

This series of instancetypes requires nodes capable of running DPDK workloads and being marked with the respective node-role.kubevirt.io/worker-dpdk label as such.

N Series Characteristics

Specific characteristics of this series are:

  • Hugepages - Hugepages are used in order to improve memory performance.
  • Dedicated CPU - Physical cores are exclusively assigned to every vCPU in order to provide fixed and high compute guarantees to the workload.
  • Isolated emulator threads - Hypervisor emulator threads are isolated from the vCPUs in order to reduce emaulation related impact on the workload.
  • vCPU-To-Memory Ratio (1:2) - A vCPU-to-Memory ratio of 1:2.

Development

To get started with customizing or creating your own instancetypes and preferences see DEVELOPMENT.md.

Resources

The following instancetype resources (cluster-wide and namespaced) are provided by this project:

Name vCPUs Memory
cx1.2xlarge 8 16Gi
cx1.4xlarge 16 32Gi
cx1.8xlarge 32 64Gi
cx1.large 2 4Gi
cx1.medium 1 2Gi
cx1.xlarge 4 8Gi
gn1.2xlarge 8 32Gi
gn1.4xlarge 16 64Gi
gn1.8xlarge 32 128Gi
gn1.xlarge 4 16Gi
m1.2xlarge 8 64Gi
m1.4xlarge 16 128Gi
m1.8xlarge 32 256Gi
m1.large 2 16Gi
m1.xlarge 4 32Gi
n1.2xlarge 16 32Gi
n1.4xlarge 32 64Gi
n1.8xlarge 64 128Gi
n1.large 4 8Gi
n1.medium 4 4Gi
n1.xlarge 8 16Gi
o1.2xlarge 8 32Gi
o1.4xlarge 16 64Gi
o1.8xlarge 32 128Gi
o1.large 2 8Gi
o1.medium 1 4Gi
o1.micro 1 1Gi
o1.nano 1 512Mi
o1.small 1 2Gi
o1.xlarge 4 16Gi
u1.2xlarge 8 32Gi
u1.4xlarge 16 64Gi
u1.8xlarge 32 128Gi
u1.large 2 8Gi
u1.medium 1 4Gi
u1.micro 1 1Gi
u1.nano 1 512Mi
u1.small 1 2Gi
u1.xlarge 4 16Gi

The following preference resources (cluster-wide and namespaced) are provided by this project:

Name Guest OS
alpine Alpine
centos.7 CentOS 7
centos.7.desktop CentOS 7
centos.stream8 CentOS Stream 8
centos.stream8.desktop CentOS Stream 8
centos.stream8.dpdk CentOS Stream 8
centos.stream9 CentOS Stream 9
centos.stream9.desktop CentOS Stream 9
centos.stream9.dpdk CentOS Stream 9
cirros Cirros
fedora Fedora
rhel.7 Red Hat Enterprise Linux 7
rhel.7.desktop Red Hat Enterprise Linux 7
rhel.8 Red Hat Enterprise Linux 8
rhel.8.desktop Red Hat Enterprise Linux 8
rhel.8.dpdk Red Hat Enterprise Linux 8
rhel.9 Red Hat Enterprise Linux 9
rhel.9.desktop Red Hat Enterprise Linux 9
rhel.9.dpdk Red Hat Enterprise Linux 9
ubuntu Ubuntu
windows.10 Microsoft Windows 10
windows.10.virtio Microsoft Windows 10 (virtio)
windows.11 Microsoft Windows 11
windows.11.virtio Microsoft Windows 11 (virtio)
windows.2k12 Microsoft Windows Server 2012
windows.2k12.virtio Microsoft Windows Server 2012 (virtio)
windows.2k16 Microsoft Windows Server 2016
windows.2k16.virtio Microsoft Windows Server 2016 (virtio)
windows.2k19 Microsoft Windows Server 2019
windows.2k19.virtio Microsoft Windows Server 2019 (virtio)
windows.2k22 Microsoft Windows Server 2022
windows.2k22.virtio Microsoft Windows Server 2022 (virtio)

common-instancetypes's People

Contributors

0xfelix avatar akrejcir avatar avivtur avatar codingben avatar dominikholler avatar kubevirt-bot avatar lyarwood avatar opokornyy avatar prnaraya avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

common-instancetypes's Issues

Add `micro` size for `U` instance type series to replace `server.micro`

/kind enhancement

What happened:

Super useful for testing in small dev envs when using something like CirrOS or Alpine.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

`centos.stream9` should not enable EFI and secureboot until the cloud images support UEFI

/kind bug

What happened:

The current CentOS 9 Stream cloud images do not support UEFI boot:

$ virt-filesystems -a CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 --all --long --uuid -h
Name      Type       VFS Label MBR Size Parent   UUID
/dev/sda1 filesystem xfs -     -   7.8G -        a2e35d7f-1574-4fc3-aebd-5e60991d3841
/dev/sda1 partition  -   -     83  7.8G /dev/sda -
/dev/sda  device     -   -     -   10G  -        -

As such using the current centos.stream9 preference that enables efi and secureboot leads to an unbootable guest:

$ virtctl create vm --instancetype u1.medium   --preference centos.stream9 --volume-containerdisk name:centos,src:quay.io/containerdisks/centos-stream:9   --name centos | kubectl apply -f -
[..]
$ virtctl console centos
[..]
BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x6)/Pci(0x0,0x0): Not Found
BdsDxe: No bootable option or device was found.
BdsDxe: Press any key to enter the Boot Manager Menu.

What you expected to happen:

efi and secureboot are not enabled within the centos.stream9 preference until UEFI boot is supported by the cloud images and associated kubevirt/containerdisks.

How to reproduce it (as minimally and precisely as possible):

As above.

Anything else we need to know?:

Found downstream by https://issues.redhat.com/browse/CNV-32936

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

Sync stable branch releases with the corresponding `kubevirt/kubevirt` and `kubevirt/ssp-operator` stable release branches

What happened:

$subject

What you expected to happen:

This behaviour to be automated.

How to reproduce it (as minimally and precisely as possible):

Release a non-latest version of common-instancetypes.

Additional context:
Add any other context about the problem here.

Environment:

  • KubeVirt version (use virtctl version): N/A
  • Kubernetes version (use kubectl version): N/A
  • VM or VMI specifications: N/A
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): N/A
  • Kernel (e.g. uname -a): N/A
  • Install tools: N/A
  • Others: N/A

Windows 11 and 2k22 use TPMs without persistent storage

What happened:

$subject, this can result in unbootable guests when using BitLocker encryption.

What you expected to happen:

Persistent TPM storage to be provided by the Windows 11 and 2k22 preferences to avoid guests using BitLocker encryption becoming unbootable.

How to reproduce it (as minimally and precisely as possible):

  • Boot a Windows 11/2k22 VM using the associated preferences
  • Use BitLocker encryption within the guest
  • Reboot

Additional context:

Reported downstream against CNV v4.16.0 https://issues.redhat.com/browse/CNV-39710

Environment:

  • KubeVirt version (use virtctl version): N/A
  • Kubernetes version (use kubectl version): N/A
  • VM or VMI specifications: N/A
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): N/A
  • Kernel (e.g. uname -a): N/A
  • Install tools: N/A
  • Others: N/A

Add test coverage for resource labels

/kind bug

What happened:

#49

There's no coverage ensuring that the resource labels are correct at present. This could easily be added to the functest job or as a standalone static check after generation.

Flatten the repository structure

Is your feature request related to a problem? Please describe:

The repository is making extensive use of kustomize, which at some places makes it harder than necessary to understand how everything comes together.

Describe the solution you'd like:

We should try to flatten the repo structure and avoid unnecessary deeply nested directory structures and workarounds to Kustomize's behavior where possible.

E.g. it would be nice to flatten instancetypes from this

instancetypes/cx:
1 VirtualMachineClusterInstancetype VirtualMachineInstancetype kustomization.yaml

to this:

instancetypes/cx1:
cx1.yaml sizes.yaml kustomization.yaml

Describe alternatives you've considered:
A clear and concise description of any alternative solutions or features you've considered.

Additional context:
Add any other context or screenshots about the feature request here.

Flatten instance type directories

/kind enhancement

The current structure is pretty nested and could be simplified, for example:

$ tree common-instancetypes/instancetypes/hyperscale/n/
common-instancetypes/instancetypes/hyperscale/n/
├── 1
│   ├── 2xlarge
│   │   └── kustomization.yaml
│   ├── 4xlarge
│   │   └── kustomization.yaml
│   ├── 8xlarge
│   │   └── kustomization.yaml
│   ├── base
│   │   ├── kustomization.yaml
│   │   └── n1.yaml
│   ├── kustomization.yaml
│   ├── large
│   │   └── kustomization.yaml
│   ├── medium
│   │   └── kustomization.yaml
│   └── xlarge
│       └── kustomization.yaml
├── base
│   ├── kustomization.yaml
│   └── n.yaml
├── kustomization.yaml
└── sizes
    ├── 2xlarge
    │   ├── 2xlarge.yaml
    │   └── kustomization.yaml
    ├── 4xlarge
    │   ├── 4xlarge.yaml
    │   └── kustomization.yaml
    ├── 8xlarge
    │   ├── 8xlarge.yaml
    │   └── kustomization.yaml
    ├── large
    │   ├── kustomization.yaml
    │   └── large.yaml
    ├── medium
    │   ├── kustomization.yaml
    │   └── medium.yaml
    └── xlarge
        ├── kustomization.yaml
        └── xlarge.yaml

17 directories, 24 files

As compared to https://github.com/fabiand/instanceTypes/tree/main/series/n1

Remove deprecated legacy instance types

/kind enhancement

What happened:

$ kubectl get virtualmachineclusterinstancetypes \
    -linstancetype.kubevirt.io/deprecated=true
selecting podman as container runtime
NAME                     AGE
highperformance.large    3h53m
highperformance.medium   3h53m
highperformance.small    3h53m
server.large             3h53m
server.medium            3h53m
server.micro             3h53m
server.small             3h53m
server.tiny              3h53m

Simplify the `O` instance type class to inherit directly from `U`

/kind enhancement

What happened:

This class duplicates the U class with the addition of memory overcommit however the current implementation in the repo copies the entire kustomization structure. This can and should be simplified to just inherit N into O with the addition of memory covercommit.

Make `instancetype.kubevirt.io/class` and `instancetype.kubevirt.io/version` labels

/kind enhancement

What happened:

$subject, these would be useful as labels as we could do things like the following:

$ kubectl get virtualmachineclusterinstancetypes \
    -linstancetype.kubevirt.io/class=Overcommitted

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

Windows 11 preferences require only one core

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind enhancement

What happened:

Windows 11 requires at least two cores (see here) but the preferences in common-instancetypes require only one CPU.

What you expected to happen:

Preferences should require at least two CPUs.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

The request is invalid: spec.template.spec.domain.resources.requests.memory: spec.template.spec.domain.resources.requests.memory '0' must be equal to or larger than page size spec.template.spec.domain.hugepages.size '1Gi'

/kind bug

Tracking bug for the following KubeVirt bug and PR:

kubevirt/kubevirt#9102

kubevirt/kubevirt#9103

$ make cluster-down && make cluster-up && make cluster-sync && make cluster-functest
[..]
virtualmachine.kubevirt.io "vm-highperformance.small" deleted
selecting docker as container runtime
The request is invalid: spec.template.spec.domain.resources.requests.memory: spec.template.spec.domain.resources.requests.memory '0' must be equal to or larger than page size spec.template.spec.domain.hugepages.size '2Mi'
functest failed on instance type m1.2xlarge
make: *** [Makefile:51: cluster-functest] Error 1

Reconsider default huge page requirement for compute and memory IT classes

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind enhancement

What happened:

Using a compute or memory instance type requires a node with huge pages support, which is not a default setting, so out of the box, compute and memory instance types are unusable.

When trying to create a VM in one of these instance types, it fails with an unhelpful error that points to scheduling, but does not mention huge pages as the reason.

What you expected to happen:

Either better messaging to bring to light the lack of huge pages or another way to introduce to admins the ability to require huge pages in an instance type.

How to reproduce it (as minimally and precisely as possible):

Create a vm with compute or memory instance type on a cluster where nodes do not have huge page support.

Anything else we need to know?:

Huge pages make sense across the board for VM handling nodes, but fine tuning the number of huge pages per compute node is a per-cluster exercise, and fits better in a performance and tuning guide than a default.

O series description typo

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
typo, should be based on U series

https://github.com/kubevirt/common-instancetypes/blob/main/common-instancetypes/instancetypes/hyperscale/o/base/o.yaml#L8

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

Missing labels on `VirtualMachineClusterInstancetype` o1 series - cpu/memory

/kind bug
/cc @RoniKishner

What happened:

The o1 series (overcommit) VirtualMachineClusterInstancetype we deploy are missing cpy/memory labels:

  • instancetype.kubevirt.io/cpu
  • instancetype.kubevirt.io/memory

Those labels help in getting a list of optional resources which we can chose from.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

https://bugzilla.redhat.com/show_bug.cgi?id=2223539

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

Consider merging CX and N use-cases

Is this a BUG REPORT or FEATURE REQUEST?:
Currently Network (N series) and Compute Exclusive (CX series) differ by only one aspect: vNUMA:

vNUMA - Physical NUMA topology is reflected in the guest in order to optimize guest sided cache utilization.

However the network use case also requires vNUMA for it's proper work. For example, DPDK workloads require the CPUs scheduled for the VM to be of the same NUMA node. This is currently done by using the Node Tuning Operator in order to enforce "single-numa-topology".
Please consider joining these two use cases, as it can reduce your maintenance work in this project.

Uncomment only one, leave it on its own line:

/kind bug
/kind enhancement

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • KubeVirt version (use virtctl version):
  • Kubernetes version (use kubectl version):
  • common-templates version:
  • VM or VMI specifications:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.