Comments (7)
jbpratt
commented on July 17, 2024
1
Fixed the private image @usrbinkat
from kargo.
jbpratt
commented on July 17, 2024
i-it just worked 🎉
❯ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6ddb68799c-68rt8 1/1 Running 0 8m7s
kube-system calico-node-q86mq 1/1 Running 0 9m23s
kube-system coredns-8474476ff8-q7btk 1/1 Running 0 7m14s
kube-system kube-apiserver-node1 1/1 Running 0 11m
kube-system kube-controller-manager-node1 1/1 Running 1 11m
kube-system kube-multus-ds-c9g6s 1/1 Running 0 6m33s
kube-system kube-proxy-b7xts 1/1 Running 0 11m
kube-system kube-scheduler-node1 1/1 Running 1 11m
kube-system nodelocaldns-87l9j 1/1 Running 0 6m45s
~ 567ms
❯ oc describe node | grep arm
Labels: beta.kubernetes.io/arch=arm64
kubernetes.io/arch=arm64
Architecture: arm64
from kargo.
jbpratt
commented on July 17, 2024
ran into the first of many to come
mister main [?] 3s
❯ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-57d89b9548-22rxt 1/1 Running 0 117s
cert-manager cert-manager-cainjector-5bcf77b697-sw6ft 1/1 Running 0 117s
cert-manager cert-manager-webhook-9cb88bd6d-r8cvm 1/1 Running 0 117s
cluster-network-addons cluster-network-addons-operator-549b8f8966-mqmwx 0/1 Error 0 62s
kube-system calico-kube-controllers-6ddb68799c-68rt8 1/1 Running 0 117m
kube-system calico-node-q86mq 1/1 Running 0 118m
kube-system coredns-8474476ff8-q7btk 1/1 Running 0 116m
kube-system kube-apiserver-node1 1/1 Running 0 121m
kube-system kube-controller-manager-node1 1/1 Running 1 121m
kube-system kube-multus-ds-c9g6s 1/1 Running 0 116m
kube-system kube-proxy-b7xts 1/1 Running 0 121m
kube-system kube-scheduler-node1 1/1 Running 1 121m
kube-system nodelocaldns-87l9j 1/1 Running 0 116m
mister main [?] 768ms
❯ k logs pod/cluster-network-addons-operator-549b8f8966-mqmwx -n cluster-network-addons
standard_init_linux.go:228: exec user process caused: exec format error
from kargo.
jbpratt
commented on July 17, 2024
Going to document this a bit better for future reference:
- Install and boot an RPi4 running Fedora 34 (https://pagure.io/arm-image-installer)
sudo arm-image-installer \
--image /tmp/Fedora-Server-34-1.2.aarch64.raw.xz \
--addkey ~/.ssh/id_rsa.pub \
--resizefs \
--target rpi4 \
--media /dev/sda
- Configure a user + bridge interface (
br0) - Execute Kargo to bring up a cluster against the host(s)
- Wait (just long enough to fix a coffee and stretch)
- Install the developer arm64 build (https://kubevirt.io/user-guide/operations/installation/#experimental-arm64-developer-builds)
LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)
kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml
kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml
- KubeVirt is now deployed!
kubevirt virt-api-6f994b8b7b-b77q4 1/1 Running 0 27m
kubevirt virt-api-6f994b8b7b-dlr8w 1/1 Running 0 27m
kubevirt virt-controller-7f4bc56796-644zf 1/1 Running 0 27m
kubevirt virt-controller-7f4bc56796-tlzql 1/1 Running 0 27m
kubevirt virt-handler-rf7fx 1/1 Running 0 27m
kubevirt virt-operator-78f55c7745-tcwms 1/1 Running 0 29m
kubevirt virt-operator-78f55c7745-wnzm8 1/1 Running 0 29m
Took the existing fedora-br0 template and changed it to my arm64 image (ghcr.io/jbpratt/qubo/fedora:34-arm64)
fedora-br0-arm64.yml
---
# https://github.com/kubevirt/kubevirt/blob/master/docs/cloud-init.md
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
name: fedora-br0
namespace: kargo
labels:
app: kargo
spec:
running: true
template:
spec:
evictionStrategy: LiveMigrate
nodeSelector:
node-role.kubernetes.io/kubevirt: ""
domain:
clock:
utc: {}
timer: {}
cpu:
cores: 1
sockets: 1
threads: 2
model: host-passthrough
dedicatedCpuPlacement: false
devices:
rng: {}
autoattachPodInterface: false
autoattachSerialConsole: true
autoattachGraphicsDevice: true
networkInterfaceMultiqueue: false
disks:
- name: containerdisk
bootOrder: 1
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: enp1s0
model: virtio
bridge: {}
features:
acpi:
enabled: true
smm:
enabled: true
firmware:
bootloader:
efi:
secureBoot: true
machine:
type: q35
resources:
limits:
memory: 2G
requests:
memory: 2G
devices.kubevirt.io/kvm: "1"
hostname: fedora-br0
networks:
- name: enp1s0
multus:
networkName: kargo-net-attach-def-br0
terminationGracePeriodSeconds: 0
accessCredentials:
- sshPublicKey:
source:
secret:
secretName: kargo-sshpubkey-kc2user
propagationMethod:
qemuGuestAgent:
users:
- "kc2user"
volumes:
- name: containerdisk
containerDisk:
image: ghcr.io/jbpratt/qubo/fedora:34-arm64
imagePullPolicy: Always
- name: cloudinitdisk
cloudInitNoCloud:
networkData: |
version: 2
ethernets:
enp1s0:
dhcp4: true
dhcp6: true
dhcp-identifier: mac
userData: |
#cloud-config
hostname: fedora-br0
ssh_pwauth: true
disable_root: true
chpasswd:
list: |
kc2user:kc2user
expire: False
users:
- name: kc2user
shell: /bin/bash
lock_passwd: false
sudo: ['ALL=(ALL) NOPASSWD:ALL']
groups: sudo,wheel
growpart:
mode: auto
devices: ['/']
ignore_growroot_disabled: true
package_upgrade: true
packages:
- vim
- screenfetch
runcmd:
- "screenfetch"
❯ k apply -f kargo/vm/fedora-br0-arm64.yml
The request is invalid:
* spec.template.spec.domain.machine.type: spec.template.spec.domain.machine.type is not supported: q35 (allowed values: [virt*])
* spec.template.spec.evictionStrategy: LiveMigration feature gate is not enabled
Deleting those bits allows the schema validation to pass, and creating again results in a FailedCreate for the VM.
kargo 0s Warning FailedCreate virtualmachine/fedora-br0 Error creating virtual machine instance: admission webhook "virtualmachineinstances-create-validator.kubevirt.io" denied the request: UEFI secure boot is currently not supported on aarch64 Arch
Let's disable secure boot as well. The diff of the file looks like this now:
diff --git a/kargo/vm/fedora-br0-arm64.yml b/kargo/vm/fedora-br0-arm64.yml
index 8ddf458..435fc92 100644
diff --git a/kargo/vm/fedora-br0-arm64.yml b/kargo/vm/fedora-br0-arm64.yml
index 8ddf458..435fc92 100644
--- a/kargo/vm/fedora-br0-arm64.yml
+++ b/kargo/vm/fedora-br0-arm64.yml
@@ -11,7 +11,6 @@ spec:
running: true
template:
spec:
- evictionStrategy: LiveMigrate
nodeSelector:
node-role.kubernetes.io/kubevirt: ""
domain:
@@ -42,17 +41,6 @@ spec:
- name: enp1s0
model: virtio
bridge: {}
- features:
- acpi:
- enabled: true
- smm:
- enabled: true
- firmware:
- bootloader:
- efi:
- secureBoot: true
- machine:
- type: q35
resources:
It seems the virtual machine is created successfully, but it isn't being schedule to create an instance.
kargo 0s Normal SuccessfulCreate virtualmachine/fedora-br0 Started the virtual machine by creating the new virtual machine instance fedora-br0
Looking at the logs of virt-controller, it is stuck in a loop trying to schedule
{"component":"virt-controller","level":"info","msg":"TSC Freqency node update status: 0 updated, 0 skipped, 0 errors","pos":"nodetopologyupdater.go:47","timestamp":"2021-11-17T23:53:24.397214Z"}
{"component":"virt-controller","level":"info","msg":"reenqueuing VirtualMachineInstance kargo/fedora-br0","pos":"vmi.go:254","reason":"failed to render launch manifest: Failed to locate network attachment definition kargo/kargo-net-attach-def-br0","timestamp":"2021-11-17T23:54:02.789833Z"}
{"component":"virt-controller","level":"error","msg":"Skipping TSC frequency updates on all nodes","pos":"nodetopologyupdater.go:54","reason":"failed to calculate lowest TSC frequency for nodes: no schedulable node exposes a tsc-frequency","timestamp":"2021-11-17T23:54:27.367711Z"}
{"component":"virt-controller","level":"info","msg":"TSC Freqency node update status: 0 updated, 0 skipped, 0 errors","pos":"nodetopologyupdater.go:47","timestamp":"2021-11-17T23:54:27.367962Z"}
It looks like there aren't many great matches for your search 😺
from kargo.
usrbinkat
commented on July 17, 2024
set virtualization to pvm (disable hvm) and re-try
if that works, then we'll know to focus on details around the /dev/kvm device
from kargo.
usrbinkat
commented on July 17, 2024
Failed on image pull
Failed to pull image "ghcr.io/jbpratt/qubo/fedora:34-arm64": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized
Steps:
- Write image
sudo arm-image-installer \
--image ./Fedora-Server-34-1.2.aarch64.raw.xz \
--addkey ~/.ssh/id_rsa.pub --resizefs --target rpi4 \
--media /dev/sde --showboot --args 'kvm-arm.mode=nvhe'
- Deploy per your instructions
- Copy kubeconfig to local system
- Create ssh secret
kubectl create secret generic kargo-sshpubkey-kc2user \
--namespace default --dry-run=client -oyaml \
--from-file=key1=$HOME/.ssh/id_rsa.pub \
| kubectl apply -f -
- Add network attachment definition
- Remove node selector and other erroneous fields
- Apply the following manifest
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: kargo-net-attach-def-br0
spec:
config: '{"cniVersion":"0.3.1","name":"br0","plugins":[{"type":"bridge","bridge":"br0","ipam":{}},{"type":"tuning"}]}'
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: fedora-aarch64
labels:
app: kargo
spec:
running: true
template:
metadata:
creationTimestamp: null
spec:
accessCredentials:
- sshPublicKey:
propagationMethod:
qemuGuestAgent:
users:
- kc2user
source:
secret:
secretName: kargo-sshpubkey-kc2user
domain:
clock:
timer: {}
utc: {}
cpu:
cores: 1
model: host-passthrough
sockets: 1
threads: 2
devices:
autoattachGraphicsDevice: true
autoattachPodInterface: false
autoattachSerialConsole: true
disks:
- bootOrder: 1
disk:
bus: virtio
name: containerdisk
- disk:
bus: virtio
name: cloudinitdisk
interfaces:
- bridge: {}
model: virtio
name: enp1s0
networkInterfaceMultiqueue: false
rng: {}
machine:
type: virt
resources:
limits:
memory: 2G
requests:
devices.kubevirt.io/kvm: "1"
memory: 2G
hostname: fedora-br0
networks:
- multus:
networkName: kargo-net-attach-def-br0
name: enp1s0
terminationGracePeriodSeconds: 0
volumes:
- containerDisk:
image: ghcr.io/jbpratt/qubo/fedora:34-arm64
imagePullPolicy: Always
name: containerdisk
- cloudInitNoCloud:
networkData: |
version: 2
ethernets:
enp1s0:
dhcp4: true
dhcp6: true
dhcp-identifier: mac
userData: |
#cloud-config
hostname: fedora-br0
ssh_pwauth: true
disable_root: true
chpasswd:
list: |
kc2user:kc2user
expire: False
users:
- name: kc2user
shell: /bin/bash
lock_passwd: false
sudo: ['ALL=(ALL) NOPASSWD:ALL']
groups: sudo,wheel
growpart:
mode: auto
devices: ['/']
ignore_growroot_disabled: true
package_upgrade: true
packages:
- vim
- screenfetch
runcmd:
- "screenfetch"
name: cloudinitdisk
Next Steps: re-try with credentials for ready-to-run aarch64 image
from kargo.
jbpratt
commented on July 17, 2024
Going to go ahead and close this exploration issue out. Further work can receive a new ticket.
from kargo.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kargo.