Add support for Zen 3 march about pve-edge-kernel HOT 20 CLOSED

Hi, I have decided not to offer pre-built kernels for non-generic microarchitectures anymore. The performance benefit of optimizing the kernel for a specific microarchitecture is unfortunately negligible and I don't have the time to maintain multiple builds.

from pve-edge-kernel.

I don't have any Zen 3 machines in my cluster (yet),

Not trying to push, but, you are more than welcome on my machine.

Zen 3 optimizations has only landed in GCC 11

Thanks for the tip! I didn't know. But I might be okay on compiling edge gcc and its edgy friends. :)

Note that the Zen 2 kernel is identical to the generic kernel, except that it is tuned for the Zen 2 micro-architecture (via GCC's -march=znver2 option)

Yeah, I guessed this. I imagine Zen 3 also might be a compiler flag, simple enough to be within my pay grade. :) I might even try it! :)

It's nice that you make edge kernels more easily available to Proxmox peeps. Thank you again, Fabian!

from pve-edge-kernel.

@Ramalama2

You might try to install GCC 11 from Debian Experimental. You can then just follow the steps in the README to obtain the repository and build the kernel as follows:

make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-march=znver3"

This will also build OpenZFS for you and include it in the kernel package.

The configuration for these kernels is based on the original Proxmox configs (which is based in turn on Ubuntu). See

As for my build environment, I use a Ubuntu Focal-based 64-core system for the Github Actions Runner.

from pve-edge-kernel.

Okay here we go:
Build env: Debian Bullseye + Experimental repo (gcc-11 + other packages)
Compilation Flags: make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-O3 -march=znver3"
Kernel: 5.11.5 + openzfs 2.0.3
Download: https://cloud.golima.de/s/jMHp4poNJ3Dq27C

Removed:

-d CONFIG_XEN \
-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_DRM_I915 \
-d CONFIG_DRM_XEN \
-d CONFIG_DRM_GMA500 \
-d CONFIG_SND_FIREWIRE \
-d CONFIG_SND_XEN_FRONTEND \
-d CONFIG_HYPERV \
-d CONFIG_SURFACE_PLATFORMS \
-d CONFIG_NET_VENDOR_GOOGLE \
-d CONFIG_CHROME_PLATFORMS \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Why removed? -> Because it is an amd zen3 hypervisor, doesn't need intel and doesn't need GPU/Sound/Wifi/BT etc...

Cheers

from pve-edge-kernel.

Hi Tom,

Thank you for your message! Given that I don't have any Zen 3 machines in my cluster (yet), I haven't thought about adding a build for it. Note that the Zen 2 kernel is identical to the generic kernel, except that it is tuned for the Zen 2 micro-architecture (via GCC's -march=znver2 option)

As for adding Zen 3 builds, it seems that support for Zen 3 optimizations has only landed in GCC 11, which has not been released in Ubuntu as far as I know. Though, it should not matter much, using the generic or Zen 2 kernel should work fine for you.

from pve-edge-kernel.

Thank you for the nice comment!

If you want to try it yourself, and manage to install GCC 11, I expect the following command to achieve what you want (given that you have cloned the repository):

make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-march=znver3"

from pve-edge-kernel.

Hey guys, im interested in zen3 either 😂

What are you using as build environment @fabianishere ?

Cause I don't think, we can use anything, it needs to have gcc11 + dwarf from debian testing etc, not that super easy i think 🙈

If i can get it working, i can probably help out building and uploading it.

And the kernels got gigantic, holy fuck, i remember 15y back, how small they were, whatever, offtopic 😂

So please provide some info for a good build environment.
And another thing is, do you use menuconfig and set anything when you build these? Cause i didn't seen that any is provided.

Additionally, how it works now with openzfs? This needs to be cloned inside the kernel somehow either?
Or recompiled afterwards on the system?

Sorry, im a bit outdated 🤦‍♂️

from pve-edge-kernel.

Thanks, im going to try out after work!

from pve-edge-kernel.

Hi, just a small update,

Im compiling 5.11.5 right now, but didn't used -j`nproc --all`, so it will take forever xD
But i used the -O3 flag additionally.
It looks quite good with gcc-11, so far no problems. I using debian bullseye with the experimental repo as environment.

However, are there ways to improve the rules somehow?
because there are so many unnecessary drivers that blows the kernel up, that no one needs on proxmox or a hypervisor.
like 80211 for example and a ton of gpu drivers, i mean even nvidia & amd is unneccessary, because 99,9% of users will pass them through and blacklist those xD

Just looking for ways to slim down the kernel and maybe optimize it more for zen3.

Another thing is, we don't need some intel drivers on an amd kernel & opposite.

However, when the compilation is done, i will send a link here, for anyone interrested.
Cheers

from pve-edge-kernel.

You can pass your own kernel configuration file if you prefer, but this requires some changes. To change the Proxmox-specific options, modify:

If you also want to remove the Ubuntu/Debian configuration options, modify:

from pve-edge-kernel.

I simply added this into pve-edge-kernel/debian/rules into the PVE_CONFIG_OPTS category xD

-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Looked like the easiest way for me to remove the bloat, those things are self explaining, tryed to remove Wireless/BT/Intel/GPU driver for amd based kernel on a hypervisor.

However, there are more things that needs to get removed, everyone that want to help making a list is greatly welcome:

isdn
misdn
thermal/intel
media/radio
media/cec
media/dvb
media/usb/dvb
media/tuners
drivers/input/touchscreen
drivers/input/tablet
drivers/input/joystick
drivers/firewire
drivers/net/ieee802154
drivers/net/ethernet/qualcomm
drivers/net/ethernet/huawei
drivers/net/ethernet/google
drivers/net/ethernet/amazon
drivers/net/ethernet/3com
drivers/net/dsa/hirschmann
drivers/misc/cardreader
drivers/gpu/drm/i915 - gvt-g (but not available on amd systems anyway, until iris-xe pci card & until gvt-g drivers support those)
drivers/video/backlight
arch/x86/xen
arch/x86/ia32 - needed?
drivers/staging/media/atomisp
drivers/platform/chrome
drivers/platform/surface
drivers/pcmcia
drivers/iio - what is this?
net/mac802154
net/wireless
sound/xen
sound/soc/intel
sound/firewire
sound/pci - is there anything that is needed?
sound/soc/intel

This looks like a random list, but i think no one needs this in the zen kernel.
There is for sure a ton more, but i don't have that much clue and need some help xD

However, here is the kernel (without these in the kernel: amd/radeon/nouveau/bt/wifi etc):
https://cloud.golima.de/s/HJs582PrX4cpTza
(kernel+headers+libc)
EDIT: Don't use this, some packages failed, needs a recompile, i will update this. See below, i missed some packages!

Makefile.config:555: No sys/sdt.h found, no SDT events are defined, please install systemtap-sdt-devel or systemtap-sdt-dev
Makefile.config:603: No libunwind found. Please install libunwind-dev[el] >= 1.1 and/or set LIBUNWIND_DIR
Makefile.config:782: Python support disabled by user
Makefile.config:833: Old version of libbfd/binutils things like PE executable profiling will not be available
Makefile.config:894: No libzstd found, disables trace compression, please install libzstd-dev[el] and/or set LIBZSTD_DIR
Makefile.config:905: No libcap found, disables capability support, please install libcap-devel/libcap-dev
Makefile.config:973: No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-dev[el]/libbabeltrace-ctf-dev
Makefile.config:999: No alternatives command found, you need to set JDIR= to point to the root of your Java directory

Auto-detecting system features:
...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ OFF ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ OFF ]
...                     libpython: [ OFF ]
...                     libcrypto: [ on  ]
...                     libunwind: [ OFF ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ OFF ]
...        disassembler-four-args: [ OFF ]

That means for me, that we need some additional packages:
apt install systemtap-sdt-dev libunwind-dev libzstd-dev libcap-dev libbabeltrace-dev libpython3.9-dev libperl-dev binutils-dev

(i have installed default-jdk either, dunno what the compiler wants, but whatever, enough diskspace is available)

Will Provide an updated Package in a bit.

from pve-edge-kernel.

Okay, after removing binutils-dev, it works....
Forgot to link python-config... Or didn't knew before... fixed now anyway too xD

but this here doesnt work;

And that feels somehow super wrong.

How does that section looks in your compiler?

from pve-edge-kernel.

https://lists.debian.org/debian-release/2015/02/msg00080.html

Well i found this, seems like it's intended.
But it doesn't make any sense, why the hell they provide libbfd if it has an incompatible license at all?

That means that everything is right?

from pve-edge-kernel.

Okay, i tested what i compiled 🤣

FML, this has less todo with the compilation, and more with the fact of 5.11 itself.
But all my errors are gone!

• ethtool let you set fixed 2,5GB/s speed (normally it works only through autoneg, but autoneg at this weird 2,5G standard is slow af) (this isn't even possible with mainline intel driver, but with 5.11 it is xD)
• RDRAND Zen3 issues with zen3 (systemd) are fixed!!!! This driven me crazy xD
• EDAC ECC, is fully working!

FreeBSD 12.2 (OpenSense) doesn't boots anymore with cpu "host" -> Setted to EPYC and it boots...
Slowly im getting mad of FreeBSD, so many bugs, that doesn't have even todo with new hardware, like q35 passthrough...

However, everything works perfectly!!! Im happy that i did this!
Thanks for help @fabianishere

I can try to provide the builds from time to time!

from pve-edge-kernel.

Hey guys, small update, as next 5.11.7 is comming, since it has more changes as the previous ones:

Changelog 5.11.7:
- iommu/amd: Fix performance counter initialization
- Full Changelog (305 Changes): https://lwn.net/Articles/849642/

Changelog 5.11.6:
- Full Changelog (37 Changes): https://lwn.net/Articles/849101/

Changelog 5.11.5:
- Full Changelog (45 Changes): https://lwn.net/Articles/848837/

Changelog 5.11.4:
- Full Changelog (104 Changes): https://lwn.net/Articles/848616/

But it will take a bit, im trying to compile the new kernel with AOCC(v3.0 Clang LLVM 12)+LTO(LLD)+O3 for Zen 2 & Zen 3
If someone wants to try AOCC, you need this patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c75173a26948363bdd11a0d5b90bd012ce4cc2e7

Wish me luck xD

from pve-edge-kernel.

@fabianishere

While the complete building process of 5.11.7 runs really well, i get almost no warnings anymore. Seems like either the compiler or code was improved...

Im running on building 5.11.7 into this issue:

# finalize
/sbin/depmod -b debian/pve-edge-kernel-5.11.7-1/ 5.11.7-1
depmod: WARNING: could not open modules.order at /root/pve-edge-kernel/build/debian/pve-edge-kernel-5.11.7-1//lib/modules/5.11.7-1: No such file or directory
depmod: WARNING: could not open modules.builtin at /root/pve-edge-kernel/build/debian/pve-edge-kernel-5.11.7-1//lib/modules/5.11.7-1: No such file or directory
# Autogenerate blacklist for watchdog devices (see README)
install -m 0755 -d debian/pve-edge-kernel-5.11.7-1/lib/modprobe.d
ls debian/pve-edge-kernel-5.11.7-1/lib/modules/5.11.7-1/kernel/drivers/watchdog/ > watchdog-blacklist.tmp
ls: cannot access 'debian/pve-edge-kernel-5.11.7-1/lib/modules/5.11.7-1/kernel/drivers/watchdog/': No such file or directory
make[1]: *** [debian/rules:166: .install_mark] Error 2
make[1]: Leaving directory '/root/pve-edge-kernel/build'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:88: pve-edge-kernel-5.11.7-1_5.11.7-1_amd64.deb] Error 2

I guess there is something missing from ubuntu/debian to build the install packages?

from pve-edge-kernel.

Yeah, okay, i gotcha...

I didn't knew that i need to checkout a specific ubuntu-mainline version xD

from pve-edge-kernel.

• https://cloud.golima.de/s/E7KiWLFgzSp4MNP (Zen2 + Zen3) (Build: gcc-11 +O3)

Changelog 5.11.7:

• iommu/amd: Fix performance counter initialization
• Full Changelog (305 Changes): https://lwn.net/Articles/849642/

Removed from Kernel:

-d CONFIG_XEN \
-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_DRM_I915 \
-d CONFIG_DRM_XEN \
-d CONFIG_DRM_GMA500 \
-d CONFIG_SND_FIREWIRE \
-d CONFIG_SND_XEN_FRONTEND \
-d CONFIG_HYPERV \
-d CONFIG_SURFACE_PLATFORMS \
-d CONFIG_NET_VENDOR_GOOGLE \
-d CONFIG_CHROME_PLATFORMS \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Didn't managed to build AOCC with LTO yet, running into some errors that needs fixing first....

from pve-edge-kernel.

For everyone that's interrested:
https://cloud.golima.de/s/E7KiWLFgzSp4MNP
Please keep in mind, to revert apparmor features back to stock.

I provide there Znver3 buils, build based on official pve 5.11 repo.
If someone wants to build his own kernel for another microarchitecture, use (kernel-update.sh) script.
There are no modules removed, it's 1:1 official, just updated and build with znver3.
There probably won't be 5.12 kernels.

Cheers

from pve-edge-kernel.

i see it similar after all the time of doing this and testing.
But it took me long to come to this conclusion.
I will still provide some builds, but it's gonna happen 1-2x a month, not for every change anymore.

Thanks for your effort fabian, i learned here much!

Edit: Probably you can simply provide an "core2" build. There is a minimal performance benefit, and everyone can use it. Even an 2012 atom has sse3.
But that's up to you. Just can be the golden middle 😂

However, thank you very much and cheers 👍

from pve-edge-kernel.