A third party build of Python 3.10 is failing on Oracle UEK R6 and Oracle UEK R7:

-bash-4.2$ uname -r
5.4.17-2136.330.7.1.el7uek.x86_64
-bash-4.2$ ./python3.10
-bash: ./python3.10: cannot execute binary file

bash-4.4$ uname -r
5.15.0-204.147.6.2.el8uek.x86_64

bash-4.4$ ./python3.10
bash: ./python3.10: cannot execute binary file: Exec format error

The same program seems to work fine on Red Hat Linux kernel 3.10 and 4.18, as well as a custom compile of Linux kernel 6.1 and Linux kernel 6.6.

After a deep dive, I found that the issue seems to be due to the "Reserved VA" feature introduced with this commit:

commit a48aa31a29ea85c9c08d88ac9adb1cb07b7ee670
Author: Khalid Aziz <[email protected]>
Date:   Thu Aug 8 14:01:10 2019 -0600

    mm: Allow userspace to reserve VA range for use by userspace only
    ...

For whatever reason - the python 3.10 binary has p_offset == 0, p_filesz == 0, and p_memsz == 0. They have a python 3.9 binary which is ok, and a python 3.10 binary which is not ok. The python 3.9 is "not stripped" while the python 3.10 is "stripped". This makes me suspect that their version of binutils had a bug, and something like "strip" corrupted the PT_NOTE program header. I have not been able to reproduce the corruption yet. I only know that the corruption exists:

bash$ od -A d -t x1 --skip-bytes 400 --read-bytes 56 python3.10
0000400 04 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00
0000416 54 02 40 00 00 00 00 00 54 02 40 00 00 00 00 00
0000432 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000448 04 00 00 00 00 00 00 00
0000456

The Oracle UEK R6 and Oracle UEK R7 both parse the elf headers and fail with ENOEXEC if they encounter this file. The Red Hat Linux kernel and the upstream kernel, does not implement this parsing, and does not fail.

I was able to patch the ELF header in-place, correcting the p_offset to be the offset into the file of the first note, and p_filesz/p_memsz to be the offset past the second note:

bash$ od -A d -t x1 --skip-bytes 400 --read-bytes 56 python3.10-fixed
0000400 04 00 00 00 04 00 00 00 78 07 00 00 00 00 00 00
0000416 54 02 40 00 00 00 00 00 54 02 40 00 00 00 00 00
0000432 44 00 00 00 00 00 00 00 44 00 00 00 00 00 00 00
0000448 04 00 00 00 00 00 00 00
0000456 

With the python3.10-fixed, UEK R6 and UEK R7 both work again:

-bash-4.2$ uname -r
5.4.17-2136.330.7.1.el7uek.x86_64
-bash-4.2$ ./python3.10-fixed
Python 3.10.8 (main, Nov 24 2022, 16:36:59) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

bash-4.4$ uname -r
5.15.0-204.147.6.2.el8uek.x86_64
bash-4.4$ ./python3.10-fixed
Python 3.10.8 (main, Nov 24 2022, 16:36:59) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

I believe the problem code is here (fs/binfmt_elf.c):

static int get_elf_notes(struct linux_binprm *bprm, struct elf_phdr *phdr, char **notes, size_t *notes_sz)
{
        char *data;
        size_t datasz;
        int ret;

        if (!phdr)
                return 0;

        datasz = phdr->p_filesz;
        if ((datasz > MAX_FILE_NOTE_SIZE) || (datasz < sizeof(struct elf_note)))
                return -ENOEXEC;
        …
}

If p_filesz==0, then it will not be > MAX_FILE_NOTE_SIZE, but it will be < sizeof(struct elf_note), therefore it will fail with ENOEXEC.

I believe it makes sense to fail (not necessarily ENOEXEC, but perhaps) if the ELF data structure appears to be corrupt in such a way that the parsing is inconclusive as to what is intended. If the notes segment has an unexpectedly short p_filesz, then it is not possible to read the header fields to make further decisions. However, p_filesz==0 is a special case.

I do think that ENOEXEC is asking for trouble. If non-UEK kernels are happy with the files - corrupt, or non-compliant, or some situation not tested for, but UEK kernel fails, this seems to not pass the robustness principle to be liberal with what you will receive, but conservative with what you produce. In this case, I don't want to have to switch back to Red Hat for the users of this tool, who are not users of the "Reserved VA" extension. This is essentially a robustness issue.

If I look at binutils, glibc, and other parsers of ELF, it is common to ignore segments that have p_filesz==0. Commit history suggest that attempts have been made to make it stricter in the past, but they were relaxed due to real-life situations where p_filesz==0 is possible.

binutils has code like this:

  /* Read in program headers and parse notes.  */
  for (i = 0; i < i_ehdr.e_phnum; ++i, ++i_phdr)
    {
      ...
      if (i_phdr->p_type == PT_NOTE && i_phdr->p_filesz > 0)
      ...

There could be other bits of the "Reserved VA" code in UEK R6 and UEK R7 that also error if p_filesz == 0. I found the one that seemed most suspect.

There could be other issues, although patching only the p_offset / p_filesz / p_memsz in the file seemed to resolve the issue for me.

I did notice a second issue, but so far it doesn't seem to have caused an issue:

        off = round_up(sizeof(note.nhdr) + NOTE_NAME_SZ,
                       ELF_GNU_PROPERTY_ALIGN);
        if (off > n)
                return -ENOEXEC;

The parse_elf_properties function in UEK fs/binfmt_elf.c seems to presume a fixed / constant sized ELF_GNU_PROPERTY_ALIGN based upon the target platform. 4 bytes on 32-bit system, and 8 bytes on 64-bit system. However, Linux seems to have a well established extension whereby it is acceptable to use either 4 byte alignment or 8 byte alignment for program segments and sections, and it depends upon the section s_addralign value where s_addralign <= 4 will use a value of 4, and s_addralign == 8 will use a value of 8. These section then have to be contained within program segments that have p_align that matches. In the case of notes, it is common to have two PT_NOTE segments - one with p_align == 4 and one with p_align == 8.

Here is a reference for the last issue, fixed upstream in binutils:

https://sourceware.org/bugzilla/show_bug.cgi?id=22444

There seems to be a regression between UEK 5.4.17-2136.304.4.5 and the previous 5.4.17-2102.201.3 version on OL7 with regards to running third-party binaries that are linked against different glibc versions than the one included in the OS.

After updating to this version, running certain binaries fails with a cannot execute binary file error. I was able to reproduce this with the binary tarball distribution of Percona Server for MySQL, as well as a snap install of MicroK8S (as reported on the K8S forum).

I initially reported this issue here, but wanted to make sure this is getting sufficient visibility. Here's a quick example on how to reproduce this:

Boot OL7 using linux-uek-5.4.17-2102.201.3, download Percona Server for MySQL and note that the mysql binary works as expected:

[vagrant@localhost ~]$ cat /etc/oracle-release 
Oracle Linux Server release 7.9
[vagrant@localhost ~]$ uname -r
5.4.17-2102.201.3.el7uek.x86_64
[vagrant@localhost ~]$ wget -q https://downloads.percona.com/downloads/Percona-Server-5.7/Percona-Server-5.7.36-39/binary/tarball/Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz.sha256sum
[vagrant@localhost ~]$ wget -q https://downloads.percona.com/downloads/Percona-Server-5.7/Percona-Server-5.7.36-39/binary/tarball/Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz
[vagrant@localhost ~]$ sha256sum -c Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz.sha256sum 
Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz: OK
[vagrant@localhost ~]$ tar xf Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz
[vagrant@localhost ~]$ ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql --version
./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql Ver 14.14 Distrib 5.7.36-39, for Linux (x86_64) using 6.0

Now perform a yum update and reboot into the 5.4.17-2136.304.4.5 kernel:

[vagrant@localhost ~]$ uname -r
5.4.17-2136.304.4.5.el7uek.x86_64
[vagrant@localhost ~]$ ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql --version
-bash: ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql: cannot execute binary file

Note that this error only seems to occur with third-party binaries that are linked against a different glibc version (as it's the case for the Percona Server binary and the MicroK8S snap). The rest of the OS seems to be working fine.

Description:
Since the UEK7u2, SR-IOV is broken for Mellanox ConnectX4, if one or both ports are configured for ethernet mode.
The same works fine if the ports are configure for IB mode.
Additionally, everything is fine on UEK7u1 regardless of the configuration.

Diagnostic info:
Port 0 is in IB mode
Port 1 is in ETH mode

Output of lspci :

lspci |grep Mellanox

0b:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0b:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
0b:00.2 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.3 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.4 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.5 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.6 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
0b:00.7 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

Output of from dmesg:

dmesg |grep mlx

[ 1.349024] mlx5_core 0000:0b:00.0: firmware version: 12.28.2006
[ 1.349050] mlx5_core 0000:0b:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.071873] mlx5_core 0000:0b:00.0: Port module event: module 0, Cable plugged
[ 4.265254] mlx5_core 0000:0b:00.1: firmware version: 12.28.2006
[ 4.265304] mlx5_core 0000:0b:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.683438] mlx5_core 0000:0b:00.1: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)
[ 4.687324] mlx5_core 0000:0b:00.1: Port module event: module 1, Cable plugged
[ 4.932149] mlx5_core 0000:0b:00.1: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 4.941115] mlx5_core 0000:0b:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0 basic)
[ 4.943224] mlx5_core 0000:0b:00.1 enp11s0f1np1: renamed from eth0
[ 5.655608] mlx5_core 0000:0b:00.1: Successfully registered panic handler for port 1
[ 5.773746] mlx5_core 0000:0b:00.1: mlx5_cmd_out_err:803:(pid 814): QUERY_HCA_CAP(0x100) op_mod(0x40) failed, status bad parameter(0x3), syndrome (0x5add95), err(-22)
[ 5.787044] mlx5_core 0000:0b:00.1: mlx5_device_enable_sriov:82:(pid 814): failed to enable eswitch SRIOV (-22)
[ 5.787047] mlx5_core 0000:0b:00.1: mlx5_sriov_enable:168:(pid 814): mlx5_device_enable_sriov failed : -22
[ 5.787222] mlx5_core 0000:0b:00.1 mlxen0: renamed from enp11s0f1np1
[ 6.213460] mlx5_core 0000:0b:00.0 mlxib0: renamed from ib0
[ 7.054459] mlx5_core 0000:0b:00.1 mlxen0: Link up
[ 7.057046] 8021q: adding VLAN 0 to HW filter on device mlxen0
[ 8.055388] IPv6: ADDRCONF(NETDEV_CHANGE): mlxen0: link becomes ready
[ 8.284101] IPv6: ADDRCONF(NETDEV_CHANGE): mlxib0: link becomes ready
[ 8.557689] IPv6: ADDRCONF(NETDEV_CHANGE): mlxib0: link becomes ready
[ 8.580025] br1: port 1(mlxen0) entered blocking state
[ 8.580028] br1: port 1(mlxen0) entered disabled state
[ 8.580067] device mlxen0 entered promiscuous mode
[ 8.580957] br1: port 1(mlxen0) entered blocking state
[ 8.580960] br1: port 1(mlxen0) entered listening state
[ 8.595823] mlx5_core 0000:0b:00.1: mlx5e_fs_set_rx_mode_work:843:(pid 156): S-tagged traffic will be dropped while C-tag vlan stripping is enabled
[ 10.635172] br1: port 1(mlxen0) entered learning state
[ 25.931048] br1: port 1(mlxen0) entered forwarding state

Manually trying to add VFs to the ETH port results with the following error:

echo 0 > /sys/class/net/mlxen0/device/sriov_numvfs
echo 7 > /sys/class/net/mlxen0/device/sriov_numvfs
-bash: echo: write error: Invalid argument

The same works just fine for the IB port:

echo 0 > /sys/class/net/mlxib0/device/sriov_numvfs
echo 7 > /sys/class/net/mlxib0/device/sriov_numvfs

Hello.
I would like to draw your attention to the innovations of the upstream kernel and discuss the problem of intercepting the make_request_fn () function. I am concerned about a kernel 5.9 issue where intercepting the q->make_request_fn() function became impossible.

The problem is that several out-of-tree kernel modules can't run on it (only with a trick with x64_CR0_WP). First of all, this applies to backup tools. Many of them use their kernel modules to create snapshots of block devices. ASMFD will also not be able to connect to the block layer.

I have studied your kernel and see that these block layer changes are not present in your kernel. Not yet. But after some time, you will probably make a synchronization with the upstream kernel and the problem will appear to you.

I am trying to develop and offer to upstream kernel a solution to this problem https://github.com/SergeiShtepa/linux/tree/blk-filter-v0 . I believe that you may be interested in this issue and want to discuss it. I am sure that the Generic Block Device Filter module can be useful for Oracle.

While accessing SMB shares with Linux SMB kernel client, If the number of SMB responses from server with status of STATUS_IO_TIMEOUT exceeds a specified threshold (NUM_STATUS_IO_TIMEOUT), we reconnect the connection. But we do not return the mid, or the credits returned for the mid, or reduce the number of in-flight requests. This bug could result in the server->in_flight count to go bad, and also cause a leak in the mids. Which results in undesired behavior by SMB client as flow control of SMB client depends on credits.

Bug has been fixed in mainline kernel, marked for stable kernels and also has been backported to all major distros.

Link to mainline commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=69cba9d3c1284e0838ae408830a02c4a063104bc

Was found in the Linux kernels implementation (UEK6) of reading SVC RDMA counters. Reading the counter sysctl panics the system. This allows a local attacker with local access ot be able to create a denial of service while the system reboots.

The panic log is pasted below:

[ 54.696004] BUG: unable to handle page fault for address: 00005633bd69cd50
[ 54.696166] #PF: supervisor write access in kernel mode
[ 54.696321] #PF: error_code(0x0003) - permissions violation
[ 54.696481] PGD 438f0d067 P4D 438f0d067 PUD 41656a067 PMD 437e7c067 PTE 80000003ea5e9867
[ 54.696665] Oops: 0003 [#1] SMP NOPTI
[ 54.696844] CPU: 13 PID: 3918 Comm: sysctl Kdump: loaded Tainted: P OE 5.4.17-2102.203.6.el8uek.x86_64 #2
[ 54.697047] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014
[ 54.697278] RIP: 0010:memcpy_erms+0x6/0x9
[ 54.697486] Code: ff ff ff 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[ 54.697964] RSP: 0018:ffffc1bacb323dd0 EFLAGS: 00010297
[ 54.698208] RAX: 00005633bd69cd50 RBX: 0000000000000002 RCX: 0000000000000002
[ 54.698462] RDX: 0000000000000002 RSI: ffffc1bacb323ddf RDI: 00005633bd69cd50
[ 54.698731] RBP: ffffc1bacb323e18 R08: ffffc1bacb323ee8 R09: 0000000000000000
[ 54.698992] R10: 0000000000000000 R11: ffffc1bacb323de0 R12: ffffc1bacb323ee8
[ 54.699257] R13: ffffc1bacb323e38 R14: 00005633bd69cd50 R15: 0000000000000002
[ 54.699529] FS: 00007f7c04190940(0000) GS:ffff9d122f940000(0000) knlGS:0000000000000000
[ 54.699817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 54.700102] CR2: 00005633bd69cd50 CR3: 0000000437e76003 CR4: 0000000000360ee0
[ 54.700401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 54.700712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 54.701015] Call Trace:
[ 54.701330] ? svcrdma_counter_handler+0xbe/0x10c [rpcrdma]
[ 54.701667] proc_sys_call_handler+0x1a0/0x1ad
[ 54.702057] proc_sys_read+0x11/0x13
[ 54.702428] __vfs_read+0x1b/0x34
[ 54.702778] vfs_read+0x99/0x152
[ 54.703109] ksys_read+0x61/0xd2
[ 54.703445] __x64_sys_read+0x1a/0x1c
[ 54.703803] do_syscall_64+0x60/0x1cb
[ 54.704155] entry_SYSCALL_64_after_hwframe+0x170/0x0
[ 54.704498] RIP: 0033:0x7f7c035305b5
[ 54.704857] Code: fe ff ff 50 48 8d 3d 82 f7 09 00 e8 85 fe 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 e5 6f 2d 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 54.705593] RSP: 002b:00007ffd1cdbfc38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 54.705984] RAX: ffffffffffffffda RBX: 00005633bd6b33a0 RCX: 00007f7c035305b5
[ 54.706373] RDX: 0000000000002000 RSI: 00005633bd69cd50 RDI: 0000000000000006
[ 54.706774] RBP: 0000000000000d68 R08: 00005633bd69ed50 R09: 0000000000000003
[ 54.707166] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000002000
[ 54.707560] R13: 00005633bd69ed60 R14: 0000000000000000 R15: 0000000000000000
[ 54.707966] Modules linked in: binfmt_misc dm_mod vhost_net vhost vhost_iotlb tap xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_tables nfnetlink tun bridge rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm openvswitch 8021q garp mrp nf_conncount stp nf_nat llc nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr intel_rapl_common isst_if_common sunrpc i40iw ib_uverbs skx_edac nfit ib_core libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt iTCO_vendor_support kvm ipmi_ssif irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate acpi_ipmi mei_me intel_uncore ioatdma ipmi_si i2c_i801 pcspkr mei joydev lpc_ich dca wmi ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad xfs libcrc32c sd_mod t10_pi sg ast i2c_algo_bit drm_vram_helper

This is fixed by upstream commit: 3292739 sysctl: pass kernel pointers to ->proc_handler

The vhci-hcd driver is missing in the x86_64 version of package kernel-uek-modules-extra
This means that usbip bind fails...
Note that the aarch64 version does include both usbip-host.ko.xz and vhci-hcd.ko.xz

It seems to effect both OL8 and OL8 (UEK)
https://pkgs.org/download/kernel-uek-modules-extra

You can quickly check this as well using
ls /lib/modules/$(uname -a | cut -d' ' -f3)/kernel/drivers/usb/usbip

I have 2 machines, one with OL7 and one with OL9. They act as gateway to the Internet and do IPV4 and IPV6 NAT.

On these 2 machines I have massive WARNINGs (every few seconds) since 5.15.0-100.96.32.el9uek.x86_64 (on OL9) and 5.4.17-2136.318.7.1.el7uek.x86_64 (on OL7).

On other machines, which do no NAT, I haven't seen these WARNINGs. Going back to kernels 5.15.0-8.91.4.1.el9uek.x86_64 on OL9 and 5.4.17-2136.317.5.3.el7uek.x86_64 on OL7 fixes the problem.

Here are the WARNINGs form OL7:

Apr 18 08:58:32 stahl2 kernel: [ 236.276791] ------------[ cut here ]------------
Apr 18 08:58:32 stahl2 kernel: [ 236.278671] WARNING: CPU: 5 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xc5/0xd2
Apr 18 08:58:32 stahl2 kernel: [ 236.280615] Modules linked in: xt_multiport sctp pppoe tun pppox ppp_synctty ppp_async ppp_generic slhc ip6table_nat nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6_tables xt_nat iptable_nat nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 xt_pkttype xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter xt_TCPMSS iptable_mangle iptable_raw bochs_drm drm_vram_helper ttm drm_kms_helper drm nfsd syscopyarea pcspkr sysfillrect joydev sysimgblt qemu_fw_cfg virtio_balloon i6300esb i2c_piix4 fb_sys_fops auth_rpcgss nfs_acl lockd grace sunrpc nfs_ssc ip_tables ext4 mbcache jbd2 virtio_net net_failover virtio_blk failover ata_generic pata_acpi ata_piix libata virtio_pci serio_raw virtio_pci_legacy_dev virtio_pci_modern_dev
Apr 18 08:58:32 stahl2 kernel: [ 236.289602] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W 5.4.17-2136.318.7.1.el7uek.x86_64 #2
Apr 18 08:58:32 stahl2 kernel: [ 236.290896] Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
Apr 18 08:58:32 stahl2 kernel: [ 236.292308] RIP: 0010:sk_stream_kill_queues+0xc5/0xd2
Apr 18 08:58:32 stahl2 kernel: [ 236.293589] Code: 0e 48 89 df e8 9c f7 fe ff 8b b3 00 01 00 00 8b 83 48 01 00 00 85 c0 75 15 85 f6 75 0d 5b 41 5c 5d c3 cc cc cc cc 0f 0b eb bb <0f> 0b eb ef 0f 0b 0f 1f 44 00 00 eb e2 0f 1f 40 00 66 2e 0f 1f 84
Apr 18 08:58:32 stahl2 kernel: [ 236.296487] RSP: 0018:ffffb38b00180968 EFLAGS: 00010206
Apr 18 08:58:32 stahl2 kernel: [ 236.298069] RAX: 0000000000000000 RBX: ffff96dce98bc600 RCX: 0000000000000007
Apr 18 08:58:32 stahl2 kernel: [ 236.299451] RDX: 0000000000000001 RSI: 0000000000000d00 RDI: 0000000000000246
Apr 18 08:58:32 stahl2 kernel: [ 236.300986] RBP: ffffb38b00180978 R08: ffff96dcefd217a4 R09: 0000000000000004
Apr 18 08:58:32 stahl2 kernel: [ 236.302365] R10: 0000000000000000 R11: 00000000c7684925 R12: ffff96dce98bc6d0
Apr 18 08:58:32 stahl2 kernel: [ 236.303829] R13: ffff96dcdf0de476 R14: 0000000000000000 R15: ffff96dcdf0de476
Apr 18 08:58:32 stahl2 kernel: [ 236.305298] FS: 0000000000000000(0000) GS:ffff96dcf9d40000(0000) knlGS:0000000000000000
Apr 18 08:58:32 stahl2 kernel: [ 236.306850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 08:58:32 stahl2 kernel: [ 236.308471] CR2: 00007f27f727c000 CR3: 00000001a5adc000 CR4: 00000000000006e0
Apr 18 08:58:32 stahl2 kernel: [ 236.309943] Call Trace:
Apr 18 08:58:32 stahl2 kernel: [ 236.311350]
Apr 18 08:58:32 stahl2 kernel: [ 236.312821] inet_csk_destroy_sock+0x59/0x13f
Apr 18 08:58:32 stahl2 kernel: [ 236.314321] tcp_done+0x8a/0x111
Apr 18 08:58:32 stahl2 kernel: [ 236.315816] tcp_rcv_state_process+0x93b/0xdcc
Apr 18 08:58:32 stahl2 kernel: [ 236.317407] ? __skb_clone+0x2e/0x125
Apr 18 08:58:32 stahl2 kernel: [ 236.318919] tcp_v6_do_rcv+0x1b9/0x430
Apr 18 08:58:32 stahl2 kernel: [ 236.320375] tcp_v6_rcv+0x9ba/0xa25
Apr 18 08:58:32 stahl2 kernel: [ 236.321817] ? nf_confirm+0xb6/0x110 [nf_conntrack]
Apr 18 08:58:32 stahl2 kernel: [ 236.323301] ip6_protocol_deliver_rcu+0xd2/0x4c9
Apr 18 08:58:32 stahl2 kernel: [ 236.324716] ip6_input+0x41/0xb8
Apr 18 08:58:32 stahl2 kernel: [ 236.326090] ? ip6_protocol_deliver_rcu+0x4d0/0x4c9
Apr 18 08:58:32 stahl2 kernel: [ 236.327403] ip6_sublist_rcv_finish+0x59/0x70
Apr 18 08:58:32 stahl2 kernel: [ 236.328714] ip6_sublist_rcv+0x14a/0x1c9
Apr 18 08:58:32 stahl2 kernel: [ 236.330046] ? ip6_sublist_rcv+0x1d0/0x1c9
Apr 18 08:58:32 stahl2 kernel: [ 236.331355] ipv6_list_rcv+0x146/0x16d
Apr 18 08:58:32 stahl2 kernel: [ 236.332648] __netif_receive_skb_list_core+0x1ad/0x2e5
Apr 18 08:58:32 stahl2 kernel: [ 236.333969] netif_receive_skb_list_internal+0x1ca/0x2dc
Apr 18 08:58:32 stahl2 kernel: [ 236.335280] gro_normal_list.part.135+0x1e/0x3f
Apr 18 08:58:32 stahl2 kernel: [ 236.336594] napi_complete_done+0xd1/0x117
Apr 18 08:58:32 stahl2 kernel: [ 236.338153] virtnet_poll+0x363/0x480 [virtio_net]
Apr 18 08:58:32 stahl2 kernel: [ 236.339545] net_rx_action+0x28d/0x3f7
Apr 18 08:58:32 stahl2 kernel: [ 236.341018] __do_softirq+0xe1/0x2bc
Apr 18 08:58:32 stahl2 kernel: [ 236.342467] irq_exit+0xf5/0xfa
Apr 18 08:58:32 stahl2 kernel: [ 236.344039] do_IRQ+0x5a/0xe8
Apr 18 08:58:32 stahl2 kernel: [ 236.345424] common_interrupt+0xf/0x1d2
Apr 18 08:58:32 stahl2 kernel: [ 236.346823]
Apr 18 08:58:32 stahl2 kernel: [ 236.348283] RIP: 0010:native_safe_halt+0x12/0x18
Apr 18 08:58:32 stahl2 kernel: [ 236.349654] Code: 80 48 02 20 48 8b 00 a8 08 0f 84 60 ff ff ff eb b6 cc cc cc cc cc cc cc 55 48 89 e5 e9 07 00 00 00 0f 00 2d 92 6c 5a 00 fb f4 <5d> c3 cc cc cc cc 0f 1f 84 00 00 00 00 00 55 48 89 e5 e9 07 00 00
Apr 18 08:58:32 stahl2 kernel: [ 236.352557] RSP: 0018:ffffb38b0009be68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffda
Apr 18 08:58:32 stahl2 kernel: [ 236.354185] RAX: ffffffff93c65cd0 RBX: ffff96dc47e396c0 RCX: 0000000000000001
Apr 18 08:58:32 stahl2 kernel: [ 236.355483] RDX: ffff96dcf9d71140 RSI: ffffb38b0009be50 RDI: 0000000000000000
Apr 18 08:58:32 stahl2 kernel: [ 236.356736] RBP: ffffb38b0009be68 R08: 00000000de9bd37a R09: ffffb38b00b6be98
Apr 18 08:58:32 stahl2 kernel: [ 236.358063] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
Apr 18 08:58:32 stahl2 kernel: [ 236.359286] R13: 0000000000000000 R14: 0000000000000000 R15: ffff96dc47e396c0
Apr 18 08:58:32 stahl2 kernel: [ 236.360479] ? __sched_text_end+0x7/0x0
Apr 18 08:58:32 stahl2 kernel: [ 236.361709] default_idle+0x20/0x155
Apr 18 08:58:32 stahl2 kernel: [ 236.363034] arch_cpu_idle+0x15/0x1b
Apr 18 08:58:32 stahl2 kernel: [ 236.364197] default_idle_call+0x23/0x36
Apr 18 08:58:32 stahl2 kernel: [ 236.365412] do_idle+0x1a5/0x275
Apr 18 08:58:32 stahl2 kernel: [ 236.366447] cpu_startup_entry+0x1d/0x22
Apr 18 08:58:32 stahl2 kernel: [ 236.367477] start_secondary+0x176/0x1cc
Apr 18 08:58:32 stahl2 kernel: [ 236.368684] secondary_startup_64+0xb6/0xb6
Apr 18 08:58:32 stahl2 kernel: [ 236.369792] ---[ end trace 35af283ba7f2cd94 ]--

And here form OL9:

Apr 5 11:19:14 han kernel: ------------[ cut here ]------------
Apr 5 11:19:14 han kernel: WARNING: CPU: 2 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xd7/0xec
Apr 5 11:19:14 han kernel: Modules linked in: uas usb_storage tls tun lz4hc lz4hc_compress xt_multiport pppoe pppox ppp_generic bluetooth slhc ecdh_generic ecc 8021q garp mrp rfkill nft_chain_nat xt_MASQUERADE xt_nat ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_LOG nf_log_syslog ip_tables ip6_tables xt_pkttype xt_mark xt_connmark xt_conntrack xt_TCPMSS nft_counter xt_CT nft_compat nct6775 hwmon_vid snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec intel_rapl_msr intel_rapl_common snd_hda_core snd_hwdep edac_mce_amd snd_seq kvm_amd snd_seq_device snd_pcm kvm snd_timer snd irqbypass wmi_bmof pcspkr soundcore joydev i2c_piix4 k10temp gpio_amdpt gpio_generic acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sch_fq_codel sunrpc fuse ext4 mbcache jbd2 amdgpu crct10dif_pclmul crc32_pclmul sd_mod t10_pi drm_ttm_helper ttm sg gpu_sched i2c_algo_bit ghash_clmulni_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec aesni_intel ahci libahci
Apr 5 11:19:14 han kernel: crypto_simd cryptd r8169 drm sp5100_tco ccp libata realtek wmi video pinctrl_amd dm_mod nf_nat_sip nf_nat nf_conntrack_sip nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
Apr 5 11:19:14 han kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 5.15.0-100.96.32.el9uek.x86_64 #2
Apr 5 11:19:14 han kernel: Hardware name: To Be Filled By O.E.M. A320M-DVS R4.0/A320M-DVS R4.0, BIOS P7.20 05/10/2022
Apr 5 11:19:14 han kernel: RIP: 0010:sk_stream_kill_queues+0xd7/0xec
Apr 5 11:19:14 han kernel: Code: c0 89 c2 89 c6 89 c7 e9 97 05 67 00 48 89 df e8 5f e0 fe ff 8b 83 50 01 00 00 8b b3 00 01 00 00 85 c0 74 d5 0f 0b 85 f6 74 d3 <0f> 0b 5b 5d 31 c0 89 c2 89 c6 89 c7 e9 68 05 67 00 0f 0b eb 94 0f
Apr 5 11:19:14 han kernel: RSP: 0018:ffffb90280224c88 EFLAGS: 00010206
Apr 5 11:19:14 han kernel: RAX: 0000000000000000 RBX: ffff8dc1841eb340 RCX: 0000000000000000
Apr 5 11:19:14 han kernel: RDX: 0000000000000000 RSI: 0000000000000d00 RDI: 0000000000000000
Apr 5 11:19:14 han kernel: RBP: ffff8dc1841eb410 R08: 0000000000000000 R09: 0000000000000000
Apr 5 11:19:14 han kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8dc1841eb340
Apr 5 11:19:14 han kernel: R13: ffff8dc34917ce7e R14: 0000000000000000 R15: ffff8dc34917ce7e
Apr 5 11:19:14 han kernel: FS: 0000000000000000(0000) GS:ffff8dc4aea80000(0000) knlGS:0000000000000000
Apr 5 11:19:14 han kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 5 11:19:14 han kernel: CR2: 00007feffdd37b88 CR3: 0000000134bdc000 CR4: 00000000003506e0
Apr 5 11:19:14 han kernel: Call Trace:
Apr 5 11:19:14 han kernel:
Apr 5 11:19:14 han kernel: inet_csk_destroy_sock+0x55/0x113
Apr 5 11:19:14 han kernel: tcp_fin+0x123/0x19b
Apr 5 11:19:14 han kernel: tcp_data_queue+0x478/0x598
Apr 5 11:19:14 han kernel: tcp_rcv_state_process+0x2b9/0x7bc
Apr 5 11:19:14 han kernel: tcp_v6_do_rcv+0x1a1/0x496
Apr 5 11:19:14 han kernel: tcp_v6_rcv+0xd7d/0xdfa
Apr 5 11:19:14 han kernel: ? nf_ct_deliver_cached_events+0x7f/0xb0 [nf_conntrack]
Apr 5 11:19:14 han kernel: ? nf_confirm+0xd3/0x110 [nf_conntrack]
Apr 5 11:19:14 han kernel: ip6_protocol_deliver_rcu+0xcc/0x515
Apr 5 11:19:14 han kernel: ip6_input+0xad/0xb6
Apr 5 11:19:14 han kernel: ? ip6_protocol_deliver_rcu+0x520/0x515
Apr 5 11:19:14 han kernel: __netif_receive_skb_one_core+0x63/0xa1
Apr 5 11:19:14 han kernel: process_backlog+0x98/0x16c
Apr 5 11:19:14 han kernel: __napi_poll+0x2a/0x1a6
Apr 5 11:19:14 han kernel: net_rx_action+0x25e/0x316
Apr 5 11:19:14 han kernel: ? enqueue_hrtimer+0x2f/0x6e
Apr 5 11:19:14 han kernel: __do_softirq+0xd0/0x2a5
Apr 5 11:19:14 han kernel: __irq_exit_rcu+0xc7/0xf1
Apr 5 11:19:14 han kernel: common_interrupt+0x80/0x98
Apr 5 11:19:14 han kernel:
Apr 5 11:19:14 han kernel:
Apr 5 11:19:14 han kernel: asm_common_interrupt+0x22/0x27
Apr 5 11:19:14 han kernel: RIP: 0010:native_safe_halt+0xb/0x10
Apr 5 11:19:14 han kernel: Code: 1b 91 19 02 48 03 04 d5 00 4b 8b ad 0f b6 57 09 8b 74 d0 04 8b 3c d0 e9 33 4f 3e ff cc cc cc eb 07 0f 00 2d 07 af 58 00 fb f4 e0 9d 36 00 eb 07 0f 00 2d f7 ae 58 00 f4 e9 d1 9d 36 00 cc 0f
Apr 5 11:19:14 han kernel: RSP: 0018:ffffb902800dbe70 EFLAGS: 00000246
Apr 5 11:19:14 han kernel: RAX: 0000000000004000 RBX: 0000000000000001 RCX: 0000000000000000
Apr 5 11:19:14 han kernel: RDX: ffff8dc4aea80000 RSI: ffff8dc30ce52000 RDI: ffff8dc30ce52064
Apr 5 11:19:14 han kernel: RBP: ffff8dc30ce52064 R08: ffffffffae4f2340 R09: 0000000000000000
Apr 5 11:19:14 han kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Apr 5 11:19:14 han kernel: R13: ffffffffae4f23c0 R14: 0000000000000001 R15: 0000000000000000
Apr 5 11:19:14 han kernel: acpi_idle_do_entry+0x64/0x8d
Apr 5 11:19:14 han kernel: acpi_idle_enter+0x88/0xf7
Apr 5 11:19:14 han kernel: cpuidle_enter_state+0x89/0x35d
Apr 5 11:19:14 han kernel: cpuidle_enter+0x29/0x40
Apr 5 11:19:14 han kernel: cpuidle_idle_call+0x143/0x1de
Apr 5 11:19:14 han kernel: do_idle+0x81/0xd2
Apr 5 11:19:14 han kernel: cpu_startup_entry+0x19/0x1b
Apr 5 11:19:14 han kernel: secondary_startup_64_no_verify+0xc2/0x0
Apr 5 11:19:14 han kernel:
Apr 5 11:19:14 han kernel: ---[ end trace e2cdde453b50c9f5 ]---

Hi,
After last system update, my wi-fi adapter is missing.
nmtui in radio option tells me: wi-fi hardware: Missing Software: enabled.
When I load previours kernel on system start, wi-fi is present in the system. Also when I start Windows on this machine wi-fi is present on it.

System: Oracle Linux Server release 9.3

lspci:

Network controller: Intel Corporation Cannon Point-LP CNVi [Wireless-AC] (rev 30)
Ethernet controller: Intel Corporation Ethernet Connection (6) I219-LM (rev 30)

lshw -C network

  *-network:0 UNCLAIMED     
       description: Network controller
       product: Cannon Point-LP CNVi [Wireless-AC]
       vendor: Intel Corporation
       physical id: 14.3
       bus info: pci@0000:00:14.3
       version: 30
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix cap_list
       configuration: latency=0
       resources: memory:cc33c000-cc33ffff

kernels:

vmlinuz-5.14.0-284.30.0.1.el9_2.x86_64
vmlinuz-5.14.0-284.30.1.el9_2.x86_64
vmlinuz-5.15.0-106.131.4.el9uek.x86_64
vmlinuz-5.15.0-200.131.27.el9uek.x86_64
vmlinuz-5.14.0-362.8.1.el9_3.x86_64
vmlinuz-5.15.0-200.131.27.1.el9uek.x86_64

lspci -nn | grep -i net

00:14.3 Network controller [0280]: Intel Corporation Cannon Point-LP CNVi [Wireless-AC] [8086:9df0] (rev 30)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (6) I219-LM [8086:15bd] (rev 30)

grep -i 8086 /lib/modules/$(uname -r)/modules.* | grep -i 9df0
/lib/modules/5.15.0-200.131.27.1.el9uek.x86_64/modules.alias:alias pci:v00008086d00009DF0sv*sd*bc*sc*i* iwlwifi

grep -i 8086 /lib/modules/$(uname -r)/modules.* | grep -i 15bd
/lib/modules/5.15.0-200.131.27.1.el9uek.x86_64/modules.alias:alias pci:v00008086d000015BDsv*sd*bc*sc*i* e1000e

sudo modprobe iwlwifi
modprobe: ERROR: could not insert 'iwlwifi': Invalid argument

Hello Oracle,

Please pick up the following OCFS2-related bugfix:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9a934c7d88413a35861387a11e901554810b122

Description:

MADV_DOEXEC and MADV_DONTEXEC should be redefined within a separate numerical range otherwise break binary compatibility with mainline kernels.

Diagnostic Info:

Kernel v5.4 build from uek6/u3 has a commit a91ae4f, which had two madvise mode numbers defined as below:

#define MADV_DOEXEC	22		/* do inherit across exec */
#define MADV_DONTEXEC	23		/* don't inherit across exec */

In comparison, Linux mainline v5.4 does not have such definitions, the top number was 21, see v5.4 219d54332

Moving forward, Linux mainline started to use number 22 and 23, at its 5.14-rc1 timeframe, see torvalds/linux@4ca9b38.

#define MADV_POPULATE_READ	22	/* populate (prefault) page tables readable */
#define MADV_POPULATE_WRITE	23	/* populate (prefault) page tables writable */

According to Linux man-pages, the way to tell whether MADV_POPULATE_WRITE is supported on a testing system is:

MADV_POPULATE_WRITE (since Linux 5.14)
madvise(0, 0, advice) will return zero iff advice is supported by the kernel and can be relied on to probe for support.

As a result, when we do a syscall madvise(0, 0, 23) on UEKR6 v5.4.17 kernel will return 0 which means supported, while Linux v5.4 mainline returns -1 that means not-supported. The duplicate definition breaks the binary compatibility.

This issue is currently causing a practical failure on OpenJDK. See the ticket JDK-8324776 and discussion upon for details.

Other issue:

Kernel v5.15 on uek7/u2 has a similar problem. The commit 4693c5d integrated the defs of 22 and 23 from Linux mainline, while used 24 and 25 for the two customized mode numbers.

#define MADV_POPULATE_READ	22	/* populate (prefault) page tables readable */
#define MADV_POPULATE_WRITE	23	/* populate (prefault) page tables writable */
#define MADV_DOEXEC	24		/* do inherit across exec */
#define MADV_DONTEXEC	25		/* don't inherit across exec */

This created another incompatibility against Linux mainline's mode MADV_DONTNEED_LOCKED 24 introduced by torvalds/linux@9457056 since v5.18-rc1, and mode MADV_COLLAPSE 25 added by torvalds/linux@7d8faaf since v6.1-rc1.

See details at mainline b401b621:

#define MADV_POPULATE_READ	22	/* populate (prefault) page tables readable */
#define MADV_POPULATE_WRITE	23	/* populate (prefault) page tables writable */
#define MADV_DONTNEED_LOCKED	24	/* like DONTNEED, but drop locked pages too */
#define MADV_COLLAPSE	25		/* Synchronous hugepage collapse */

Proposed changes:

Redefine the customized modes MADV_DOEXEC and MADV_DONTEXEC within a separate numerical range, for example 101, 102.

As such it can avoid binary compatibility broken issues, UEK6 and UEK7 can also have same definitions of these two modes, and future UEKs do not need to move them to any new numbers, better for maintenance.

Any other similar or workable solution is also acceptable.

Lost wifi on Dell XPS 13 laptop after installing latest kernel on OL 9.2.

Offending kernel version: vmlinuz-5.15.0-200.131.27.el9uek.x86_64

Broken network adapter:
*-network
description: Wireless interface
product: Wireless-AC 9260
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:3a:00.0
logical name: wlp58s0
version: 29
serial: 34:13:e8:99:2b:c2
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
configuration: broadcast=yes driver=iwlwifi driverversion=5.15.0-106.131.4.el9uek.x86_64 firmware=46.ff18e32a.0 9260-th-b0-jf-b0- ip= latency=0 link=yes multicast=yes wireless=IEEE 802.11
resources: irq:16 memory:dc200000-dc203fff

Reverting to previous kernel fixed issue.

Hi,

I have for the last few days been troubleshooting issues with multiple OL8 servers - all OLVM nodes.

It looks like the latest UEK 5.15.x kernel breaks the nodes in multiple ways - I have created 2 forum posts regarding this with details

qla2xxx (fc) : https://forums.oracle.com/ords/apexds/post/olvm-ol8-warning-lastest-5-15-uek-kernel-breaks-fibre-chann-0376

OLVM network issues on bonded interfaces : https://forums.oracle.com/ords/apexds/post/olvm-ol8-x-help-some-nodes-we-cannot-do-any-network-changes-6280

Both these issues went away by either installing the 4.18.x kernel or an older 5.15 UEK kernel. I have seen the network issues on all OL8 node servers updates updated within the last 2 weeks, and the fc issue only on the latest uek kernel.

Description: /sbin/criu check is failing with "Error (criu/kerndat.c:1048): Unexpected error from clone3" on OracleLinux8.4

bash-4.4# /sbin/criu check --version
Version: 3.15
GitID: f9533f0

bash-4.4# /sbin/criu check --all
Error (criu/util.c:631): exited, status=1
Error (criu/util.c:631): exited, status=1
Error (criu/kerndat.c:1048): Unexpected error from clone3
: Invalid argument
Error (criu/kerndat.c:1181): kerndat_has_clone3_set_tid failed when initializing kerndat.
Error (criu/crtools.c:213): Could not initialize kernel features detection.
Error (criu/kerndat.c:1048): Unexpected error from clone3
: Invalid argument
Error (criu/kerndat.c:1181): kerndat_has_clone3_set_tid failed when initializing kerndat.
Error (criu/crtools.c:213): Could not initialize kernel features detection.
bash-4.4#

bash-4.4# cat /etc/oracle-release
Oracle Linux Server release 8.4

bash-4.4# uname -r
5.4.17-2102.203.6.el8uek.x86_64

I have opened bug for CRIU - checkpoint-restore/criu#1683

Its been updated in above bug that - An incomplete Backport of clone3 in the Oracle kernel.

Also bug - checkpoint-restore/criu#1592 is updated with below notes:

Originally I was a bit confused, actually the problem is in Oracle UEK v5.4 based kernel.
Due to commit d5f2322 which effectively breaks clone3 compatibility checks in kernel.
CRIU can't detect set_tid support right. So it's Oracle UEK v5.4 which should be fixed.