Host kernel panic with "gpio-pci-idio-16" sample about libvfio-user HOT 6 CLOSED

Noticing the following panic from host:

[ 207.412488] kernel BUG at drivers/vfio/vfio_iommu_type1.c:953!
[ 207.482338] invalid opcode: 0000 [#1] SMP PTI
[ 207.534494] CPU: 6 PID: 3739 Comm: qemu-system-x86 Tainted: G OE 5.4.0-rc8-upstream #1
[ 207.643845] Hardware name: Oracle Corporation ORACLE SERVER X6-2L/ASM,MOBO TRAY,2U, BIOS 39100000 09/22/2017
[ 207.761526] RIP: 0010:vfio_iommu_type1_ioctl+0x732/0xaac [vfio_iommu_type1]
[ 207.844868] Code: c0 0f 85 7b f9 ff ff c7 45 c0 18 00 00 00 e9 23 fc ff ff 48 c7 c0 ef ff ff ff eb 89 83 45 88 01 83 7d 88 0a 0f 8e 08 ff ff ff <0f> 0b e8 77 64 70 c0 48 8b 55 a8 31 c9 e9 b5 fa ff ff 0f 0b e9 5d
[ 208.069706] RSP: 0018:ffffc15c49d13d50 EFLAGS: 00010202
[ 208.132242] RAX: 0000000000000000 RBX: ffffc15c49d13d98 RCX: ffffa0ee2a6c0240
[ 208.217663] RDX: ffffa0dde7dcef40 RSI: 0000000080000000 RDI: 0000000080000000
[ 208.303085] RBP: ffffc15c49d13e00 R08: ffffa0de3f7bc1e0 R09: 0000000000000000
[ 208.388506] R10: 0000000000000000 R11: ffffffffffffff83 R12: 00007f7793351210
[ 208.473935] R13: 0000000000000000 R14: ffffa0de21b9c5c0 R15: ffffa0cf039ee9c0
[ 208.559362] FS: 00007f7793354700(0000) GS:ffffa0de3f180000(0000) knlGS:0000000000000000
[ 208.656228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 208.725037] CR2: 00007f778c3ce000 CR3: 0000000ff3db4001 CR4: 00000000003626e0
[ 208.810461] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 208.895887] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 208.981310] Call Trace:
[ 209.010594] ? kvm_vm_ioctl+0x329/0x9c0 [kvm]
[ 209.062731] ? lru_cache_add_active_or_unevictable+0x35/0xa0
[ 209.130467] vfio_fops_unl_ioctl+0x6d/0x260 [vfio]
[ 209.187806] do_vfs_ioctl+0xaa/0x600
[ 209.230581] ? __audit_syscall_entry+0xdd/0x130
[ 209.284801] ? handle_mm_fault+0xea/0x200
[ 209.332767] ksys_ioctl+0x67/0x90
[ 209.372412] __x64_sys_ioctl+0x1a/0x20
[ 209.417268] do_syscall_64+0x60/0x1c0
[ 209.461080] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 209.521552] RIP: 0033:0x7f781de87a57
[ 209.564322] Code: 44 00 00 48 8b 05 19 14 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e9 13 2d 00 f7 d8 64 89 01 48
[ 209.789162] RSP: 002b:00007f77933511f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 209.879785] RAX: ffffffffffffffda RBX: 000055d68f010010 RCX: 00007f781de87a57
[ 209.965209] RDX: 00007f7793351210 RSI: 0000000000003b72 RDI: 0000000000000013
[ 210.050630] RBP: 000055d68f010020 R08: 0000000000000000 R09: 0000000000000000
[ 210.136052] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7793351210
[ 210.221474] R13: 00007f7793351200 R14: 000055d68f010010 R15: 0000000000000000
[ 210.306895] Modules linked in: muser(OE) vfio_mdev mdev vfio_iommu_type1 vfio xt_REDIRECT xt_nat xt_CHECKSUM iptable_mangle xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge stp llc ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rpcrdma sunrpc rdma_ucm sb_edac intel_powerclamp ib_iser coretemp kvm_intel rdma_cm iw_cm ib_umad libiscsi ib_ipoib ib_cm scsi_transport_iscsi kvm mlx5_ib ib_uverbs irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ib_core ghash_clmulni_intel iTCO_wdt aesni_intel iTCO_vendor_support mxm_wmi ipmi_si ses enclosure scsi_transport_sas crypto_simd cdc_ether cryptd usbnet sg glue_helper mii ioatdma mei_me pcspkr mei i2c_i801 ipmi_devintf acpi_power_meter acpi_pad ipmi_msghandler lpc_ich wmi ip_tables xfs libcrc32c mlx5_core sd_mod mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
[ 210.306926] i2c_algo_bit drm_vram_helper ttm drm ixgbe ahci libahci libata nvme crc32c_intel nvme_core mlxfw mdio pci_hyperv_intf megaraid_sas ptp pps_core dca dm_mirror dm_region_hash dm_log dm_mod
[ 211.567352] ---[ end trace 4aa072ea23271ec1 ]---
[ 211.626580] RIP: 0010:vfio_iommu_type1_ioctl+0x732/0xaac [vfio_iommu_type1]
[ 211.710618] Code: c0 0f 85 7b f9 ff ff c7 45 c0 18 00 00 00 e9 23 fc ff ff 48 c7 c0 ef ff ff ff eb 89 83 45 88 01 83 7d 88 0a 0f 8e 08 ff ff ff <0f> 0b e8 77 64 70 c0 48 8b 55 a8 31 c9 e9 b5 fa ff ff 0f 0b e9 5d
[ 211.936903] RSP: 0018:ffffc15c49d13d50 EFLAGS: 00010202
[ 212.000177] RAX: 0000000000000000 RBX: ffffc15c49d13d98 RCX: ffffa0ee2a6c0240
[ 212.086382] RDX: ffffa0dde7dcef40 RSI: 0000000080000000 RDI: 0000000080000000
[ 212.172548] RBP: ffffc15c49d13e00 R08: ffffa0de3f7bc1e0 R09: 0000000000000000
[ 212.258718] R10: 0000000000000000 R11: ffffffffffffff83 R12: 00007f7793351210
[ 212.344835] R13: 0000000000000000 R14: ffffa0de21b9c5c0 R15: ffffa0cf039ee9c0
[ 212.430983] FS: 00007f7793354700(0000) GS:ffffa0de3f180000(0000) knlGS:0000000000000000
[ 212.528576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 212.598093] CR2: 00007f778c3ce000 CR3: 0000000ff3db4001 CR4: 00000000003626e0
[ 212.684273] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 212.770451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 212.856611] Kernel panic - not syncing: Fatal exception
[ 212.920090] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 213.045979] ---[ end Kernel panic - not syncing: Fatal exception ]---

from libvfio-user.

From code inspection the bug appears to be from the following in vfio_dma_do_unmap()

if (dma_last == dma) {
BUG_ON(++retries > 10);
} else {
dma_last = dma;
retries = 0;
}

from libvfio-user.

Are there any other messages in dmesg prior to the stack trace? Does gpio-pci-idio-16 print anything? Can you enable debug messages in libmuser and muser.ko and retest? In the mean time I'll try to reproduce it with the exact same commits. Looks like VFIO complains that some DMA area is not getting unmapped, which means libmuser/muser.ko aren't unmapping as requested. The kernel version we're using is v5.3.10 and QEMU is v4.1.0, both stable.

from libvfio-user.

Regarding QEMU v4.1.0-1750-g591b3bd, what's commit 591b3bd? Can you retest with something available upstream?

from libvfio-user.

Nevermind, I was able to reproduce this, even on v.5.3.10. Using 2G of RAM triggers it while using 1G does not.

from libvfio-user.

@swapnili found and fixed the problem, please use the latest version of muser.

from libvfio-user.