snu-csl / nvmevirt Goto Github PK

I attempted to compile NVMeVirt with BASE_SSD set to ZNS_PROTOTYPE. I successfully compiled the program and initialized NVMe when I set NR_NAMESPACES to 1. However, I encountered compilation errors when NR_NAMESPACES was set to 2.

To configure the setup, I modified the Kbuild file by setting CONFIG_ZNS := y and uncommenting ccflags-$(CONFIG_NVMEVIRT_ZNS) += -DBASE_SSD=ZNS_PROTOTYPE. Additionally, I updated the ssd_config.h file to reflect the desired value for NR_NAMESPACES.

Are there any configurations that I may have overlooked?

The following lines contain the error messages I got:

make -C /lib/modules/5.15.0-107-generic/build M=/home/os/nvmevirt modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-107-generic'
  CC [M]  /home/os/nvmevirt/main.o
  CC [M]  /home/os/nvmevirt/pci.o
  CC [M]  /home/os/nvmevirt/admin.o
  CC [M]  /home/os/nvmevirt/io.o
  CC [M]  /home/os/nvmevirt/dma.o
  CC [M]  /home/os/nvmevirt/ssd.o
  CC [M]  /home/os/nvmevirt/zns_ftl.o
  CC [M]  /home/os/nvmevirt/zns_read_write.o
  CC [M]  /home/os/nvmevirt/zns_mgmt_send.o
  CC [M]  /home/os/nvmevirt/zns_mgmt_recv.o
  CC [M]  /home/os/nvmevirt/channel_model.o
  LD [M]  /home/os/nvmevirt/nvmev.o
  MODPOST /home/os/nvmevirt/Module.symvers
ERROR: modpost: "kv_init_namespace" [/home/os/nvmevirt/nvmev.ko] undefined!
ERROR: modpost: "simple_init_namespace" [/home/os/nvmevirt/nvmev.ko] undefined!
ERROR: modpost: "conv_init_namespace" [/home/os/nvmevirt/nvmev.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:133: /home/os/nvmevirt/Module.symvers] Error 1
make[2]: *** Deleting file '/home/os/nvmevirt/Module.symvers'
make[1]: *** [Makefile:1830: modules] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-107-generic'
make: *** [Makefile:8: default] Error 2

Hi. I am trying to run Samsung’s OpenMPDK KVBench and sample programs (sample_code_async and sample_code_sync) on NVMeVirt’s KVSSD. I was able to configure nvmevirt to run as 10GB KVSSD on my physical machine running Ubuntu 22.04 (Ubuntu kernel v5.19.0-43-generic). Following is the output from dmesg:

After that I tried to use both kernel and userspace drivers from Samsung OpenMPDK/KVSSD (https://github.com/OpenMPDK/KVSSD) v1.6.0.
I use the pre-built userspace device driver (udd) found under KVSSD/PDK/core/lib/libkvnvmedd.a and compile SNIA KV API udd option. After that I try to setup SPDK environment as follows-
sudo KVSSD/PDK/core/tools/setup.sh
with following output-

dmesg output is as follows –

After that I use one thread to write 100 KV requests (8B key, 512B value) with queue depth=2 using sample_code_async binary-
sudo KVSSD/PDK/core/build/sample_code_async -d 0001:10:00.0 -n 100 -q 2 -o 1 -k 8 -v 512 -t 1

I get the same error when I try to use sample_code_sync program as well.
Following is dmesg log

Could you please help in debugging this issue?

The kernel device driver from OpenMPDK/KVSSD does not seem to support Linux kernel v5.15.x.

Could you please share the steps you followed to run KVBench with NVMeVirt’s KVSSD? Also, could you please share which driver (kernel or user-space) and its version you used while running KVBench?

Please let me know if you need any other details.

Thank you in advance.

Hello, when I tried to enable CONFIG_NVMEVIRT_ZNS := y in the Makefile file and compiled it, the following error occurred when inserting the compiled file into the kernel module:

My /etc/default/grub file settings are as follows:

May I ask what may be the reason? thank you

Hi!

I've been playing around with nvmevirt and wanted to create a small demo app using xnvme and nvmevirt. I can create a virtual device and see it through lsblk and sudo nvme list.

When I try to issue sudo xnvme enum, the execution blocks for a few seconds, then lists the device. When I then run sudo nvme list again, the output look like this:

The virtual device is still somehow there but not functional anymore. I was wondering whether anyone has experienced something like this before. Was also wondering if this issue is related to nvmevirt or xnvme.

Thanks!

When configuring Fio workload through job file, We want to measure the performance of latency and bandwidth of each thread according to the number of jobs while keeping the total number of inflight requests and io size fixed.

Below is an example of a fio job file.
----------- 1 thread ---------------------
[global]
filename=/dev/ng0n1
ioengine=io_uring_cmd
cmd_type=nvme
bs=4k
io_size=120g
numjobs=1
iodepth=64

[1]
rw=write
offset=0%
size=80%

----------- 2 thread ---------------------
[global]
filename=/dev/ng0n1
ioengine=io_uring_cmd
cmd_type=nvme
bs=4k
io_size=60g
numjobs=1
iodepth=32

[1]
rw=write
offset=0%
size=40%

[2]
rw=write
offset=40%
size=40%

In real SSD(Samsung 970 evo), as the number of threads increases, latency remains constant, and in the case of bandwidth, the bandwidth of one thread is divided evenly.

But in nvmevirt, as the number of threads increases, latency varies for each thread, and in the case of bandwidth, the total number of threads is not the same as the bandwidth when only one thread is running.

When using NVMevirt, what results are expected when an io request is made by multi-threads?

As a result, does nvmevirt guarantee dispatching using the round robin method, which is the basic arbitration, according to the nvme protocol?

Hi, I tested the NVMeVirt KVSSD with Samsung's openMPDK test_async sample code. There are some errors on the uNVMe side (uNVMe/driver/external/spdk-18.04.1/lib/nvme/nvme_pcie.c::1234).

The command line as follow.
sudo ./sample_code_async -d 0001:10:00.0 -n 100 -q 64 -o 1 -k 16 -v 4096

Using fio(with job file - # of session 8 / depth = 32 / bs = 256K / time base), this module generates kernel panic with below kernel message

[ 4892.002303] BUG: scheduling while atomic: nvmev_io_worker/21188/0x00000002
[ 4892.002409] CPU: 65 PID: 21188 Comm: nvmev_io_worker Tainted: G OE 6.2.5 #2
[ 4892.002413] Call Trace:
[ 4892.002415]
[ 4892.002419] dump_stack_lvl+0x48/0x70
[ 4892.002430] dump_stack+0x10/0x20
[ 4892.002432] __schedule_bug+0x64/0x80
[ 4892.002439] __schedule+0xef7/0x14c0
[ 4892.002446] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[ 4892.002452] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[ 4892.002454] schedule+0x68/0x110
[ 4892.002456] schedule_preempt_disabled+0x15/0x30
[ 4892.002459] __mutex_lock.constprop.0+0x6f4/0x770
[ 4892.002461] ? sysvec_apic_timer_interrupt+0x4e/0xb0
[ 4892.002464] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[ 4892.002466] __mutex_lock_slowpath+0x13/0x20
[ 4892.002467] mutex_lock+0x3e/0x50
[ 4892.002469] msi_domain_get_virq+0xf3/0x160
[ 4892.002475] nvmev_signal_irq+0x2d/0xa0 [nvmev]
[ 4892.002483] nvmev_io_worker+0x900/0xb10 [nvmev]
[ 4892.002489] ? __pfx_nvmev_io_worker+0x10/0x10 [nvmev]
[ 4892.002494] kthread+0xeb/0x120
[ 4892.002498] ? __pfx_kthread+0x10/0x10
[ 4892.002501] ret_from_fork+0x29/0x50
[ 4892.002506]

why this error occur during fio? Is some method to solve this problem?

Under ZNS type, when insmod module prompts: kernel BUG at /home/hello/tzh/nvmevirt/nvmevirt/zns_ftl.c:102!
This is because storage space starts from 1M offset (source code: config->storage_size = memmap_size - MB(1)), so the parameter memmap_size must be set to the original value + 1M.
In addition, due to the increase in memmap_size, the insmod kernel will detect whether the used memory space (memmap_start---memmap_start+memmap_size) is reserved space. Therefore, when configuring the reserved memory space, you must also configure at least 1M more memory.

Currently, rmmod does not work correctly.

Hello everyone,

While using NVMeVirt for comparison tests on ZNS, I've noticed that certain sequential read workloads on ZNS are much slower than the conventional SSD configuration, despite having the same NAND configuration in ssd_config.h.

Used fio workload:

[global]
ioengine=libaio
filename=/dev/nvme1n1

group_reporting
direct=1
fdatasync=1
overwrite=0
fallocate=none
runtime=10s

[seq1m_q8t1_read]
size=2G
io_size=2G
io_limit=2G

rw=read
iodepth=8
bs=1M
numjobs=1

Using local_clock(), I've narrowed the reason down to the zns_read() function consuming too much CPU cycles.
Under the specified fio workload, 99.9% percentile on zns_read() takes 80us while conv_read() only takes 681ns (115x difference). This seems to be an enough delay to fluctuate the actual I/O performance characteristics.

Adding an artificial udelay(80) under conv_read() indeed seems to bring the performance with those 2 inline with each other.

Is this a design fault in NVMeVirt, or would it be possible to fix this behavior?

Thanks in advance.

For using nvmevirt, I try to insert module by using same method.
nvmev.ko module is inserted well, but something problem come out.

Problem is that nvmevirt has just one queue...
I post kernel message. Plz help me

[ 368.914046] nvme nvme2: pci function 0001:10:00.0
[ 368.914087] ------------[ cut here ]------------
[ 368.914088] WARNING: CPU: 54 PID: 1074 at drivers/pci/msi/msi.h:121 pci_msi_setup_msi_irqs+0x37/0x50
[ 368.914097] Modules linked in: nvmev(OE+) intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common ipmi_ssif isst_if_mbox_msr isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp binfmt_misc kvm_intel nls_iso8859_1 kvm irqbypass rapl intel_cstate cmdlinepart spi_nor input_leds joydev mei_me iTCO_wdt mtd intel_pmc_bxt mei iTCO_vendor_support ioatdma dca intel_pch_thermal acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr nvme_fabrics efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ast i2c_algo_bit drm_shmem_helper drm_kms_helper rndis_host cdc_ether syscopyarea usbhid usbnet sysfillrect crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 mii
[ 368.914167] aesni_intel hid crypto_simd nvme sysimgblt spi_intel_pci drm i40e cryptd nvme_core vmd spi_intel ahci i2c_i801 xhci_pci libahci nvme_common i2c_smbus xhci_pci_renesas
[ 368.914183] CPU: 54 PID: 1074 Comm: kworker/54:2 Tainted: G OE 6.2.5 #2
[ 368.914186] Hardware name: Supermicro SYS-240P-TNRT/X12QCH+, BIOS 1.7 10/06/2023
[ 368.914188] Workqueue: events work_for_cpu_fn
[ 368.914196] RIP: 0010:pci_msi_setup_msi_irqs+0x37/0x50
[ 368.914199] Code: 85 c0 74 26 f6 40 28 01 74 20 55 89 f2 48 81 c7 d0 00 00 00 31 f6 48 89 e5 e8 35 c8 8f ff 5d 31 d2 31 f6 31 ff c3 cc cc cc cc <0f> 0b b8 ed ff ff ff 31 d2 31 f6 31 ff c3 cc cc cc cc 0f 1f 80 00
[ 368.914201] RSP: 0018:ffffb3761cd03c98 EFLAGS: 00010246
[ 368.914203] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 368.914205] RDX: 0000000000000011 RSI: 0000000000000001 RDI: ffff99068cd36000
[ 368.914207] RBP: ffffb3761cd03cf8 R08: 0000000000000000 R09: 0000000000000000
[ 368.914209] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 368.914210] R13: 0000000000000000 R14: 0000000000000001 R15: ffff99068cd36000
[ 368.914211] FS: 0000000000000000(0000) GS:ffff999623e00000(0000) knlGS:0000000000000000
[ 368.914213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 368.914214] CR2: 0000562fbcbd1078 CR3: 000000904818c002 CR4: 00000000007706e0
[ 368.914216] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 368.914217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 368.914219] PKRU: 55555554
[ 368.914220] Call Trace:
[ 368.914222]
[ 368.914223] ? __pci_enable_msix_range+0x355/0x560
[ 368.914229] pci_alloc_irq_vectors_affinity+0xbe/0x120
[ 368.914236] pci_alloc_irq_vectors+0x11/0x30
[ 368.914239] nvme_pci_enable+0x7b/0x5b0 [nvme]
[ 368.914250] nvme_probe+0x336/0x810 [nvme]
[ 368.914257] local_pci_probe+0x48/0xb0
[ 368.914262] work_for_cpu_fn+0x17/0x30
[ 368.914266] process_one_work+0x21c/0x440
[ 368.914270] worker_thread+0x200/0x3f0
[ 368.914273] ? __pfx_worker_thread+0x10/0x10
[ 368.914276] kthread+0xeb/0x120
[ 368.914281] ? __pfx_kthread+0x10/0x10
[ 368.914285] ret_from_fork+0x29/0x50
[ 368.914293]
[ 368.914294] ---[ end trace 0000000000000000 ]---
[ 368.914301] ------------[ cut here ]------------
[ 368.914301] WARNING: CPU: 54 PID: 1074 at drivers/pci/msi/msi.h:127 pci_msi_teardown_msi_irqs+0x22/0x60
[ 368.914305] Modules linked in: nvmev(OE+) intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common ipmi_ssif isst_if_mbox_msr isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp binfmt_misc kvm_intel nls_iso8859_1 kvm irqbypass rapl intel_cstate cmdlinepart spi_nor input_leds joydev mei_me iTCO_wdt mtd intel_pmc_bxt mei iTCO_vendor_support ioatdma dca intel_pch_thermal acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr nvme_fabrics efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ast i2c_algo_bit drm_shmem_helper drm_kms_helper rndis_host cdc_ether syscopyarea usbhid usbnet sysfillrect crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 mii
[ 368.914351] aesni_intel hid crypto_simd nvme sysimgblt spi_intel_pci drm i40e cryptd nvme_core vmd spi_intel ahci i2c_i801 xhci_pci libahci nvme_common i2c_smbus xhci_pci_renesas
[ 368.914362] CPU: 54 PID: 1074 Comm: kworker/54:2 Tainted: G W OE 6.2.5 #2
[ 368.914365] Hardware name: Supermicro SYS-240P-TNRT/X12QCH+, BIOS 1.7 10/06/2023
[ 368.914366] Workqueue: events work_for_cpu_fn
[ 368.914369] RIP: 0010:pci_msi_teardown_msi_irqs+0x22/0x60
[ 368.914372] Code: 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 8b 87 f8 02 00 00 48 81 c7 d0 00 00 00 48 89 e5 48 85 c0 74 06 f6 40 28 01 75 20 <0f> 0b b9 ff ff 00 00 31 d2 31 f6 e8 7e bc 8f ff 5d 31 c0 31 d2 31
[ 368.914373] RSP: 0018:ffffb3761cd03c90 EFLAGS: 00010246
[ 368.914375] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 368.914376] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99068cd360d0
[ 368.914378] RBP: ffffb3761cd03c90 R08: 0000000000000000 R09: 0000000000000000
[ 368.914379] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 368.914380] R13: 0000000000000000 R14: 0000000000000001 R15: ffff99068cd36000
[ 368.914381] FS: 0000000000000000(0000) GS:ffff999623e00000(0000) knlGS:0000000000000000
[ 368.914382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 368.914384] CR2: 0000562fbcbd1078 CR3: 000000904818c002 CR4: 00000000007706e0
[ 368.914385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 368.914386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 368.914387] PKRU: 55555554
[ 368.914388] Call Trace:
[ 368.914389]
[ 368.914389] __pci_enable_msix_range+0x45c/0x560
[ 368.914392] pci_alloc_irq_vectors_affinity+0xbe/0x120
[ 368.914396] pci_alloc_irq_vectors+0x11/0x30
[ 368.914398] nvme_pci_enable+0x7b/0x5b0 [nvme]
[ 368.914405] nvme_probe+0x336/0x810 [nvme]
[ 368.914411] local_pci_probe+0x48/0xb0
[ 368.914413] work_for_cpu_fn+0x17/0x30
[ 368.914417] process_one_work+0x21c/0x440
[ 368.914420] worker_thread+0x200/0x3f0
[ 368.914423] ? __pfx_worker_thread+0x10/0x10
[ 368.914426] kthread+0xeb/0x120
[ 368.914429] ? __pfx_kthread+0x10/0x10
[ 368.914432] ret_from_fork+0x29/0x50
[ 368.914437]
[ 368.914438] ---[ end trace 0000000000000000 ]---
[ 368.916402] nvme nvme2: 1/0/0 default/read/poll queues

There are many unused functions in dma.c. Especially, prandom_bytes() incurrs a compile error in the latest 6.x kernels.

Hello, I can't manage to use nvmevirt because the Virtual NVMe device is removed after its creation by nvmevirt.
Here is the kernel log (sudo dmesg):

[  159.230037] nvmev: loading out-of-tree module taints kernel.
[  159.230088] nvmev: module verification failed: signature and/or required key missing - tainting kernel
[  159.230441] NVMeVirt: Storage : 100100000 + 3ff00000
[  159.231287] NVMeVirt: [NVMEV_NAMESPACE_INIT] ns=0 ns_addr=00000000507c09b5 ns_size=1023(MiB) 
[  159.231329] PCI host bridge to bus 0001:10
[  159.231331] pci_bus 0001:10: root bus resource [io  0x0000-0xffff]
[  159.231334] pci_bus 0001:10: root bus resource [mem 0x00000000-0xffffffffffff]
[  159.231335] pci_bus 0001:10: root bus resource [bus 00-ff]
[  159.231343] pci 0001:10:00.0: [0c51:0101] type 00 class 0x010802
[  159.231347] pci 0001:10:00.0: reg 0x10: [mem 0x100000000-0x100003fff 64bit]
[  159.231351] pci 0001:10:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[  159.231353] pci 0001:10:00.0: enabling Extended Tags
[  159.231409] pci 0001:10:00.0: Adding to iommu group 22
[  159.231751] NVMeVirt: Successfully created virtual PCI bus (node 0)
[  159.231993] NVMeVirt: nvmev_io_worker_0 started on cpu 4 (node 0)
[  159.232017] NVMeVirt: nvmev_dispatcher started on cpu 3 (node 0)
[  159.232119] nvme nvme1: pci function 0001:10:00.0
[  159.232142] NVMeVirt: Successfully created Virtual NVMe deivce
[  159.234347] BUG: unable to handle page fault for address: ffff8d2cffffe000
[  159.234352] #PF: supervisor read access in kernel mode
[  159.234355] #PF: error_code(0x0000) - not-present page
[  159.234358] PGD 80ce01067 P4D 80ce01067 PUD 0 
[  159.234363] Oops: 0000 [#1] SMP NOPTI
[  159.234367] CPU: 3 PID: 3521 Comm: nvmev_dispatche Tainted: G           OE     5.15.0-73-generic #80~20.04.1-Ubuntu
[  159.234372] Hardware name: Gigabyte Technology Co., Ltd. B550M DS3H/B550M DS3H, BIOS F16 11/09/2022
[  159.234374] RIP: 0010:nvmev_proc_admin_sq+0x7c/0xf20 [nvmev]
[  159.234384] Code: 89 f8 45 89 ce 48 8b 53 10 4d 63 da 45 89 d7 44 8b 63 0c 49 c1 eb 06 41 83 e7 3f 45 8d 42 01 4e 8b 2c da 49 c1 e7 06 4d 01 fd <41> 0f b6 55 00 80 fa 06 0f 84 75 06 00 00 0f 87 74 04 00 00 80 fa
[  159.234387] RSP: 0018:ffffb688c12ebe78 EFLAGS: 00010286
[  159.234391] RAX: ffff8d2e4c75e000 RBX: ffff8d2e4f0c8740 RCX: 0000000000000001
[  159.234393] RDX: ffff8d2e418be670 RSI: 0000000000000000 RDI: 0000000000000001
[  159.234396] RBP: ffffb688c12ebed0 R08: 0000000000000001 R09: 0000000000000000
[  159.234398] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  159.234400] R13: ffff8d2cffffe000 R14: 0000000000000000 R15: 0000000000000000
[  159.234402] FS:  0000000000000000(0000) GS:ffff8d351fac0000(0000) knlGS:0000000000000000
[  159.234405] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  159.234408] CR2: ffff8d2cffffe000 CR3: 000000015447e000 CR4: 0000000000750ee0
[  159.234410] PKRU: 55555554
[  159.234412] Call Trace:
[  159.234415]  <TASK>
[  159.234419]  nvmev_dispatcher+0x90/0x250 [nvmev]
[  159.234428]  ? __proc_file_read+0x240/0x240 [nvmev]
[  159.234434]  kthread+0x12a/0x150
[  159.234440]  ? set_kthread_struct+0x50/0x50
[  159.234444]  ret_from_fork+0x22/0x30
[  159.234451]  </TASK>
[  159.234452] Modules linked in: nvmev(OE) nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi edac_mce_amd nouveau snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi kvm_amd snd_hda_codec snd_hda_core snd_hwdep kvm snd_pcm mxm_wmi drm_ttm_helper crct10dif_pclmul ghash_clmulni_intel snd_seq_midi ttm aesni_intel snd_seq_midi_event crypto_simd cryptd joydev input_leds snd_rawmidi drm_kms_helper binfmt_misc wmi_bmof rapl cec snd_seq rc_core i2c_algo_bit snd_seq_device gigabyte_wmi fb_sys_fops snd_timer syscopyarea sysfillrect k10temp snd sysimgblt ccp video soundcore mac_hid sch_fq_codel msr parport_pc ppdev drm lp parport ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid r8169 nvme xhci_pci gpio_amdpt ahci crc32_pclmul i2c_piix4 nvme_core realtek libahci xhci_pci_renesas wmi gpio_generic
[  159.234537] CR2: ffff8d2cffffe000
[  159.234539] ---[ end trace ad6c13d79058f9ea ]---
[  159.234542] RIP: 0010:nvmev_proc_admin_sq+0x7c/0xf20 [nvmev]
[  159.234549] Code: 89 f8 45 89 ce 48 8b 53 10 4d 63 da 45 89 d7 44 8b 63 0c 49 c1 eb 06 41 83 e7 3f 45 8d 42 01 4e 8b 2c da 49 c1 e7 06 4d 01 fd <41> 0f b6 55 00 80 fa 06 0f 84 75 06 00 00 0f 87 74 04 00 00 80 fa
[  159.234552] RSP: 0018:ffffb688c12ebe78 EFLAGS: 00010286
[  159.234555] RAX: ffff8d2e4c75e000 RBX: ffff8d2e4f0c8740 RCX: 0000000000000001
[  159.234558] RDX: ffff8d2e418be670 RSI: 0000000000000000 RDI: 0000000000000001
[  159.234560] RBP: ffffb688c12ebed0 R08: 0000000000000001 R09: 0000000000000000
[  159.234562] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  159.234564] R13: ffff8d2cffffe000 R14: 0000000000000000 R15: 0000000000000000
[  159.234567] FS:  0000000000000000(0000) GS:ffff8d351fac0000(0000) knlGS:0000000000000000
[  159.234570] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  159.234572] CR2: ffff8d2cffffe000 CR3: 000000015447e000 CR4: 0000000000750ee0
[  159.234575] PKRU: 55555554
[  220.657584] nvme nvme1: I/O 8 QID 0 timeout, disable controller
[  220.765576] nvme nvme1: Device shutdown incomplete; abort shutdown
[  220.793639] nvme nvme1: Identify Controller failed (-4)
[  220.793642] nvme nvme1: Removing after probe failure status: -5

I followed the instructions from README (didn't even change Makefile configurations) and am using Ubuntu 20.04.1 with Linux kernel version 5.15.0-73-generic. I reserved 1G of memory from 4G (GRUB_CMDLINE_LINUX="memmap=1G\\\$4G").
As you can see, a Virtual NVMe device is successfully created but then removed after probe fails with a -5 status. insmod takes quite a long time (a little over a minute I think), and I found out that the nvme_dispatcher does not end properly.
I tried diverse memory mappings, but nothing worked. I even tried another version of Ubuntu (22.04 LTS - uses linux kernel version 5.19) but that didn't work either.

I would really appreciate it if anyone could help me.
Thank you!

The license has been changed to GPL v3. We need to change the License section at the end of the README.md file to reflect this change (both the license version and the associated link).

Hi,

This is really an amazing work! I'm wondering if CMB (controller memory buffer) supported in nvmevirt to perform peer-to-peer data transfer? I did not see this part in README, and it seems not supported from what lspci tells:

Latency: 0
NUMA node: 0
Region 0: Memory at 800000000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at <unassigned> [disabled] [size=2K]

If CMB not supported, how should I run peer-to-peer data transfer between virtual nvme devices?

Thanks!

Hello, does NVMeVirt (commit b262533) currently have SPDK support?
NVMeVirt works fine on my machines for the NVM, ZNS, and SSD configs when used in the kernel. However, whenever I unbind the virtual device from the kernel, I can not get it to bind to another driver (vfio, uio_pci_generic) or even the original nvme driver. It fails with the following messages (debugging enabled):

As a result the device can no longer be used.
To reproduce (change with relevant address):

echo 0001:10:00.0 | sudo tee /sys/bus/pci/devices/0001\:10\:00.0/driver/unbind
echo 0001:10:00.0 | sudo tee /sys/bus/pci/drivers/nvme/bind

or:

sudo ./spdk/scripts/setup.sh
sudo ./spdk/scripts/setup.sh reset

We have disabled intremap and make use of Linux kernel 5.19.2. Note that currently it is run from within QEMU (6.1.0) since we can not deploy it on the host machine.

Is this unbinding/binding a known issue? If so, is there a way to solve it or an alternative to use SPDK?
Thank you in advance!

When SSD performs GC during write-intensive workloads, it can fail to reclaim one free_line in one do_gc() invocation, leading to exhaust all free_lines. This can happen when the victim block contains many valid pages.
(Reported by [email protected] via email)

I tried to use it on WSL2, linux kernel version is 5.10.16.3-microsoft-standard-WSL2, but hit interrupt drop error:

The dmesg log is like:
__common_interrupt: 4.34 No irq handler for vector

Driver can receive the first two admin command and it seems the nvme driver can't receive interrupt of nvme0q0 and report command timeout finally.
The dmesg log is like this:

[ 2499.689475] WSL2: Performing memory compaction.
[ 2501.640750] nvme nvme0: I/O 8 QID 0 timeout, disable controller
[ 2502.140780] nvme nvme0: Identify Controller failed (-4)
[ 2502.140782] nvme nvme0: Removing after probe failure status: -5

Hello, I've been using nvmevirt recently to build SSD devices for some I/O API research. When using CONFIG_NVMEVIRT_SSD, testing a device with spdk_perf across multiple threads yields similar IOPS (consistent with the description from SPDK official documentation).

However, when I use nvmevirt to build with the CONFIG_NVMEVIRT_NVM option, the results from spdk_perf testing noticeably decrease as the number of threads increases, and I haven't found any relevant explanations.

Before reaching out to the SPDK community for assistance, I want to confirm if this is possibly due to the emulator configuration. Thanks again.

Hi. If I understood correctly, conventional and ZNS SSDs actually reflect the latency of NAND flash cells when they are added to "completed_time" in ssd_advance_nand() and passed to "nsecs_target" via "__insert_req_sorted", and finally added to an nvme_proc_table entry. However, there seems to be no such behavior in Optane SSDs or KVSSDs. It seems that the only things that are added are read/write time, delay, and trailing defined in main.c. Did I miss anything?
Also, for KVSSDs, in the paper it says that
"For KVSSD, the size of key-value pairs is small; hence its performance tends to be bound by the host-device interfacing performance and the key indexing time, rather than the performance of the storage media."
Does this mean that even with a proper backend for KVSSD (instead of the simple kv_ftl), the performance of KVSSD would not change?

dma currently doesn't have a finalization function to cleanup dma related structures,
This blocks from reusing the dma channel after rmmod since pre-allocated dma channel is not yet released.

JBD2: no valid journal superblock found
[ 449.510702] EXT4-fs (nvme1n1): error loading journal

I am using Ubuntu 20.04 under VMware, with kernel version 5.15-0-58. After compiling with make, insmod has no problems. However, after executing mkfs.ext4 /dev/nvme1n1, I encounter this error when attempting to mount.

Then I uninstalled nvme again and ran modprobe nvme, and the following error occurred.

[ 726.076392] nvme nvme1: I/O 0 QID 0 timeout, disable controller
[ 726.276816] nvme nvme1: Identify Controller failed (-4)
[ 726.276822] nvme nvme1: Removing after probe failure status: -5

I want to emulate multiple SSDs on my server and compile multiple kernel modules. However, it reports that "Unable to create PCI bus".

I wonder if NVMeVirt supports emulating multiple SSDs now. In FAST'23 paper, it seems that the feature is not ready yet. If it is enabled now, how to emulate multiple SSDs? Thanks.

In the following function, the return value can vary from 0 to 3
if the CELL_MODE has been defined as CELL_MODE_TLC in ssd_config.h.

static inline uint32_t get_cell(struct ssd *ssd, struct ppa *ppa)
{
	struct ssdparams *spp = &ssd->sp;
	return (ppa->g.pg / spp->pgs_per_flashpg) % (spp->cell_mode + 1);
}

In the code below, pg_4kb_rd_lat and pg_rd_lat are arrays with length 3 (which is from MAX_CELL_TYPES)
So when the return value of get_cell() is 3, it invokes out of range error.

uint64_t ssd_advance_nand(struct ssd *ssd, struct nand_cmd *ncmd)
{
	...
	spp = &ssd->sp;
	lun = get_lun(ssd, ppa);
	ch = get_ch(ssd, ppa);
	cell = get_cell(ssd, ppa);
	remaining = ncmd->xfer_size;

	switch (c) {
	case NAND_READ:
		/* read: perform NAND cmd first */
		nand_stime = max(lun->next_lun_avail_time, cmd_stime);

		if (ncmd->xfer_size == 4096) {
			nand_etime = nand_stime + spp->pg_4kb_rd_lat[cell];
		} else {
			nand_etime = nand_stime + spp->pg_rd_lat[cell];
		}
	...
}

I think the right way would be to divide by spp->cell_mode instead of spp->cell_mode + 1.

Does nvmevirt support zenfs？
When I run this command
./plugin/zenfs/util/zenfs mkfs --zbd= --aux_path=
it outpus
Failed to create file system, error: IO error: Input/output error

dmesg outputs
[28650.297992] nvme2n1: I/O Cmd(0x1) @ LBA 0, 2 blocks, I/O Error (sct 0x1 / sc 0xbc)
[28650.298003] I/O error, dev nvme2n1, sector 0 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0