create filesystem fails for me

[xinglin]

Hi,

I started to play with this qemu-nvme ocssd simulator. I am able to create the pblk device but when I tried to create a filesystem on top of it, the kernel crashes. What did I do wrong? Thanks,

guest OS: debian 10, with kernel 5.2.0-rc5+, downloaded from github.

command to start qemu:
qemu-system-x86_64 -m 2g -hda mydisk.img -enable-kvm -cpu host -nographic -append "console=ttyS0 root=/dev/sda1" -kernel linux/arch/x86_64/boot/bzImage -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img -device nvme,drive=nvme01,serial=123456789,id=lnvm

xing@debian:~$ sudo nvme lnvm create -d nvme0n1 -n mydevice -t pblk -b 0 -e 7
[sudo] password for xing:
[   33.349066] pblk mydevice: Default OP insufficient, adjusting OP to 16
[   33.364653] pblk mydevice: luns:8, lines:60, secs:1630208, buf entries:512

xing@debian:~$ sudo fdisk -l
Disk /dev/nvme0n1: 8 GiB, 8589934592 bytes, 2097152 sectors
Disk model: QEMU NVMe OCSSD Ctrl
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/mydevice: 6.2 GiB, 6677331968 bytes, 1630208 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


xing@debian:~$ sudo mkfs -t ext4 /dev/mydevice
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done
Creating filesystem with 1630208 4k blocks and 408000 inodes
Filesystem UUID: b29146ad-e13c-42cb-a1d0-c1178c6c6f17
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: [   53.873897] WARNING:0
[   53.876577] Modules linked in:
[   53.877434] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc5+ #1
[   53.879151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14
[   53.881732] RIP: 0010:pblk_bio_free_pages+0x63/0x70
[   53.882814] Code: 48 c1 e3 04 49 c1 e5 04 49 8b 46 70 4c 89 e6 48 8b 3c 18 48 85
[   53.886861] RSP: 0018:ffffa90040003dc0 EFLAGS: 00010006
[   53.888006] RAX: 0000000000000008 RBX: 0000000000000003 RCX: 0000000000000005
[   53.889557] RDX: 0000000000000003 RSI: ffff9a3bba26d9c0 RDI: ffff9a3bb9235800
[   53.891112] RBP: ffffa90040003de0 R08: 0000000000000002 R09: 0000000000000000
[   53.892661] R10: ffffa90040003b78 R11: 0000000005a9f7cf R12: ffff9a3bbbe0bbd8
[   53.894210] R13: ffff9a3bb9235a90 R14: ffffa90040e67ce0 R15: ffff9a3bbbe0bb80
[   53.895765] FS:  0000000000000000(0000) GS:ffff9a3bbda00000(0000) knlGS:00000000
[   53.897520] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   53.898785] CR2: 00007f0a5cfd4000 CR3: 000000006ae68005 CR4: 00000000001606f0
[   53.900342] Call Trace:
[   53.900898]  <IRQ>
[   53.901360]  pblk_end_w_bio+0x9b/0xe0
[   53.902178]  pblk_end_io_write+0x14f/0x1d0
[   53.903095]  nvm_end_io+0x3d/0x40
[   53.903836]  nvme_nvm_end_io+0x2e/0x50
[   53.904675]  blk_mq_end_request+0xb3/0x130
[   53.905585]  nvme_complete_rq+0x8d/0x230
[   53.906468]  nvme_pci_complete_rq+0x4c/0xc0
[   53.907398]  blk_done_softirq+0x92/0xc0
[   53.908252]  __do_softirq+0xe4/0x2f3
[   53.909051]  irq_exit+0xb6/0xc0
[   53.909754]  do_IRQ+0x8a/0xe0
[   53.910431]  common_interrupt+0xf/0xf
[   53.911249]  </IRQ>
[   53.911727] RIP: 0010:native_safe_halt+0x12/0x20
[   53.912748] Code: 00 0f 00 2d c2 f3 59 00 f4 5d c3 0f 1f 00 66 2e 0f 1f 84 00 09
[   53.916805] RSP: 0018:ffffffffa9e03e10 EFLAGS: 00000246 ORIG_RAX: fffffffffffff8
[   53.918466] RAX: ffffffffa926aad0 RBX: 0000000000000000 RCX: 0000000000000001
[   53.920012] RDX: ffff9a3bbda2be00 RSI: 0000000c822e9656 RDI: 0000000000000000
[   53.921554] RBP: ffffffffa9e03e10 R08: 000000000001da40 R09: 0000000c95416656
[   53.923107] R10: ffff9a3bbffacca8 R11: 0000000000000000 R12: 0000000000000000
[   53.924661] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   53.926216]  ? __cpuidle_text_start+0x8/0x8
[   53.927150]  default_idle+0x22/0x150
[   53.927942]  arch_cpu_idle+0x15/0x20
[   53.928735]  default_idle_call+0x23/0x30
[   53.929603]  do_idle+0x1cb/0x280
[   53.930320]  cpu_startup_entry+0x1d/0x20
[   53.931198]  rest_init+0xae/0xb0
[   53.931921]  arch_call_rest_init+0xe/0x1b
[   53.932808]  start_kernel+0x501/0x520
[   53.933620]  x86_64_start_reservations+0x24/0x26
[   53.934641]  x86_64_start_kernel+0x74/0x77
[   53.935545]  secondary_startup_64+0xa4/0xb0
[   53.936467] ---[ end trace 6ff2439226ef9969 ]---
[   53.939978] WARNING: CPU: 0 PID: 0 at drivers/lightnvm/pblk-core.c:329 pblk_bio0
[   53.941983] Modules linked in:
[   53.942682] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W         5.2.0-rc51
[   53.944342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14
[   53.946785] RIP: 0010:pblk_bio_free_pages+0x63/0x70
[   53.947851] Code: 48 c1 e3 04 49 c1 e5 04 49 8b 46 70 4c 89 e6 48 8b 3c 18 48 85
[   53.951885] RSP: 0018:ffffa90040003dc0 EFLAGS: 00010006
[   53.953026] RAX: 0000000000000008 RBX: 0000000000000001 RCX: 0000000000000007
[   53.954583] RDX: 0000000000000001 RSI: ffff9a3bba26d9c0 RDI: ffff9a3bb9235800
[   53.956127] RBP: ffffa90040003de0 R08: 0000000000000002 R09: 0000000000000000
[   53.957675] R10: ffffa90040003b78 R11: 000000000000b738 R12: ffff9a3bbbe0bbd8
[   53.959238] R13: ffff9a3bb9235a90 R14: ffffa90040e67d28 R15: ffff9a3bbbe0bb80
[   53.960804] FS:  0000000000000000(0000) GS:ffff9a3bbda00000(0000) knlGS:00000000
[   53.962585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   53.963850] CR2: 00007f5c4da9b000 CR3: 0000000077d0a002 CR4: 00000000001606f0
[   53.965417] Call Trace:
[   53.965973]  <IRQ>
[   53.966448]  pblk_end_w_bio+0x9b/0xe0
[   53.967271]  pblk_end_io_write+0x14f/0x1d0
[   53.968187]  nvm_end_io+0x3d/0x40
[   53.968934]  nvme_nvm_end_io+0x2e/0x50
[   53.969771]  blk_mq_end_request+0xb3/0x130
[   53.970693]  nvme_complete_rq+0x8d/0x230
[   53.971582]  nvme_pci_complete_rq+0x4c/0xc0
[   53.972517]  blk_done_softirq+0x92/0xc0
[   53.973375]  __do_softirq+0xe4/0x2f3
[   53.974177]  irq_exit+0xb6/0xc0
[   53.974888]  do_IRQ+0x8a/0xe0
[   53.975563]  common_interrupt+0xf/0xf
[   53.976380]  </IRQ>
[   53.976862] RIP: 0010:native_safe_halt+0x12/0x20
[   53.977884] Code: 00 0f 00 2d c2 f3 59 00 f4 5d c3 0f 1f 00 66 2e 0f 1f 84 00 09
[   53.981975] RSP: 0018:ffffffffa9e03e10 EFLAGS: 00000246 ORIG_RAX: fffffffffffff8
[   53.983648] RAX: ffffffffa926aad0 RBX: 0000000000000000 RCX: 0000000000000001
[   53.985226] RDX: ffff9a3bbda2be00 RSI: ffffffffa9e03de0 RDI: 0000000000000000
[   53.986807] RBP: ffffffffa9e03e10 R08: 0000001003f93c5c R09: 0000000000000000
[   53.988387] R10: ffffa900414afd08 R11: 0000000000000000 R12: 0000000000000000
[   53.989962] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   53.991547]  ? __cpuidle_text_start+0x8/0x8
[   53.992483]  default_idle+0x22/0x150
[   53.993287]  arch_cpu_idle+0x15/0x20
[   53.994093]  default_idle_call+0x23/0x30
[   53.994980]  do_idle+0x1cb/0x280
[   53.995715]  cpu_startup_entry+0x1d/0x20
[   53.996589]  rest_init+0xae/0xb0
[   53.997320]  arch_call_rest_init+0xe/0x1b
[   53.998215]  start_kernel+0x501/0x520
[   53.999041]  x86_64_start_reservations+0x24/0x26
[   54.000065]  x86_64_start_kernel+0x74/0x77
[   54.000980]  secondary_startup_64+0xa4/0xb0
[   54.001917] ---[ end trace 6ff2439226ef996a ]---

[xinglin]

Doing simple IOs directly to the device using dd seems to work fine.

$ dd if=~/sometextfile.txt of=/dev/mydevice bs=1K count=50
$ dd if=/dev/mydevice bs=1K count=50 | hexdump -c

[birkelund]

Hi,

I cannot reproduce on latest (5.3.0-rc2). Could you try that?

[xinglin]

Hi,

I could not find 5.3.0-rc2 branch from https://github.com/OpenChannelSSD/linux. Where do I find that?

[minwooim]

@xinglin ,
Maybe @birkelund is talking about the kernel repository.
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

[birkelund]

I meant mainline (aka Linus' tree). But the for-5.3/core branch in OpenChannel/linux should be fine too.

[xinglin]

switched to use kernel 5.3.0-rc2 and this indeed solves the problem. I can create an ext4 on a pblk device and mount it. Maybe it would help if the below instructions could be updated, since there is no real need to use a custom kernel. The kernel from mainline just works. Thanks for the help!

Use a custom kernel
The development of the lightnvm subsystem is hosted here:

git clone https://github.com/OpenChannelSSD/linux.git

and the latest work is found in the for-4.XX/core branch.

[birkelund]

Thanks @xinglin, I've created a pull request with updated documentation.