On Mon, Aug 22, 2022 at 9:30 AM Thomas Deutschmann whissi@whissi.de wrote:
Hi,
I can now reproduce using fio:
I looked around in MariaDB issue tracker and found https://jira.mariadb.org/browse/MDEV-26674 which lead me to https://github.com/MariaDB/server/commit/de7db5517de11a58d57d2a41d0bc6f38b6f... -- it's a conditional based on $KV and I hit that kernel regression during one of my bisect attempts (see https://lore.kernel.org/all/701f3fc0-2f0c-a32c-0d41-b489a9a59b99@whissi.de/).
Setting innodb_use_native_aio=OFF will prevent the problem.
This helped me to find https://github.com/axboe/fio/issues/1195 so I now have a working reproducer for fio.
$ cat reproducer.fio [global] direct=1 thread=1 norandommap=1 group_reporting=1 time_based=1 ioengine=io_uring
rw=randwrite bs=4096 runtime=20 numjobs=1 fixedbufs=1 hipri=1 registerfiles=1 sqthread_poll=1
[filename0] directory=/srv/machines/fio size=200M iodepth=1 cpus_allowed=20
...now call fio like "fio reproducer.fio". After one successful fio run, fsfreeze will already hang for me.
Hmm.. I still cannot repro the hang in my test. I have:
[root@eth50-1 ~]# mount | grep mnt /dev/md0 on /root/mnt type ext4 (rw,relatime,stripe=384) [root@eth50-1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom vda 253:0 0 32G 0 disk ├─vda1 253:1 0 2G 0 part /boot └─vda2 253:2 0 30G 0 part / nvme0n1 259:0 0 4G 0 disk └─md0 9:0 0 12G 0 raid5 /root/mnt nvme2n1 259:1 0 4G 0 disk └─md0 9:0 0 12G 0 raid5 /root/mnt nvme3n1 259:2 0 4G 0 disk └─md0 9:0 0 12G 0 raid5 /root/mnt nvme1n1 259:3 0 4G 0 disk └─md0 9:0 0 12G 0 raid5 /root/mnt
[root@eth50-1 ~]# history 381 fio iou/repro.fio 382 fsfreeze --freeze /root/mnt 383 fsfreeze --unfreeze /root/mnt 384 fio iou/repro.fio 385 fsfreeze --freeze /root/mnt 386 fsfreeze --unfreeze /root/mnt ^^^^^^^^^^^^^^ all works fine.
Did I miss something?
Thanks, Song