Hello guys,
During the last few months, I felt a performance regression when using read() and write() on my high-speed Nvme SSD (about 7GB/s).
To get more precise information about it I quickly developed benchmark tool basically running read() or write() in a loop to simulate a sequential file read or write. The tool also measures the real time consumed by the loop. Finally, the tool can call open() with or without O_DIRECT.
I ran the tests on EXT4 and Exfat with following settings (buffer values have been set for best result): - Write settings: buffer 400mb * 100 - Read settings: buffer 200mb - Drop caches before non-direct read/write test
With this hardware: - CPU AMD Ryzen 7600X - RAM DDR5 5200 32GB - SSD Kingston Fury Renegade 4TB with 4K LBA
Here are some results I got with last upstream kernels (default config): +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | ~42GB | O_DIRECT | Linux 6.2.0 | Linux 6.3.0 | Linux 6.4.0 | Linux 6.5.0 | Linux 6.5.5 | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | Ext4 (sector 4k) | | | | | | | | Read | no | 7.2s (5800MB/s) | 7.1s (5890MB/s) | 8.3s (5050MB/s) | 13.2s (3180MB/s) | 13.2s (3180MB/s) | | Write | no | 12.0s (3500MB/s) | 12.6s (3340MB/s) | 12.2s (3440MB/s) | 28.9s (1450MB/s) | 28.9s (1450MB/s) | | Read | yes | 6.0s (7000MB/s) | 6.0s (7020MB/s) | 5.9s (7170MB/s) | 5.9s (7100MB/s) | 5.9s (7100MB/s) | | Write | yes | 6.7s (6220MB/s) | 6.7s (6290MB/s) | 6.9s (6080MB/s) | 6.9s (6080MB/s) | 6.9s (6970MB/s) | | Exfat (sector ?) | | | | | | | | Read | no | 7.3s (5770MB/s) | 7.2s (5830MB/s) | 9s (4620MB/s) | 13.3s (3150MB/s) | 13.2s (3180MB/s) | | Write | no | 8.3s (5040MB/s) | 8.9s (4750MB/s) | 8.3s (5040MB/s) | 18.3s (2290MB/s) | 18.5s (2260MB/s) | | Read | yes | 6.2s (6760MB/s) | 6.1s (6870MB/s) | 6.0s (6980MB/s) | 6.5s (6440MB/s) | 6.6s (6320MB/s) | | Write | yes | 16.1s (2610MB/s) | 16.0s (2620MB/s) | 18.7s (2240MB/s) | 34.1s (1230MB/s) | 34.5s (1220MB/s) | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+
Please note that I rounded some values to clarify readiness. Small variations can be considered as margin error.
Ext4 results: cached reads/writes time have increased of almost 100% from 6.2.0 to 6.5.0 with a first increase with 6.4.0. Direct access times have stayed similar though. Exfat results: performance decrease too with and without direct access this time.
I realize there are thousands of commits between, plus the issue can come from multiple kernel parts such as the page cache, the file system implementation (especially for Exfat), the IO engine, a driver, etc. The results also showed that there is not only a specific version impacted. Anyway, at the end the performance have highly decreased.
If you want to verify my benchmark tool source code, please ask.
PS: sending again as only text body is accepted
Regards
Florent DELAHAYE
On Sat, Sep 30, 2023 at 07:07:06PM +0200, Florent DELAHAYE wrote:
Hello guys,
During the last few months, I felt a performance regression when using read() and write() on my high-speed Nvme SSD (about 7GB/s).
To get more precise information about it I quickly developed benchmark tool basically running read() or write() in a loop to simulate a sequential file read or write. The tool also measures the real time consumed by the loop. Finally, the tool can call open() with or without O_DIRECT.
I ran the tests on EXT4 and Exfat with following settings (buffer values have been set for best result):
- Write settings: buffer 400mb * 100
- Read settings: buffer 200mb
- Drop caches before non-direct read/write test
With this hardware:
- CPU AMD Ryzen 7600X
- RAM DDR5 5200 32GB
- SSD Kingston Fury Renegade 4TB with 4K LBA
Here are some results I got with last upstream kernels (default config): +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | ~42GB | O_DIRECT | Linux 6.2.0 | Linux 6.3.0 | Linux 6.4.0 | Linux 6.5.0 | Linux 6.5.5 | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | Ext4 (sector 4k) | | | | | | | | Read | no | 7.2s (5800MB/s) | 7.1s (5890MB/s) | 8.3s (5050MB/s) | 13.2s (3180MB/s) | 13.2s (3180MB/s) | | Write | no | 12.0s (3500MB/s) | 12.6s (3340MB/s) | 12.2s (3440MB/s) | 28.9s (1450MB/s) | 28.9s (1450MB/s) | | Read | yes | 6.0s (7000MB/s) | 6.0s (7020MB/s) | 5.9s (7170MB/s) | 5.9s (7100MB/s) | 5.9s (7100MB/s) | | Write | yes | 6.7s (6220MB/s) | 6.7s (6290MB/s) | 6.9s (6080MB/s) | 6.9s (6080MB/s) | 6.9s (6970MB/s) | | Exfat (sector ?) | | | | | | | | Read | no | 7.3s (5770MB/s) | 7.2s (5830MB/s) | 9s (4620MB/s) | 13.3s (3150MB/s) | 13.2s (3180MB/s) | | Write | no | 8.3s (5040MB/s) | 8.9s (4750MB/s) | 8.3s (5040MB/s) | 18.3s (2290MB/s) | 18.5s (2260MB/s) | | Read | yes | 6.2s (6760MB/s) | 6.1s (6870MB/s) | 6.0s (6980MB/s) | 6.5s (6440MB/s) | 6.6s (6320MB/s) | | Write | yes | 16.1s (2610MB/s) | 16.0s (2620MB/s) | 18.7s (2240MB/s) | 34.1s (1230MB/s) | 34.5s (1220MB/s) | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+
Please note that I rounded some values to clarify readiness. Small variations can be considered as margin error.
Ext4 results: cached reads/writes time have increased of almost 100% from 6.2.0 to 6.5.0 with a first increase with 6.4.0. Direct access times have stayed similar though. Exfat results: performance decrease too with and without direct access this time.
I realize there are thousands of commits between, plus the issue can come from multiple kernel parts such as the page cache, the file system implementation (especially for Exfat), the IO engine, a driver, etc. The results also showed that there is not only a specific version impacted. Anyway, at the end the performance have highly decreased.
If you want to verify my benchmark tool source code, please ask.
Have you tried something like fio instead of a new benchmark tool? That way others can test and verify the results on their systems as that is a well-known and tested benchmark tool.
Also, are you sure you just haven't been hit by the spectre fixes that slow down the I/O path a lot? Be sure you have feature parity on those older kernels please. Many of the ones you list above do NOT have those required changes.
thanks,
greg k-h
Le samedi 30 septembre 2023 à 20:33 +0200, Greg KH a écrit :
On Sat, Sep 30, 2023 at 07:07:06PM +0200, Florent DELAHAYE wrote:
Hello guys,
During the last few months, I felt a performance regression when using read() and write() on my high-speed Nvme SSD (about 7GB/s).
To get more precise information about it I quickly developed benchmark tool basically running read() or write() in a loop to simulate a sequential file read or write. The tool also measures the real time consumed by the loop. Finally, the tool can call open() with or without O_DIRECT.
I ran the tests on EXT4 and Exfat with following settings (buffer values have been set for best result):
- Write settings: buffer 400mb * 100
- Read settings: buffer 200mb
- Drop caches before non-direct read/write test
With this hardware:
- CPU AMD Ryzen 7600X
- RAM DDR5 5200 32GB
- SSD Kingston Fury Renegade 4TB with 4K LBA
Here are some results I got with last upstream kernels (default config): +------------------+----------+------------------+----------------- -+-- ----------------+------------------+------------------+
~42GB | O_DIRECT | Linux 6.2.0 | Linux 6.3.0 |
Linux 6.4.0 | Linux 6.5.0 | Linux 6.5.5 | +------------------+----------+------------------+----------------- -+-- ----------------+------------------+------------------+
Ext4 (sector 4k) | | | | | | Read | no | 7.2s (5800MB/s) | 7.1s (5890MB/s) |
8.3s (5050MB/s) | 13.2s (3180MB/s) | 13.2s (3180MB/s) |
Write | no | 12.0s (3500MB/s) | 12.6s (3340MB/s) |
12.2s (3440MB/s) | 28.9s (1450MB/s) | 28.9s (1450MB/s) |
Read | yes | 6.0s (7000MB/s) | 6.0s (7020MB/s) |
5.9s (7170MB/s) | 5.9s (7100MB/s) | 5.9s (7100MB/s) |
Write | yes | 6.7s (6220MB/s) | 6.7s (6290MB/s) |
6.9s (6080MB/s) | 6.9s (6080MB/s) | 6.9s (6970MB/s) |
Exfat (sector ?) | | | | | | Read | no | 7.3s (5770MB/s) | 7.2s (5830MB/s) |
9s (4620MB/s) | 13.3s (3150MB/s) | 13.2s (3180MB/s) |
Write | no | 8.3s (5040MB/s) | 8.9s (4750MB/s) |
8.3s (5040MB/s) | 18.3s (2290MB/s) | 18.5s (2260MB/s) |
Read | yes | 6.2s (6760MB/s) | 6.1s (6870MB/s) |
6.0s (6980MB/s) | 6.5s (6440MB/s) | 6.6s (6320MB/s) |
Write | yes | 16.1s (2610MB/s) | 16.0s (2620MB/s) |
18.7s (2240MB/s) | 34.1s (1230MB/s) | 34.5s (1220MB/s) | +------------------+----------+------------------+----------------- -+-- ----------------+------------------+------------------+
Please note that I rounded some values to clarify readiness. Small variations can be considered as margin error.
Ext4 results: cached reads/writes time have increased of almost 100% from 6.2.0 to 6.5.0 with a first increase with 6.4.0. Direct access times have stayed similar though. Exfat results: performance decrease too with and without direct access this time.
I realize there are thousands of commits between, plus the issue can come from multiple kernel parts such as the page cache, the file system implementation (especially for Exfat), the IO engine, a driver, etc. The results also showed that there is not only a specific version impacted. Anyway, at the end the performance have highly decreased.
If you want to verify my benchmark tool source code, please ask.
Have you tried something like fio instead of a new benchmark tool? That way others can test and verify the results on their systems as that is a well-known and tested benchmark tool.
I understand. Yes I did and had similar results however I just ran it again to record the average results. I have run following fio commands:
write: fio --name=test-write --filename=testfio.out --readwrite=write - -blocksize=400m --size=41943040000 --ioengine=libaio --iodepth=2 (-- direct=1)
read: fio –name=test-read --filename=testfio.out --readwrite=read -- blocksize=200m --size=41943040000 --ioengine=libaio –iodepth=2 (-- direct=1)
See results below.
Also, are you sure you just haven't been hit by the spectre fixes that slow down the I/O path a lot? Be sure you have feature parity on those older kernels please. Many of the ones you list above do NOT have those required changes.
Indeed! mitigations=off clearly mitigated the performance loss, thank you! I thought it was not very relevant nowadays but it clearly still is especially from 6.4.0 to 6.5.0, however there is still a trend as you can see here (fio results with mitigations=off):
+-----------+----------+-----------------+------------------+---------- --------+------------------+ | ~42GB | O_DIRECT | Linux 6.2.0 | Linux 6.3.0 | Linux 6.4.0 | Linux 6.5.0 | +-----------+----------+-----------------+------------------+---------- --------+------------------+ | Ext4 (4k) | | | | | | | Read | no | 9.0s (4620MB/s) | 9.0s (4600MB/s) | 13.2s (3180MB/s) | 13.6s (3080MB/s) | | Write | no | 12s (3490MB/s) | 12.2s (3430MB/s) | 12.0s (3500MB/s) | 11.0s (3820MB/s) | | Read | yes | 5.9s (7070MB/s) | 5.9s (7070MB/s) | 5.8s (7200MB/s) | 5.8s (7200MB/s) | | Write | yes | 6.4s(6500MB/s) | 6.4s (6530MB/s) | 6.5s (6430MB/s) | 6.9s (6070MB/s) | +-----------+----------+-----------------+------------------+---------- --------+------------------+
-> So basically there are similar direct read/write timings, cached writes are roughly constant too however cached reads take more time (9s
13s, +~50%).
How to make sure I have feature parity across all kernels? I run a simple "make menuconfig" and exit with default options for compilation.
Regards
Florent DELAHAYE
linux-stable-mirror@lists.linaro.org
-
Florent DELAHAYE -
Greg KH