On Sunday 26 February 2012, Andrew Bradford wrote:
On Sun, Feb 26, 2012, at 09:50 AM, Arnd Bergmann wrote:
Ok. So 18MB is definitely an erase block boundary. The numbers here are not as slow as I had expected, but basically it does what's expected here.
I found that flashbench doesn't like erasesizes that aren't powers of 2, such as I can't do --erasesize=$[3*512*1024].
Yes, I've never managed to implement that. You can do it when you also set the blocksize to a multiple of three, as in
sudo ./flashbench /dev/sdc --open-au --erasesize=$[3* 512*1024] --blocksize=$[12*1024] --open-au-nr=9 --offset=$[18*1024*1024]
but that's not always ideal either, because small blocks will not be a multiple of the page size then.
That makes sense. Just for reference:
[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[3*512*1024] --blocksize=$[12*1024] --open-au-nr=9 --offset=$[18*1024*1024] 1.5MiB 11.3M/s 768KiB 12.6M/s 384KiB 12.4M/s 192KiB 11.8M/s 96KiB 10.9M/s 48KiB 9.24M/s 24KiB 6.24M/s 12KiB 3.07M/s
Ok, this is at least a good reference point to compare the other measurements to.
One thing I usually try when I'm not entirely sure about the erase block size is to play a bit more with the offset. You have established that 18MB is a boundary, as it normally is for these 3-bit-per-cell chips. If the erase size is 1.5MB, the next boundary is at 19.5MB, while for a 6 MB erase size, it would be at 24 MB.
The offsets that I would try therefore are
19MB (should be fast)
[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 11.9M/s 256KiB 11.7M/s 128KiB 11M/s 64KiB 9.63M/s 32KiB 8.27M/s 16KiB 6.21M/s 8KiB 3.3M/s
Oddly, when running that test, sometimes it has performance like that, and other times it has performance like this:
[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 4.92M/s 256KiB 4.84M/s 128KiB 4.72M/s 64KiB 4.58M/s 32KiB 4.17M/s 16KiB 3.56M/s 8KiB 2.4M/s
The best explanation I have for this is that in the second case it always does an erase somewhere in the process, which adds a fixed time overhead. That matches the observation that random writes are much slower than linear writes on average. Can you confirm that after doing an 'erase /dev/mmcblk0 0 $[32 * 1024 * 1024]', it is always fast?
The controller seems to be doing something, but I'm not really sure what. See further down this email for more info, it's not intermittent, I can make it happen repeatedly.
19.25MB (should be either like 17.75 or like 18.25, which you've both tried before)
[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024+256*1024] 512KiB 5.24M/s 256KiB 5.21M/s 128KiB 5.1M/s 64KiB 4.86M/s 32KiB 4.44M/s 16KiB 3.68M/s 8KiB 2.45M/s
It's like 17.75, not like 18.25. Would lead me to believe 1.5MiB is the real erase block size, rather than 6MiB.
right.
23.75 (should be like 17.75)
[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[23*1024*1024+768*1024] 512KiB 3.25M/s 256KiB 1.68M/s 128KiB 2.34M/s 64KiB 2.33M/s 32KiB 2.24M/s 16KiB 2.04M/s 8KiB 2.26M/s
Yes, same as 17.75.
An interesting set of steps that are repeatable where an 19MiB offset is fast, then slow, then fast. I think it's evident of the controller caching, but I'm not sure what it means:
I don't think that there is any significant caching. The controllers typically have a little bit (64 kb?) of SRAM, and the actual data is sometimes written to a temporary location in SLC flash before it gets written to permanent storage, but I would not call that caching in the strict sense.
19MiB offset is fast, like before: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 12M/s 256KiB 11.4M/s 128KiB 11.1M/s 64KiB 9.98M/s 32KiB 8.33M/s 16KiB 6.15M/s 8KiB 3.36M/s
18.5MiB offset is slower, but not slow: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024-512*1024] 512KiB 10.9M/s 256KiB 6M/s 128KiB 5.79M/s 64KiB 5.59M/s 32KiB 4.98M/s 16KiB 4.15M/s 8KiB 2.65M/s
Going back to 19MiB offset, fast till we get to 128KiB size: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 13M/s 256KiB 10.8M/s 128KiB 5.89M/s 64KiB 5.54M/s 32KiB 5.03M/s 16KiB 4.15M/s 8KiB 2.61M/s
And now 19MiB offset is the same speed as the 18.5MiB offset, and it will stay this way until I test at a different offset. Even leaving minutes between tests, as long as I don't read or write anywhere else on the card, it stays this slow: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 6M/s 256KiB 6.04M/s 128KiB 5.91M/s 64KiB 5.55M/s 32KiB 5.03M/s 16KiB 4.14M/s 8KiB 2.65M/s
But if I go test at 24MiB offset, the 512KiB size is slow, then the rest get fast (as expected): [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[24*1024*1024] 512KiB 2.41M/s 256KiB 11.7M/s 128KiB 10.8M/s 64KiB 9.93M/s 32KiB 8.23M/s 16KiB 6.2M/s 8KiB 3.36M/s
Ok, I see. The 512K number is low here because the card first needs to clean up the other erase block by writing it back into permanent MLC storage. After that is is fast.
And then going back to 19MiB offset, 512KiB is slow, but the rest are fast, as expected: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 4.98M/s 256KiB 9.89M/s 128KiB 9.32M/s 64KiB 8.79M/s 32KiB 7.5M/s 16KiB 5.54M/s 8KiB 3.23M/s
Same here, now it cleans up the segment at 24 MB in the first line, the continues as normal.
Any further testing at 19MiB offset stays fast, consistently: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 10.5M/s 256KiB 10.1M/s 128KiB 8.95M/s 64KiB 7.16M/s 32KiB 7.5M/s 16KiB 5.6M/s 8KiB 3.13M/s
I'm not sure what to make of this. Have you seen this type of behavior before? Repeating these steps, I can consistently get the same results.
The problem is that each line here writes only a partial erase block, so while the card has the chance to hide some of the garbage collection, it sometimes has to do it in the end. Since we are assuming that this card has a 1.5 MB erase block size, I would predict that you never see the slow first line when you move between 18 and 24 MB offset using --erasesize=$[3 * 512*1024], and that the effect would be less drastic if you use --erasesize=$[2*512*1024] (then it has to GC only 0.5MB instead of 1MB.
Arnd