On Saturday 25 February 2012, Andrew Bradford wrote:

On Thu, Feb 23, 2012, at 09:37 PM, Arnd Bergmann wrote:

...

On Monday 20 February 2012, Andrew Bradford wrote:

...

andrew@bigbox:~/flashbench$ sudo sfdisk -uS -l /dev/sdc

Disk /dev/sdc: 1890 cylinders, 128 heads, 32 sectors/track Units = sectors of 512 bytes, counting from 0

Device Boot    Start       End   #sectors  Id  System

/dev/sdc1 * 8192 7741439 7733248 c W95 FAT32 (LBA) /dev/sdc2 0 - 0 0 Empty /dev/sdc3 0 - 0 0 Empty /dev/sdc4 0 - 0 0 Empty

factor $[(7741439 + 1) / 2] 3870720: 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 5 7

This could be either 945 * 4 MB or 630 * 6 MB

This isn't the partition map that the card came with. I had to repartition it because I accidentally blew away the MBR. I would think that matters for your assessment.

Ah, well spotted. I normally use 'fdisk -l -u /dev/sdc' to find the exact size, and I had not realized that you used sfdisk on a somewhat addly partitioned device.

[andrew@mythdvr flashbench]$ sudo fdisk -l /dev/sdc | grep Disk Disk /dev/sdc: 3965 MB, 3965190144 bytes Disk identifier: 0x00000000 [andrew@mythdvr flashbench]$ factor 3965190144 3965190144: 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2521

That 2^18 in the factor, would that indicate 512KiB making 7563 512 KiB blocks? Or 2521 1.5MiB blocks?

The 1.5MiB blocks would make sense for SanDisk, right?

Yes, this sounds right.

Does partitioning really matter for running flashbench if I'm doing it on the entire device and not within a partition? I know the partitioning matters for aligning the partition to the underlying flash for use as a filesystem, but I can't easily see how it matters for flashbench other than to get an idea of the manufacturer's partitioning to see the erase block size.

No, the partition does not matter at all. Only the size matters here.

...

...

andrew@bigbox:~/flashbench$ sudo ./flashbench -a /dev/sdc --blocksize=1024 --count=100 ## 2MiB erase block ## 8KiB pages? hard to see

I would rerun this with --blocksize=3072, there might be more to see if in case this has a multiple of 3 in it.

[andrew@mythdvr flashbench]$ sudo ./flashbench -a /dev/sdc --blocksize=3072 --count=100 align 805306368 pre 1.05ms on 1.16ms post 1.1ms diff 87.9µs align 402653184 pre 1.06ms on 1.12ms post 1.03ms diff 74.7µs align 201326592 pre 997µs on 1.1ms post 1ms diff 96.5µs align 100663296 pre 1.12ms on 1.21ms post 1.1ms diff 97.1µs align 50331648 pre 1.15ms on 1.27ms post 1.15ms diff 124µs align 25165824 pre 1.13ms on 1.23ms post 1.14ms diff 92.6µs align 12582912 pre 1.13ms on 1.23ms post 1.13ms diff 98.4µs align 6291456 pre 1.13ms on 1.27ms post 1.13ms diff 136µs ## align 3145728 pre 1.14ms on 1.18ms post 1.15ms diff 35.1µs align 1572864 pre 1.13ms on 1.19ms post 1.12ms diff 64.3µs align 786432 pre 1.13ms on 1.16ms post 1.11ms diff 34.4µs align 393216 pre 1.15ms on 1.13ms post 1.13ms diff -6419ns align 196608 pre 1.13ms on 1.15ms post 1.12ms diff 22.6µs align 98304 pre 1.12ms on 1.15ms post 1.15ms diff 15.7µs align 49152 pre 1.16ms on 1.18ms post 1.18ms diff 10.4µs align 24576 pre 1.11ms on 1.1ms post 1.13ms diff -23212n align 12288 pre 1.17ms on 1.13ms post 1.16ms diff -26969n align 6144 pre 1.14ms on 1.12ms post 1.11ms diff -9644ns

There's the 6 MB.

Yes, that's what it looks like.

...

With the numbers like this, my guess is that it can in fact do 9 erase blocks of some other size, possibly as small as 512KB using the sandisk trick where they write all bits at once to turn an 1.5MB TLC block into a 0.5MB SLC block.

Maybe you can try playing with the offset to find the correct erase block size. If 512KB gives you the full 12MB/s in linear mode with open-au-nr=9, try misaligning this:

# natural alignment (start at default of 16MB) sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] \ --blocksize=65536 --open-au-nr=9

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[4*1024] --open-au-nr=1 512KiB 13.3M/s 256KiB 12.2M/s 128KiB 12.3M/s 64KiB 12.4M/s 32KiB 12.4M/s 16KiB 11.4M/s 8KiB 9.09M/s 4KiB 6.58M/s

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[4*1024] --open-au-nr=9 512KiB 11.6M/s 256KiB 11.7M/s 128KiB 11M/s 64KiB 9.77M/s 32KiB 8.25M/s 16KiB 6.2M/s 8KiB 3.36M/s 4KiB 1.52M/s

4KiB is about half the speed of 8KiB is about half the speed of 16KiB.

Would this be any indicator that the page size is 16KiB?

Yes, exactly

With erasesize of 512KiB, 10 open-au is still very slow.

Right, that was expected.

...

# offset by one erase block, should still be fast sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] \ --blocksize=65536 --open-au-nr=9 --offset=$[18 * 1024 * 1024 + 512 * 1024]

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[18*1024*1024+512*1024] 512KiB 12.2M/s 256KiB 11M/s 128KiB 9.98M/s 64KiB 9.87M/s 32KiB 7.62M/s 16KiB 6.16M/s 8KiB 3.26M/s

Ok, no difference to other offset. good.

...

# offset by half an erase block, only fast if actual block size is larger than 512KB sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] \ --blocksize=65536 --open-au-nr=9 --offset=$[18 * 1024 * 1024 + 256 * 1024]

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[18*1024*1024+256*1024] 512KiB 12M/s 256KiB 7.93M/s 128KiB 7.46M/s 64KiB 7.11M/s 32KiB 6.25M/s 16KiB 4.89M/s 8KiB 2.95M/s

Reasonably fast, especially compared to the next test.

It is noticeably slower than the one before though, which indicates that it's not all the same the drive.

...

# offset by half an erase block backwards, should be slow sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] \ --blocksize=65536 --open-au-nr=9 --offset=$[18 * 1024 * 1024 - 256 * 1024]

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[18*1024*1024-256*1024] 512KiB 5.77M/s 256KiB 2.32M/s 128KiB 2.28M/s 64KiB 2.26M/s 32KiB 2.16M/s 16KiB 1.99M/s 8KiB 2.19M/s

Ok. So 18MB is definitely an erase block boundary. The numbers here are not as slow as I had expected, but basically it does what's expected here.

I found that flashbench doesn't like erasesizes that aren't powers of 2, such as I can't do --erasesize=$[3*512*1024].

Yes, I've never managed to implement that. You can do it when you also set the blocksize to a multiple of three, as in

sudo ./flashbench /dev/sdc --open-au --erasesize=$[3* 512*1024] --blocksize=$[12*1024] --open-au-nr=9 --offset=$[18*1024*1024]

but that's not always ideal either, because small blocks will not be a multiple of the page size then.

One thing I usually try when I'm not entirely sure about the erase block size is to play a bit more with the offset. You have established that 18MB is a boundary, as it normally is for these 3-bit-per-cell chips. If the erase size is 1.5MB, the next boundary is at 19.5MB, while for a 6 MB erase size, it would be at 24 MB.

The offsets that I would try therefore are

19MB (should be fast) 19.25MB (should be either like 17.75 or like 18.25, which you've both tried before) 23.75 (should be like 17.75)

Arnd

On Sunday 26 February 2012, Andrew Bradford wrote:

On Sun, Feb 26, 2012, at 09:50 AM, Arnd Bergmann wrote:

...

Ok. So 18MB is definitely an erase block boundary. The numbers here are not as slow as I had expected, but basically it does what's expected here.

...

I found that flashbench doesn't like erasesizes that aren't powers of 2, such as I can't do --erasesize=$[3*512*1024].

Yes, I've never managed to implement that. You can do it when you also set the blocksize to a multiple of three, as in

sudo ./flashbench /dev/sdc --open-au --erasesize=$[3* 512*1024] --blocksize=$[12*1024] --open-au-nr=9 --offset=$[18*1024*1024]

but that's not always ideal either, because small blocks will not be a multiple of the page size then.

That makes sense. Just for reference:

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[3*512*1024] --blocksize=$[12*1024] --open-au-nr=9 --offset=$[18*1024*1024] 1.5MiB 11.3M/s 768KiB 12.6M/s 384KiB 12.4M/s 192KiB 11.8M/s 96KiB 10.9M/s 48KiB 9.24M/s 24KiB 6.24M/s 12KiB 3.07M/s

Ok, this is at least a good reference point to compare the other measurements to.

...

One thing I usually try when I'm not entirely sure about the erase block size is to play a bit more with the offset. You have established that 18MB is a boundary, as it normally is for these 3-bit-per-cell chips. If the erase size is 1.5MB, the next boundary is at 19.5MB, while for a 6 MB erase size, it would be at 24 MB.

The offsets that I would try therefore are

19MB (should be fast)

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 11.9M/s 256KiB 11.7M/s 128KiB 11M/s 64KiB 9.63M/s 32KiB 8.27M/s 16KiB 6.21M/s 8KiB 3.3M/s

Oddly, when running that test, sometimes it has performance like that, and other times it has performance like this:

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 4.92M/s 256KiB 4.84M/s 128KiB 4.72M/s 64KiB 4.58M/s 32KiB 4.17M/s 16KiB 3.56M/s 8KiB 2.4M/s

The best explanation I have for this is that in the second case it always does an erase somewhere in the process, which adds a fixed time overhead. That matches the observation that random writes are much slower than linear writes on average. Can you confirm that after doing an 'erase /dev/mmcblk0 0 $[32 * 1024 * 1024]', it is always fast?

The controller seems to be doing something, but I'm not really sure what. See further down this email for more info, it's not intermittent, I can make it happen repeatedly.

...

19.25MB (should be either like 17.75 or like 18.25, which you've both tried before)

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024+256*1024] 512KiB 5.24M/s 256KiB 5.21M/s 128KiB 5.1M/s 64KiB 4.86M/s 32KiB 4.44M/s 16KiB 3.68M/s 8KiB 2.45M/s

It's like 17.75, not like 18.25. Would lead me to believe 1.5MiB is the real erase block size, rather than 6MiB.

right.

...

23.75 (should be like 17.75)

[andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[23*1024*1024+768*1024] 512KiB 3.25M/s 256KiB 1.68M/s 128KiB 2.34M/s 64KiB 2.33M/s 32KiB 2.24M/s 16KiB 2.04M/s 8KiB 2.26M/s

Yes, same as 17.75.

An interesting set of steps that are repeatable where an 19MiB offset is fast, then slow, then fast. I think it's evident of the controller caching, but I'm not sure what it means:

I don't think that there is any significant caching. The controllers typically have a little bit (64 kb?) of SRAM, and the actual data is sometimes written to a temporary location in SLC flash before it gets written to permanent storage, but I would not call that caching in the strict sense.

19MiB offset is fast, like before: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 12M/s 256KiB 11.4M/s 128KiB 11.1M/s 64KiB 9.98M/s 32KiB 8.33M/s 16KiB 6.15M/s 8KiB 3.36M/s

18.5MiB offset is slower, but not slow: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024-512*1024] 512KiB 10.9M/s 256KiB 6M/s 128KiB 5.79M/s 64KiB 5.59M/s 32KiB 4.98M/s 16KiB 4.15M/s 8KiB 2.65M/s

Going back to 19MiB offset, fast till we get to 128KiB size: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 13M/s 256KiB 10.8M/s 128KiB 5.89M/s 64KiB 5.54M/s 32KiB 5.03M/s 16KiB 4.15M/s 8KiB 2.61M/s

And now 19MiB offset is the same speed as the 18.5MiB offset, and it will stay this way until I test at a different offset. Even leaving minutes between tests, as long as I don't read or write anywhere else on the card, it stays this slow: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 6M/s 256KiB 6.04M/s 128KiB 5.91M/s 64KiB 5.55M/s 32KiB 5.03M/s 16KiB 4.14M/s 8KiB 2.65M/s

But if I go test at 24MiB offset, the 512KiB size is slow, then the rest get fast (as expected): [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[24*1024*1024] 512KiB 2.41M/s 256KiB 11.7M/s 128KiB 10.8M/s 64KiB 9.93M/s 32KiB 8.23M/s 16KiB 6.2M/s 8KiB 3.36M/s

Ok, I see. The 512K number is low here because the card first needs to clean up the other erase block by writing it back into permanent MLC storage. After that is is fast.

And then going back to 19MiB offset, 512KiB is slow, but the rest are fast, as expected: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 4.98M/s 256KiB 9.89M/s 128KiB 9.32M/s 64KiB 8.79M/s 32KiB 7.5M/s 16KiB 5.54M/s 8KiB 3.23M/s

Same here, now it cleans up the segment at 24 MB in the first line, the continues as normal.

Any further testing at 19MiB offset stays fast, consistently: [andrew@mythdvr flashbench]$ sudo ./flashbench /dev/sdc --open-au --erasesize=$[512*1024] --blocksize=$[8*1024] --open-au-nr=9 --offset=$[19*1024*1024] 512KiB 10.5M/s 256KiB 10.1M/s 128KiB 8.95M/s 64KiB 7.16M/s 32KiB 7.5M/s 16KiB 5.6M/s 8KiB 3.13M/s

I'm not sure what to make of this. Have you seen this type of behavior before? Repeating these steps, I can consistently get the same results.

The problem is that each line here writes only a partial erase block, so while the card has the chance to hide some of the garbage collection, it sometimes has to do it in the end. Since we are assuming that this card has a 1.5 MB erase block size, I would predict that you never see the slow first line when you move between 18 and 24 MB offset using --erasesize=$[3 * 512*1024], and that the effect would be less drastic if you use --erasesize=$[2*512*1024] (then it has to GC only 0.5MB instead of 1MB.

Arnd