Hi,
At the last release meeting, the switch to ext4 by default has been mentioned. My understanding is that we reach an agreement on the switch to ext4 [1] but it still not clear if it will happen this month.
To make it happen, it will require several bug fixes: https://bugs.launchpad.net/ubuntu/+source/linux-linaro-omap/+bug/817148 https://bugs.launchpad.net/linaro-image-tools/+bug/821479 https://bugs.launchpad.net/linaro-ubuntu/+bug/822593 https://bugs.launchpad.net/linux-linaro/+bug/824545
Also, it's worth mentioning that changing the default will break previous releases if they don't have ext4 support. The workaround is linaro-media-create --rootfs option to specify the filesystem.
If there's any blockers for Linaro 11.08 time frame, let me know.
Cheers,
On Friday 12 August 2011 21:12:54 Fathi Boudra wrote:
At the last release meeting, the switch to ext4 by default has been mentioned. My understanding is that we reach an agreement on the switch to ext4 [1] but it still not clear if it will happen this month.
To make it happen, it will require several bug fixes: https://bugs.launchpad.net/ubuntu/+source/linux-linaro-omap/+bug/817148 https://bugs.launchpad.net/linaro-image-tools/+bug/821479 https://bugs.launchpad.net/linaro-ubuntu/+bug/822593 https://bugs.launchpad.net/linux-linaro/+bug/824545
Also, it's worth mentioning that changing the default will break previous releases if they don't have ext4 support. The workaround is linaro-media-create --rootfs option to specify the filesystem.
I think it's also worthwhile to look at the results of Tixy's file system simulation before we switch. If there are significant performance regressions over ext3 or if btrfs turns out to be much better than ext4, we should probably not move to ext4 at this point but rather try to get btrfs fully supported. My understanding is that there are other bugs that would need to be fixed to make that happen.
Arnd
On Fri, Aug 12, 2011 at 4:11 PM, Arnd Bergmann arnd@arndb.de wrote:
On Friday 12 August 2011 21:12:54 Fathi Boudra wrote:
At the last release meeting, the switch to ext4 by default has been mentioned. My understanding is that we reach an agreement on the switch to ext4 [1] but it still not clear if it will happen this month.
To make it happen, it will require several bug fixes: https://bugs.launchpad.net/ubuntu/+source/linux-linaro-omap/+bug/817148 https://bugs.launchpad.net/linaro-image-tools/+bug/821479 https://bugs.launchpad.net/linaro-ubuntu/+bug/822593 https://bugs.launchpad.net/linux-linaro/+bug/824545
Also, it's worth mentioning that changing the default will break previous releases if they don't have ext4 support. The workaround is linaro-media-create --rootfs option to specify the filesystem.
I think it's also worthwhile to look at the results of Tixy's file system simulation before we switch. If there are significant performance regressions over ext3 or if btrfs turns out to be much better than ext4, we should probably not move to ext4 at this point but rather try to get btrfs fully supported. My understanding is that there are other bugs that would need to be fixed to make that happen.
Yeah, if we're doing this change it seems it would make more sense to jump directly to the btrfs, unless we can demonstrate that the performance is not that superior and have any kind of blocker issues.
Do we have any kind of benchmark results comparing each filesystem when using them with SD cards around?
Thanks,
On Wed, 2011-08-17 at 00:13 -0300, Ricardo Salveti wrote:
Yeah, if we're doing this change it seems it would make more sense to jump directly to the btrfs, unless we can demonstrate that the performance is not that superior and have any kind of blocker issues.
Do we have any kind of benchmark results comparing each filesystem when using them with SD cards around?
I'm doing some benchmarking, though it's mostly being aimed at producing media access patterns to feed into a simulation tool. From these access patterns, btrfs looks a lot worse than any ext file system.
I just looked at the timestamps of my blktrace logs to get some real world timings. For untaring kernel source on one of my good performance SD cards on a Beagleboard-xM takes:
m s ext4 3:30 ext3 8:30 ext2 5:00 btrfs 13:40 nilfs 10:40 logfs 10:00
this is using default mount options for file system but with noatime.
These timings also bear out preliminary results from my simulation code. Which I'm glad of :-)
Note, I've only been looking at write performance.
On 17 August 2011 08:55, Tixy tixy@clara.net wrote:
On Wed, 2011-08-17 at 00:13 -0300, Ricardo Salveti wrote:
Yeah, if we're doing this change it seems it would make more sense to jump directly to the btrfs, unless we can demonstrate that the performance is not that superior and have any kind of blocker issues.
Do we have any kind of benchmark results comparing each filesystem when using them with SD cards around?
I'm doing some benchmarking, though it's mostly being aimed at producing media access patterns to feed into a simulation tool. From these access patterns, btrfs looks a lot worse than any ext file system.
I just looked at the timestamps of my blktrace logs to get some real world timings. For untaring kernel source on one of my good performance SD cards on a Beagleboard-xM takes:
m s ext4 3:30 ext3 8:30 ext2 5:00 btrfs 13:40 nilfs 10:40 logfs 10:00
this is using default mount options for file system but with noatime.
These timings also bear out preliminary results from my simulation code. Which I'm glad of :-)
This is odd. When I performed tests btrfs and ext4 were both about the same speed for copying a mixture of large and small files to. I was testing them using my laptop card reader though. Tixy: Do you get similar btrfs vs ext4 results using the same card on an x86 host? Is your host 64 or 32 bit? I can't find any warnings about btrfs being slow on 32 bit systems during a quick search, but ZFS certainly works better on 64 bit systems. Perhaps btrfs is more optimal on 64 bit systems as well.
One thing I noticed during Ubuntu boot on my Panda was that the mount process would say that it detected btrfs was running on a flash card and it had enabled flash mode. I don't know what is different in flash mode and I don't know if when I let Ubuntu auto-mount a flash card for testing on my Laptop if it enabled it. The only mount option I can find that sounds right is "ssd", which isn't on when I just tested it now.
On Wednesday 17 August 2011, James Tunnicliffe wrote:
m sext4 3:30 ext3 8:30 ext2 5:00 btrfs 13:40 nilfs 10:40 logfs 10:00
this is using default mount options for file system but with noatime.
These timings also bear out preliminary results from my simulation code. Which I'm glad of :-)
This is odd. When I performed tests btrfs and ext4 were both about the same speed for copying a mixture of large and small files to. I was testing them using my laptop card reader though. Tixy: Do you get similar btrfs vs ext4 results using the same card on an x86 host? Is your host 64 or 32 bit? I can't find any warnings about btrfs being slow on 32 bit systems during a quick search, but ZFS certainly works better on 64 bit systems. Perhaps btrfs is more optimal on 64 bit systems as well.
I'm pretty sure that it has little to do with the kind of host and much more with the specific card that you use. Based on the theoretical understanding of how the cards work, it is very likely that some (good) media work better with ext4 than others that would still be ok on btrfs. Also, I would expect ext4 to cope better with full drives than btrfs.
This is of course the data that we hope to get out of Tixy's research.
One thing I noticed during Ubuntu boot on my Panda was that the mount process would say that it detected btrfs was running on a flash card and it had enabled flash mode. I don't know what is different in flash mode and I don't know if when I let Ubuntu auto-mount a flash card for testing on my Laptop if it enabled it. The only mount option I can find that sounds right is "ssd", which isn't on when I just tested it now.
Good point. Tixy, do you know what mode you were testing btrfs with?
btrfs understands the 'ssd', 'ssd_spread' and 'nossd' mount options that should have a significant impact here, although it's not clear which of these is best for a flash memory card, since the characteristics of SD cards are very different from SSD.
The version of btrfs that I'm looking at does not have any optimization for cheap flash drives, only for SSD.
Arnd
At Wed, 17 Aug 2011 13:11:51 +0200, Arnd Bergmann wrote:
On Wednesday 17 August 2011, James Tunnicliffe wrote:
One thing I noticed during Ubuntu boot on my Panda was that the mount process would say that it detected btrfs was running on a flash card and it had enabled flash mode. I don't know what is different in flash mode and I don't know if when I let Ubuntu auto-mount a flash card for testing on my Laptop if it enabled it. The only mount option I can find that sounds right is "ssd", which isn't on when I just tested it now.
Good point. Tixy, do you know what mode you were testing btrfs with?
btrfs understands the 'ssd', 'ssd_spread' and 'nossd' mount options that should have a significant impact here, although it's not clear which of these is best for a flash memory card, since the characteristics of SD cards are very different from SSD.
The version of btrfs that I'm looking at does not have any optimization for cheap flash drives, only for SSD.
does anyone checked on seekwatcher[1], a visualizing tool Chris Mason has wrote. I'm pretty sure that Chris would like to hear what went good and bad for btrfs as an embedded system's root file system.
[1]:http://oss.oracle.com/~mason/seekwatcher/
just my 2 cents.
On Wednesday 17 August 2011, Yasushi SHOJI wrote:
The version of btrfs that I'm looking at does not have any optimization for cheap flash drives, only for SSD.
does anyone checked on seekwatcher[1], a visualizing tool Chris Mason has wrote. I'm pretty sure that Chris would like to hear what went good and bad for btrfs as an embedded system's root file system.
Ah, thanks for the link. This is similar to Tixy's tool, but it focuses on a different aspect of drive performance: While you mostly care about seeks on hard drives, the most important performance factor on flash memory cards is garbage collection cycles. There is some correlation between the two, but also some significant differences.
I guess it would be possible to visualize GC in seekwatcher as well, if anyone is motivated enough.
Arnd
On Wed, 2011-08-17 at 13:11 +0200, Arnd Bergmann wrote:
On Wednesday 17 August 2011, James Tunnicliffe wrote:
One thing I noticed during Ubuntu boot on my Panda was that the mount process would say that it detected btrfs was running on a flash card and it had enabled flash mode. I don't know what is different in flash mode and I don't know if when I let Ubuntu auto-mount a flash card for testing on my Laptop if it enabled it. The only mount option I can find that sounds right is "ssd", which isn't on when I just tested it now.
Good point. Tixy, do you know what mode you were testing btrfs with?
No idea. Do you know a way of finding out what default mount options are? The only documentation I've found is the man page for 'mount', but this doesn't mention btrfs at all.
btrfs understands the 'ssd', 'ssd_spread' and 'nossd' mount options that should have a significant impact here, although it's not clear which of these is best for a flash memory card, since the characteristics of SD cards are very different from SSD.
I'll try those combinations too.
On Wednesday 17 August 2011, Tixy wrote:
On Wed, 2011-08-17 at 13:11 +0200, Arnd Bergmann wrote:
On Wednesday 17 August 2011, James Tunnicliffe wrote:
One thing I noticed during Ubuntu boot on my Panda was that the mount process would say that it detected btrfs was running on a flash card and it had enabled flash mode. I don't know what is different in flash mode and I don't know if when I let Ubuntu auto-mount a flash card for testing on my Laptop if it enabled it. The only mount option I can find that sounds right is "ssd", which isn't on when I just tested it now.
Good point. Tixy, do you know what mode you were testing btrfs with?
No idea. Do you know a way of finding out what default mount options are? The only documentation I've found is the man page for 'mount', but this doesn't mention btrfs at all.
When you look in /proc/mounts while the file system is mounted, it should show exactly one of the three options (ssd, nossd, ssd_spread). You can change the active setting using e.g.
mount -o remount,ssd_spread /dev/sdd1
Arnd
On Wed, 2011-08-17 at 11:48 +0100, James Tunnicliffe wrote:
This is odd. When I performed tests btrfs and ext4 were both about the same speed for copying a mixture of large and small files to. I was testing them using my laptop card reader though. Tixy: Do you get similar btrfs vs ext4 results using the same card on an x86 host? Is your host 64 or 32 bit?
The results I have kicking around for tests on my PC, which uses a year old 64bit kernel are:
ext4 363s ext3 713s ext2 279s btrfs 208s
This was with a different SD card I think. I'll try and do more like-for-like comparison.
On Wed, 2011-08-17 at 11:48 +0100, James Tunnicliffe wrote:
On 17 August 2011 08:55, Tixy tixy@clara.net wrote:
For untaring kernel source on one of my good performance SD cards on a Beagleboard-xM takes:
m sext4 3:30 ext3 8:30 ext2 5:00 btrfs 13:40 nilfs 10:40 logfs 10:00
this is using default mount options for file system but with noatime.
These timings also bear out preliminary results from my simulation code. Which I'm glad of :-)
This is odd. When I performed tests btrfs and ext4 were both about the same speed for copying a mixture of large and small files to.
Mystery solved...
I didn't have btrfs-tools (nor nilfs nor logfs) installed. My test script didn't notice that mkfs failed because I was piping the output through tee, (which, being the end of the pipeline, always gave a success result).
This all resulted in my tests being re-run on the last file system (ext2) which is a interesting result in itself as it shows a lot worse performance compared to a fresh partition. (This is the sort of thing we expected and is on the list of things to investigate further.)
My new results for the untar on a Beagleboard-xM...
ext4 161s ext3 547s ext2 256s btrfs 139s nilfs 157s
I couldn't test logfs because, whilst mkfs worked, the mount command (or the kernel?) doesn't seem to support it.
I also tested the different btrfs mount options (ssd, nossd and ssd_spread). They don't show much difference with the untar case.
On Thursday 18 August 2011, Tixy wrote:
Mystery solved...
I didn't have btrfs-tools (nor nilfs nor logfs) installed. My test script didn't notice that mkfs failed because I was piping the output through tee, (which, being the end of the pipeline, always gave a success result).
This all resulted in my tests being re-run on the last file system (ext2) which is a interesting result in itself as it shows a lot worse performance compared to a fresh partition. (This is the sort of thing we expected and is on the list of things to investigate further.) My new results for the untar on a Beagleboard-xM...
ext4 161s ext3 547s ext2 256s btrfs 139s nilfs 157s
Ok, that is certainly an interesting and more logical result. ext3 is worse than ext2 because of the journal. ext4 has optimizations for flash media in it and btrfs is better by design.
I couldn't test logfs because, whilst mkfs worked, the mount command (or the kernel?) doesn't seem to support it.
Probably the module was not enabled in the kernel.
I also tested the different btrfs mount options (ssd, nossd and ssd_spread). They don't show much difference with the untar case.
Ok.
Arnd
On Thu, 2011-08-18 at 14:28 +0200, Arnd Bergmann wrote:
On Thursday 18 August 2011, Tixy wrote:
I couldn't test logfs because, whilst mkfs worked, the mount command (or the kernel?) doesn't seem to support it.
Probably the module was not enabled in the kernel.
It wasn't. But after enabling it and trying it out, the logfs code oopsed. I'll give it a miss.
On Thursday 18 August 2011, Tixy wrote:
On Thu, 2011-08-18 at 14:28 +0200, Arnd Bergmann wrote:
On Thursday 18 August 2011, Tixy wrote:
I couldn't test logfs because, whilst mkfs worked, the mount command (or the kernel?) doesn't seem to support it.
Probably the module was not enabled in the kernel.
It wasn't. But after enabling it and trying it out, the logfs code oopsed. I'll give it a miss.
Fair enough.
BTW, do you have any write amplification numbers to look at?
Arnd
Arnd,
On Thu, Aug 18, 2011 at 02:28:07PM +0200, Arnd Bergmann wrote:
ext4 has optimizations for flash media in it and btrfs is better by design.
Do you have a pointer to more info about what kind of optimizations for flash media ext4 has? We tried to find something, but didn't so far.
rsc
On Monday 26 September 2011, Robert Schwebel wrote:
On Thu, Aug 18, 2011 at 02:28:07PM +0200, Arnd Bergmann wrote:
ext4 has optimizations for flash media in it and btrfs is better by design.
Do you have a pointer to more info about what kind of optimizations for flash media ext4 has? We tried to find something, but didn't so far.
No, sorry. I think the block allocation work rather differently, and the move from block pointers to extents should also help, but I don't know how much of that was done specifically for flash, or what other optimizations are there.
Arnd
On Thu, 2011-08-18 at 13:18 +0100, Tixy wrote: <big snip>
I also tested the different btrfs mount options (ssd, nossd and ssd_spread). They don't show much difference with the untar case.
To follow on from this... When benchmarking debboottrap, I found the 'ssd' option is three times faster than 'nossd'. I'm guessing that this is due to fsync operations causing more rigorous flushing of data to disk when mounted as 'nossd'.
Other usecases, like copying a big file, or untar, seem to have comparable speed whichever mount option is used for btrfs.
I'll be writing up some more details of my findings soon.
-
Arnd Bergmann -
Fathi Boudra -
James Tunnicliffe -
Ricardo Salveti -
Robert Schwebel -
Tixy -
Tixy -
Yasushi SHOJI