Warning: array_diff_key(): Argument #2 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-core-functions.inc on line 135 Warning: array_diff_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-tagcore.inc on line 2862 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: array_intersect_key(): Argument #1 is not an array in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337 Warning: Invalid argument supplied for foreach() in /ssd/Dropbox/Active_Projects/SRF/4-Prototypes/srf-recognizer.inc on line 1337

ReiserFS vs ext4

I was just lamenting to a friend that I wished that Hans Reiser hadn’t gone to jail or hadn’t killed his wife, or that whole sequence of events hadn’t taken place, and the next version of reiserfs could come out instead. Honestly, I’m not entirely convinced that he did the act, I certainly wasn’t there, and at least the evidence made public doesn’t seem to cohere to serious scrutiny.  Whatever you think about the whole political/trial/issue, one thing is certain from my standpoint — I’ve had a hard time getting decent ssd-disk performance out of my main SR Doombringer server since switching from reiser to ext4.  SR servers are built for web serving performance — they maintain static caches of the dynamic content in compressed form on SSDs, and deliver those to users instead of hitting the backend PHP & database.   For that, the SR cache-repository can maintain anywhere from 500K to 1M “small files” (<16K (actual average=14.5K)).   When the fs was reiserfs, requests for files could hit near full nginx service rates (~60K/s) with low i/o utilization.  But now with ext4, its like almost 100% utilization on the disk for just regular runtime service!

Anyway, since my forward delivery nodes are still running reiser, this puts me in the position to run a benchmark between the main server and the delivery nodes.

The test:

# time find /ssd -type f | wc -l

This test just searches the filesystem for files and then counts them up after they’ve been listed.  Nothing special, just indexing through the fs tree.

SR Doombringer, Faster, More Powerful, able to leap tall buildings in a single bound or fly, whichever is more convenient for the episode. (8 core dual i7 2.6Ghz, 16GB quad channel ram, dual Crucial C300 SSDs, with trim). Basically every piece of Doombringer is faster than everything on the delivery nodes. Kernel 2.6.36.3.  This is after runtime TRIM.
Mount options:

/dev/sdc                /ssd            ext4            noatime,barrier=0,nobh,commit=10,nouser_xattr,data=writeback 0 1

Results:

481,425 files

real    0m18.668s
user    0m0.790s
sys     0m4.497s

18.6s!!!

SR Fry (a Delivery Node) (Core2 dual 3Ghz, 2G ram, OCZ Vertex, no trim). Originally the configurations of Fry and Doombringer are the same (started with duplicate disk images), same compile options/etc. Kernel 2.6.28-gentoo-r5. Apart from the differing kernel versions, not much of the software is different.  After over a year of running, no TRIM.
Mount options:

/dev/sdb                /ssd            reiserfs        notail,noatime  0 1

Results:

473,803 files

real    0m1.418s
user    0m0.352s
sys     0m1.061s

1.4s!!!

That’s more than 13x faster for reiserfs (v3, not 4)!

It should be noted that when Doom’s fs was reiser with the same ocz drives, it was crazy faster than Fry. Now it could be that for some reason the C300′s are much slower than the Vertex, but I doubt that since I only purchase things that are in the top 3 for speed and reliability. Actually, looking at the benchmarks, the C300 is 3x faster (iops) than the Vertex (not the vertex 2).  So assuming that every component in Doombringer’s system is faster than Fry, that means that 13x is my lower bound — reiserfs is at least that much faster that ext4.

When I run the test again so that the kernel has done its fs caching nonsense, the times drop to 2.7s (Doom) vs 1.3s (Fry). Still more than 2x.

Where is a stable version of reiser4? I had taken off reiser & reiser4 from doombringer because there was some problem with the fs silently destroying data. The ssd may have just been going bad, but I had trouble tracking down the source, and “something” was happening even with the replacement ssd’s.

Side note:
For those who care, the average file size across our 480K files is 14.5K. I did it this way because du will sum up disk usage including directories, where I just wanted to know what the individual file allocation was.

# find ./ -type f -exec ls -la {} ; | sed 's/.*nobody //g' | sed 's/[a-zA-Z].*//g' | awk '{ sum+=$1} END {print sum}'

Eg. the difference between on-disk utilization vs actual file size for our files on ext4 is 1.2x:

du -k : 8,486,296K vs above method : 6,817,378K

On reiserfs, it is 1.18x:

du -k : 7,865,638K vs above method : 6,665,967K