[NTLUG:Discuss] Adaptec 52445 and 8 x 5200rpm 1TB laptop drives
Christopher Cox
cjcox at acm.org
Sun Feb 3 22:29:58 CST 2013
I just wanted to post on something I worked on recently at home. I have a 8 x
1TB SAN unit at home... but in order to manipulate big data, you need big data.
I was going to look at adding another SAN, but didn't like the options for sale...
I found somebody selling an Adaptec 52445 w/BBU for about $260 (that's a very
high end SATA2 hw RAID card with 24 ports). So I bought that, two 4 drive
cables, 2 icydock 4 bay hotswap internals and 9 x 1TB 5400rpm drives (I had 3
from a recent newegg shell shocker, so I added 6 more, the extra is just so I
have a spare on hand).
I RAID5 the 8 drives together on the Adaptec and ran a bonnie++
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
perry 63G 1151 99 592137 82 171219 29 1641 78 782998 55 445.9 25
Latency 14168us 59849us 494ms 377ms 92121us 287ms
Version 1.96 ------Sequential Create------ --------Random Create--------
perry -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20622 54 +++++ +++ 25120 61 22679 59 +++++ +++ 24165 61
Latency 8049us 53us 92us 433us 21us 99us
1.96,1.96,perry,1,1359851245,63G,,1151,99,592137,82,171219,29,1641,78,782998,55,445.9,25,16,,,,,20622,54,+++++,+++,25120,61,22679,59,+++++,+++,24165,61,14168us,59849us,494ms,377ms,92121us,287ms,8049us,53us,92us,433us,21us,99us
That was "untuned"... (later is a tuned bonnie++, with XFS options to match the
RAID)
Now... the seq block numbers are good and Rewrite is a good indicator of overall
performance under normal rw workloads. I don't need that performance btw, just
thought it would be nice to post. The filesystem I used was XFS for the bench
on a SLES 11 SP2 (this is the host where I put all of my KVM virtual machines).
The host is powered by dual AMD 6128's and has 32GB of memory.
The low Random seeks, which is a good indicator of single client IOPS, are low
because these are 5400rpm drives after all... but still... even at 445, that's
not horrible.
Btw, the people I bought that card from still have more for sale, but they
realized their mistake and have the cards now priced at $399.
Next thing I need to add is a USB 3.0 PCIe.. need one that will work with a 3.0
kernel though. I welcome suggestions.
Bonnie++ after tuning...
mkfs.xfs -f -d su=256k,sw=7 -s size=4096 /dev/perry1vg/test
(this changes away from the default 512byte sector size, this is a new drive
with 4096 byte sectorsize... and sets the stripe to 256k and width to 7 which is
number of drives in array-1... for RAID5)
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
perry 63G 1142 99 545463 75 177675 31 2128 99 742694 53 467.6 26
Latency 17372us 123ms 498ms 17107us 83714us 367ms
Version 1.96 ------Sequential Create------ --------Random Create--------
perry -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 19603 54 +++++ +++ 25383 61 21534 57 +++++ +++ 24208 60
Latency 22115us 68us 95us 380us 25us 100us
1.96,1.96,perry,1,1359901631,63G,,1142,99,545463,75,177675,31,2128,99,742694,53,467.6,26,16,,,,,19603,54,+++++,+++,25383,61,21534,57,+++++,+++,24208,60,17372us,123ms,498ms,17107us,83714us,367ms,22115us,68us,95us,380us,25us,100us
After changing scheduler to deadline instead of cfq... (but notice how the
writes go down... interesting... see also lower file creates, notice lower CPU
utilization as well on writes... could be related)
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
perry 63G 1138 99 338795 43 209624 32 2117 99 787359 62 490.7 25
Latency 15466us 322ms 8917ms 21490us 162ms 89936us
Version 1.96 ------Sequential Create------ --------Random Create--------
perry -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 9088 24 +++++ +++ 26037 62 22228 59 +++++ +++ 24621 61
Latency 408us 128us 89us 339us 18us 103us
1.96,1.96,perry,1,1359902233,63G,,1138,99,338795,43,209624,32,2117,99,787359,62,490.7,25,16,,,,,9088,24,+++++,+++,26037,62,22228,59,+++++,+++,24621,61,15466us,322ms,8917ms,21490us,162ms,89936us,408us,128us,89us,339us,18us,103us
Ok.. how about the noop scheduler...
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
perry 63G 1140 99 427795 56 215260 33 2148 99 779608 62 491.9 25
Latency 14541us 223ms 8306ms 13296us 243ms 100ms
Version 1.96 ------Sequential Create------ --------Random Create--------
perry -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 21368 57 +++++ +++ 23967 60 21845 55 +++++ +++ 24902 63
Latency 340us 54us 102us 238us 61us 110us
1.96,1.96,perry,1,1359903434,63G,,1140,99,427795,56,215260,33,2148,99,779608,62,491.9,25,16,,,,,21368,57,+++++,+++,23967,60,21845,55,+++++,+++,24902,63,14541us,223ms,8306ms,13296us,243ms,100ms,340us,54us,102us,238us,61us,110us
Obviously there are a ton of other options to try including more options inside
of queue/iosched... too much and too little time :-)
More information about the Discuss
mailing list