[NTLUG:Discuss] RE: Fastest Access for moving large (video) files?

[Bryan J. Smith]

Burton Strauss <Burton_Strauss at comcast.net> wrote:
> The absolute fastest way to go would be Ultra320 SCSI
> RAID0.

Not necessarily.

First off, you have 2 variables:  
- latency
- data transfer rate (DTR)

For single processes reading/writing large files, DTR is key.
 For multiple processes, latency will clearly start affecting
DTR.  The question is where a lower DTR but higher latency is
lower performance than a higher DTR but lower latency.

Now, with that said, you have 2 components:
- disk
- interface

Commodity disk densities at a spindle of 7200rpm are now 250,
300, 320 and 400GiB, with 500GiB shortly appearing. 
Enterprise disk densities at a spindle of 10000 and 15000rpm
are currently 36, 73 and 146GiB, although 300GiB is coming
shortly.

"Raw, sequential" DTR clearly goes to commodity disk.  So if
your application is a single process controlling the disk, a
commodity 7200rpm disk will typically outperform even
15000rpm enterprise disk of a much lower density.

At the same time, even desktops can benefit from high spindle
rates _if_ multiple applications are occuring simultaneously.
 Even the 73GiB, 10000rpm Raptor typically wins most 4+
thread benchmarks against its own 320GiB, 7200rpm commodity
disks.

Now let's talk interface.  ATA is built for non-blocking I/O,
whereas SCSI is for command-queuing.  On a single process,
ATA is much lower overhead and directly driven.  For multiple
processes, SCSI quickly shows its superiority at commanding.

Now there are new Native Command Queuing (NCQ) ATA drives,
but they only go so far.  The ultimate problem for NCQ is
that it's per-disk, whereas a SCSI host adapter does it for
the entire SCSI bus (or multiple busses in many, modern SCSI
implementations).  So if there is only 1 disk, and only 1-2
processes, ATA is typically the fastest.  But if there are
more than 1 disk and/or many processes, SCSI usually takes
the crown.

Now this assumes you don't have an intelligent ATA host
adapter.  If you have an intelligent ATA host adapter, things
get interesting.  ATA is a non-blocking I/O and ASIC+SRAM
also is typically 0 latency.  If you want the ultimate in
unblocked I/O, then an ASIC+SRAM with ATA disks is ideal. 
Even for multiple processes, the ASIC can queue up commands
even with only 1-2MB of SRAM cache.  This is where 3Ware
rules.

Unfortunately, most ATA RAID solutions are either:
- Non-ASIC, old microcontroller designs
- ASIC+SRAM, but no DRAM buffering for heavy traffic

Most ATA offerings, except for the LSI Logic MegaRAID 300-8X
(IOP321, 400MHz XScale, 8-channel SATA) are old 33-100MHz
IOP30x (i960) microcontrollers.  They are typically going to
have serious issues breaking 50-60MBps, and RAID-5
performance is typically sub-30MBps.

Now while the 3Ware Escalade 7000/8000 series might shine
with it's 64-bit ASIC+SRAM design, the lack of DRAM (and only
1-2MB of SRAM, 4MB in the 12 channel versions) seriously
hurts its buffering capabilities.  They are great for queuing
I/O, but once the server is saturated with buffer commits
that exceed its memory, the 3Ware can't do much but stall. 
The lack of DRAM buffer also hurts its RAID-5 write
performance too.

Now the 3Ware Escalade 9000 series adds DRAM to it's existing
ASIC+SRAM design, but the bugs still seem to be getting
"worked out" as of even the latest 9.2 firmware.

Another solution I've been exposed to recently (and
personally own) is the NetCell SyncRAID SR3000/5000 series. 
Their RAID-XL solution is a fixed RAID-3 solution, ideal for
desktops.  Instead of using a blocked I/O (RAID-0, 10, 4 or
5) or variable number of disks (most other RAID-3
implementations), RAID-XL forces you to 2+1 disks (32-bit
card) or 4+1 disks (64-bit card).  ATA channels are 16-bit
wide, hence these specific design.  The parity is dedicated
(in RAID-3 and 4, RAID-3 for the NetCell), and done in
real-time using its on-board ASIC w/XOR engine (with 128MB
DRAM buffer as necessary).  The card itself looks like a
single ATA device on a single channel (although getting every
OS' ATA support to recognize the ID is another story --
NetCell's _finally_ went into the very recent 2.6.13-rc3 I
believe ;-).

> Second choice (much better economics) would be the 10K
> Raptor SATA RAID0.

The Raptor is the Hitachi 36/73GB U320 SCSI with SATA logic,
essentially cutting the target disk price in half.

> (I wouldn't bother w/ SATA-II or -300 as most drives can't
> yet saturate SATA-150, especially on PCI bus).

Agreed.  Although 300MBps SATA is now known as SATA-IO, and
requires a twisted pair cable (same as 300MBps SAS).  I'd
like to know if the twisted pair cable reduces re-transmits
(and possibly increases performance?).

> 2x 36GB Raptors + a decent SATA RAID card could run
> $500-700 (best prices seem to be 130 per drive and 300
> or so for the card).

3Ware Escalade 8006-2 cards are only $140.  But they are
64-bit at 66MHz (0.5GBps) and are really starving for their own
PCI (or PCI-X) channel.

> This will give you 72GB of 'work' space.  The raptors spec
> at 72 MB/s (Sustained) transfer, so RAID0 will give you
> maybe 1.7x that (with overhead and what not) or 122MB/s --

Sounds about right.

Most Bonnie benchmarks I've regularly been exposed to using
3Ware 8506-4LP cards with four (4) commodity 250-320GiB disks
in RAID-10 result in around 120-130MBps for reads, 100MBps
for writes.

> pretty close to the PCI bus limit.

The problem is the other devices on the shared, legacy PCI
bus.  Especially cheap audio chipsets that can saturate it
with channel audio streams (that are generated at the host
CPU), let alone NIC on the same bus (although some chipset
NICs are now HyperTransport integrated, e.g., nVidia, or PCIe
x1 channels, e.g., Intel).

> RAM-drive's top out at 16GB - Toshiba has one coming out at
> this size - see this month's MaximumPC.

Definitely not "cost effective" IMHO.
But if you need it, you need it.


-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith at ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)