[NTLUG:Discuss] Re: compression and Filesystem question -- true hardware/intelligent ATA RAID

[Bryan J. Smith]

Richard Geoffrion wrote:  
> I'll have to look into whether or not the 3Ware cards support
> host-swapping. 

Yes.  3Ware Escalades are _true_intelligent_ ATA RAID cards. 

They hide the entire volume from the OS, unlike those "trick BIOS" cards
where the OS still sees the "raw" ATA drives until a "driver" loads.


o  Intelligent (true hardware) ATA RAID

The _only_ intelligent ATA RAID cards I know of are:
- 3Ware Escalade 5000/6000 (32-bit ASIC, 0.25-0.5MB SRAM)
- 3Ware Escalade 7000/8000 (64-bit ASIC, 1-4MB SRAM)
- 3Ware Escalade 9000 (64-bit ASIC, 4MB SRAM, up to 128MB+? SDRAM)
- Adaptec 2400A/2800A (32-bit i960 microcontroller, up to 128MB+? SDRAM)
- Promise _Super_Trak (32-bit i960 microcontroller, up to 128MB+? SDRAM)

3Ware's 3w-xxxx driver has been in the stock kernel since 2.2.15 (yes,
that's _2.2_), although you'll want the latest for your firmware revision.
The 3w-9xxx driver is now in 2.6.6+ (it requires a new driver for the
added SDRAM).  100% GPL.

The Adaptec 2400A/2800A uses the dpt_i2o GPL driver.
Promise has its own GPL drivers for the SuperTrak.

You'll note that the Adaptec 2400A/2800A and Promise SuperTrak (_not_
the FastTrak) use the traditional microcontroller (uC)+SDRAM approach
like all intelligent, buffering (wait state) SCSI RAID cards.

3Ware uses a non-blocking I/O ASIC+SRAM approach which is more like a
high-speed switch fabric or CPU cache for the ultimate RAID-0, 1, 0+1
performance, although this means the amount of cache is drastically
reduced (not ideal for RAID-5 writes). Their new 9000 series now adds
SDRAM as well, improving RAID-5 write performance for the "best of all
worlds."

In either design (ASIC or uC), the "brains" is in the on-board firmware.
So writing GPL drivers do _not_ expose the proprietary firmware.  They
are also far more simple to write.

These cards also add intelligent, SCSI-like features since the ASIC/uC
now becomes the "brains" of the drives.  This includes sector remapping,
command queuing, read interleaving (e.g., from RAID-1 or RAID-0+1),
etc...  You get the best of both worlds, ATA performance with SCSI-like
features -- especially on the latest 3Ware Escalade 9000 series
(ASIC+SRAM for non-blocking I/O and SDRAM for buffered RAID-5 writes).

Unless I need 15+ drives with 3+ UltraSCSI channels, there is little
sense to go SCSI RAID these days.  For small-to-medium (SMB) business
servers, ATA RAID on a 3Ware Escalade is pretty unbeatable in performance.
All you need is a hot swap bay for parallel ATA drives keyed for power
down.

With SerialATA, which is built for hot-swap (7-pin data and 15-pin power,
staggered and at a fixed dimension), you basically have a SCSI SCA-like 
and all you need is a backplane to plug it into (along with a sled).
I've already built one server using SerialATA and it works most
excellently.


o  Cheap FRAID (100% software driver) ATA RAID

The following are "16-bit BIOS-only" FRAID (Free RAID) and use a
32/64-bit "software RAID driver" which are typically _not_ GPL:  
- All High Point Technologies (HPT) products
- All Promise _Fast_Trak series (not the SuperTrak) products
- All RAIDCore products (but support Linux LVM/MD directly?)
- All SiliconImage products

Other than RAIDCore, HPT, Promise and SilImage do _not_ support Linux with
GPL drivers (I need to check on RAIDCore).  This means their binary-only
drivers are version-specific.  Why?  Because the OS driver contains _all_
of the "brains."

There is a GPL replacement FRAID driver called "ataraid" which is then
called by the GPL replacement, vendor-specific drivers called hptraid,
pdcraid and silraid, respectively.  Unfortunately, compatibility and
performance is crap, and using Linux's LVM/MD is highly recommended
instead (same deal for NT 5.x's native LDM).

For more on the differences, see "Dissecting ATA RAID Options" in the
2004 April edition of Sys Admin magazine (you do get Sys Admin? ;-):   
  http://www.sysadminmag.com/articles/2004/0404/  


> I know that scsi drives can survive disconnection.

It's more than just that.  There are many other details to "hot swap"
that just the ability to disconnect from the bus.

Again, an intelligent I/O card _completely_hides_ the disk from the rest
of the system at _all_times_.  The "FRAID" cards don't do that, they use
an OS driver to "recombine" the raw ATA disks the system actually sees.
The 16-bit BIOS is merely used for setup, but is _totally_useless_ once
a 32/64-bit OS loads, so it needs a driver.

But if you want first-hand experience, I just removed (6) Maxtor 40GB
7200rpm drives and replaced it with (4) WD 80GB 7200rpm drives _live_
in my 3Ware Escalade 7800 last month.  The system was running Red Hat
Linux (RHL) 8 with the latest release 7.7 firmware, drivers and 3DM.

The procedure was:
- Umount all filesystems on the existing volume
- Yank out the drives*
- Put in the new drives*
- Use 3DM (3Ware Disk Manager) to create a new volume
- Partition with fdisk/parted
- Format filesystems
- Mount

*Requires either SerialATA (which is designed for hot swap, as long
as you use the staggered, 15-pin power connector and not any legacy
Molex power) _or_ parallel ATA with drives in bays that have a
"power down" key or similar.

The volume was /dev/sdb and was completely replaced with a whole new
set of drives.  I've also hot swapped drives before and not done a
full array replacement.

Just didn't make sense to buy a new 40GB drive, so I recycled some (4)
80GB drives from another system.  I'll use (4) good of the (6) 40GB
drives in my portable server, which I use to demo WBEL+Oracle on.

As far as the system I stole the (4) 80GB drives from, I put (2) 200GB
drives in the system.  They are in a RAID-1 configuration.

I don't build a system today without a 3Ware Escalade controller.

With the MTBF of ATA drives, apparent in their 1-year warranties today
-- and that's assuming only 5 day x 14 hour per week usage -- I don't
want a system going down.  It's worth the small added cost in my time
that I don't have to waste booting and pulling from backup.  I still
do backups, but I don't have to worry about a day of downtime if
I lose a disk.


-- 
     Linux Enthusiasts call me anti-Linux.
   Windows Enthusisats call me anti-Microsoft.
 They both must be correct because I have over a
decade of experience with both in mission critical
environments, resulting in a bigotry dedicated to
 mitigating risk and focusing on technologies ...
           not products or vendors
--
Bryan J. Smith, E.I.            b.j.smith at ieee.org