[NTLUG:Discuss] picking a filesystem in 2006...

[Chris Cox]

Richard Geoffrion wrote:
> In a different thread Chris wrote:
>>Since I've run 2TB filesystems for awhile using reiserfs, 
>>I can say it works well (2TB because historically I was 
>>using LVM1).
> 
>>From that I gather LVM1 had a 2TB limit.
> 
>>I'd make sure that your ext2 utilities are up to date.  
>><SNIP>
> 
>>If you just can't stand reiserfs (v3), I'd probably look at
> xfs and jfs.  
> 
> I have this to say about ext3 and reiserfs as it relates to using dirvish.  ext3 outperforms reiserfs by..oh..it has to be 4000% if not more!   There is some fundamental underlying process that makes reiserfs slow when running dirvish-expire to remove the 'vaults'.  Hmm..let me see if I can re-google the answer.... hang on a second.

Not sure.  While I do see some long delays on SCSI, I know fiber
mounts happen almost instantaneously... perhaps not related to
the filesystem (???).

In all fairness, I think we saw the same mount delay with ext3
and I think with jfs (not sure on jfs).

I'm curious about the 4000% claim.  My benchmarks didn't
show anything like that, but I'll admit, we used a large
scale bonnie++ test and a large file NFS test... didn't
do extreme testing.  Overall, ext3 came out on top.. but
just barely... we chose reiserfs for the flexibility
in growing filesystems while mounted.

I can share the results.... I think I have them posted
somewhere (let me look).

> 
> {music plays in the background}
> 
> Ok...I'm back..sorry it took so long.  I wasn't able to find the whys-and-wherefores but I did find a benchmark test that shows some of the results.   I dunno.. when I try to remove a large directory it takes forever on reiserfs.  And let's talk talk about mounting times!  wow!  ext3 is dang near instant where ReiserFS takes so loooooooongggg.  (benchmarkes - http://linuxgazette.net/102/piszcz.html )
> 
> 
>>I still believe there is a major fundamental bug inside of
>>ext2 (and therefore 3 as well).  Times I've lost an ext3 
>>filesystem, the results were NOT pretty at all.
> 
> What fundamental bugs?  Google searches only bring up OS/Distro version bugs and debunk these unwritten fundamental bugs that no one deems worthy to quote.  :)
> 

I can't document it... it happens when I start seeing weird
values like ... freespace = 134223423 and total space = -23423221

Running ext3 fsck sometimes fixes the problem... sometimes
you end up rebuilding.  It's certainly an unwritten
fundamental bug... just don't know what is causing it.
Might not be an ext3 problem, but it affects ext3.
(could be a different kernel issue??)  And yes... I only
see this problem on Red Hat boxes (but again, we don't
run ext3 anywhere else... our large shared storage areas
are reiserfs managed by SUSE).


>><snip>
>>The reiserfs boxes are infinitely more flexible, since the 
>>LVMs and filesystems can be resized without unmounting.  
>>A nifty thing in the enterprise.
> 
> Hmm...Isn't ext3 resizeable?  I see where others talk about it.

An online resizer??  I didn't know about it.  I know you can
resize, but I believe it's not an online resizer (things may
have changed though).

> 
>>LVM2 has its share of warts.... getting better... certainly
>>nice to get beyond some of the limits of LVM1, 
>><snip the implecations> :)

That implication that that when it was just Sistina.. it was
moving along and then got stuck and the mud after the
acquisition by you know who... (hint, it's the company
who said enterprises didn't need logical volume management
when they delivered their first "enterprise" class
distribution).

> 
> (important question to follow:)
> OK..so should LVM2 even CARE about a partition on a >2TB /dev ?  Is running 'parted' and using GPT signatures/EFI even necessary?  (see the wikipedia article which is very informative:  http://en.wikipedia.org/wiki/GUID_Partition_Table )
> 
> cfdisk sees 4.2TB...so I'm guessing that it is not an issue with an OS Num_of_LBA-bit addressing issue.  The OS is seeing the device properly.  I'm just wondering what the real issue is.  I've been at this for so long now that I fear I'm swimming in useless/wrong/too much information.
> 
>>Reiser4 looks downright awesome.... I may deploy this a
>>bit early.  
> 
> Scary...it just seems real scary to trust data to something so new when there are so many other options available.  Now...if it was the only solution or for use in non-critical data...I could see it.
> 
>>Some of the numbers will make you faint...
> Somewhere in my research I read that it was just now getting to the point where it was fast enough (on par with Reiser3) to be usable...of course those posts may have been dated..
> 

Well.. I certainly want to take a look... there's a lot of misinformation
out there about reiserfs v3... and even more about reiser4.

>>I realize it likely has some bugs... but man oh man...
>>Will make dealing with really large filesystems much
>>better (the rest all stink at it... even reiserfs v3 suffers
>>from some of the same problems with regards to large
>>tree removals, etc..... though ext3 is measurably worse). 
>>I know for sure that ext3 and reiserfs will take several days to
>>fsck a 4TB filesystem.... should an fsck become necessary.
>>Ideally fsck becomes a thing of the past with Reiser4 (as
>>it matures).
> 
> 
> reiserfsck is relatively fast.   The problem comes in when one has to do 

reiserfsck isn't that fast on a large populated filesystem.. and yes...
doing a tree rebuild (since it's contructing the meta data from
scratch based on the data it finds on the drive) is very slow.

> a --rebuild tree!  It takes a day on a 230 GB volume!  It'll take WEEKS 
> to do a 4TB volume!  That is unacceptable!    Hey..that's what I need -- 
> a benchmark on the filesystem that fsks the fasted.
>