[NTLUG:Discuss] 4.5TB volume formats out to 95Gig *Half-way solved*

[Chris Cox]

Richard Geoffrion wrote:
> Richard Geoffrion wrote:
>> Richard wrote:
>>   
>>> <snip>
>>>
>>> The new 4.5TB array is built and
>>> configured as an LVM volume and I'm only seeing 95Gigs!
>>>   
>>>     
>> Well, it appears that I'm gonna have to use 'parted' and use the new GPT 
>> (GUID Partition Tables).  ...more to come...
>>
>>   
> Maybe ext2/3 has a limitation!?!
> 
> Help from  
> http://www.gnu.org/software/parted/manual/html_chapter/parted_7.html#SEC67  
> seemed to be the key, but I had some strange results.  After following 
> the instructions and creating my 'pv' on /dev/sdg the 'vg' showed that 
> it was 4.1TB.
>          PV /dev/sdg   VG prepress   lvm2 [4.09 TB / 0    free]
> 
> So I formatted it with the command:
>      mke2fs -j -L PrepressDrive -O dir_index,has_journal,sparse_super -T 
> largefile  /dev/prepress/drive
> 
> But after I mounted it, df still only showed 96gig.
>      /dev/mapper/prepress-drive   95G   33M   91G   1% /prepress
> 
> So I unmounted it and formatted /dev/prepress/drive with reiserfs.  What 
> do you know!  it sees all of the space!!!!!
>      /dev/mapper/prepress-drive  4.1T   33M  4.1T   1% /prepress
> 
> 
> So....I recreated the LVM volume on /dev/sdg1 (the partition created 
> with 'parted') and formatted it with ext2 -- NO JOY!  Only 96Gig.  
> Further research seems to show that EXT2/3 has a 4TB limit. But why the 
> 96GB???--space left over between the ext2/3 limit and the end of the 
> disk space?
> 
> So my questions now are...
> 1) What filesystem SHOULD I use?   After reading some evil stuff about 
> ReiserFS...I'm wary. (metadata vs block journaling)

??? (noting that enterprises have and are still relying on just
metadata journaling from even commercial vendors)

> 2) If I'm using LVM, do I even NEED to partition the drive to see 
> greater than 2TB? What is it on the system that determines the operating 
> system's ability to use the drive space?
> 3) Any good hints on how to fill up and test 4TB of space to verify data 
> integrity won't be compromised by something like ..say....a "drive wrap"?

Since I've run 2TB filesystems for awhile using reiserfs, I can say
it works well (2TB because historically I was using LVM1).

Not sure about the ext3 problem.  I know I've done benchmarks with
ext3 when we were determining what fs to use (we tested reiserfs,
ext3 and jfs... xfs was still having some issues).

I'd make sure that your ext2 utilities are up to date.  I've certainly
seen problems similar to this on RHAS 2.1 (a Red Hat 7.1 derivative).
But obviously, that's pretty extreme and I would not expect it to
the same problem.. but maybe there was a buggy release of the
ext2/3 utilities (????) at some point.

If you just can't stand reiserfs (v3), I'd probably look at
xfs and jfs.  I still believe there is a major fundamental bug
inside of ext2 (and therefore 3 as well).  Times I've lost an
ext3 filesystem, the results were NOT pretty at all.

All of our Red Hat EL systems are ext3 (because that is what
Red Hat trusts) and all of our SUSE Enterprise boxes are
reiserfs (because that is what Novell trusts).  The reiserfs
boxes are infinitely more flexible, since the LVMs and
filesystems can be resized without unmounting.  A nifty thing
in the enterprise.

LVM2 has its share of warts.... getting better... certainly
nice to get beyond some of the limits of LVM1, just not
as feature rich (at least reliably).  I won't implicate
their new owner :)

Reiser4 looks downright awesome.... I may deploy this a
bit early.  Some of the numbers will make you faint...
I realize it likely has some bugs... but man oh man...
Will make dealing with really large filesystems much
better (the rest all stink at it... even reiserfs v3 suffers
from some of the same problems with regards to large
tree removals, etc..... though ext3 is measurably worse).  I know
for sure that ext3 and reiserfs will take several days to
fsck a 4TB filesystem.... should an fsck become necessary.
Ideally fsck becomes a thing of the past with Reiser4 (as
it matures).