[NTLUG:Discuss] Need help debugging simple script commands

[Val W. Harris]

> 
> 
> OK, I see where I've done a poor job of explaining - sorry!
> 
> The duplicate removal process is not failing because the lines don't
> match, it is failing because there is some catastrophic problem with
> the file.
> 
> Here's why I say that:
> 
> After running the steps to remove duplicates I browse the 6.5 meg
> file and can see multiple sets of duplicates on the very first
> page (they are sorted together). I copied the first 5 or 6 pages from
> that file and created a new file. I then ran the same commands run
> previously to remove the duplicates on this new small file. All of the
> duplicates were removed.
> 
> Now what?
> 
> Rick
> 
> 
> _______________________________________________
> http://www.ntlug.org/mailman/listinfo/discuss
> 


I misunderstood the problem.  If the duplicate lines are indeed
identical as seen by xxd, then the only thing that comes to mind is
that sort is running out of temporary file space, abending, without
a reasonable message.  The only way I can think of to test this
hypothesis would be to run the process with successfully decreasing
file sizes until you find the largest input file size that completes
successfully.  If you're comfortable with reading source, look at the
sort source code and check out how it handles it's file buffering.

Truthfully, I wouldn't expect this to be the problem; but Linux still
suprises me on occasion!

Val
-- 
Val W. Harris            vwharris at airmail.net

"Money is an article which may be used as a  universal
  passport to everywhere except  heaven, and a universal
  provider of everything except happiness." - Anonymous