[NTLUG:Discuss] Need help debugging simple script commands

[Rick Matthews]

> You'll need a hexdump program and an ASCII chart.  I recommend xxd.
> Do a man on xxd and ascii.  Take a snippet of your bad output file
> ( the one with bad dups in it ), and run it through xxd.  The third
> column of output is the ascii text.  It'll contain blanks for the
> nonprintable characters.  If all of the printable characters match
> over the two offending lines, look in the second column.  This is the
> hex value of each character.  Presumably, some of the nonprinting
> characters don't match.  You can identify their hex value and then
> look it up on the ASCII chart.

OK, I see where I've done a poor job of explaining - sorry!

The duplicate removal process is not failing because the lines don't
match, it is failing because there is some catastrophic problem with
the file.

Here's why I say that:

After running the steps to remove duplicates I browse the 6.5 meg
file and can see multiple sets of duplicates on the very first
page (they are sorted together). I copied the first 5 or 6 pages from
that file and created a new file. I then ran the same commands run
previously to remove the duplicates on this new small file. All of the
duplicates were removed.

Now what?

Rick