[NTLUG:Discuss] eliminating lines with the same information

[Kenneth Loafman]

Normalize the file to only email addresses like line 1 (sed or any regex
capable editor should do the trick), then use:

  $ sort < input | uniq -u > output

to generate the list.

...Ken

Lance Simmons wrote:
> I have a text file with several thousand email addresses, many of
> which are duplicates. I've used "sort" and "uniq" to make the list
> smaller, but there are still almost a thousand..
> 
> But I still have many duplicates.  For example, three lines in the file might be
> 
>   jsmith at abc.org
>   "John Smith" <jsmith at abc.org>
>   "Mr. John Smith" <jsmith at abc.org>
> 
> Obviously, I'd like to get rid of two of those lines without having to
> manually go through and decide which to keep.  And I don't care about
> keeping names, I'm only interested in addresses.
> 
> Also, the duplicates are not all on lines near each other, so even if
> I wanted to do it manually, it would be a huge hassle.
> 
> Any suggestions?
>