[NTLUG:Discuss] Why would a command stop working?

[Rick Matthews]

----- Val W. Harris wrote: -----

> Did the processes generating the lines in file1 change?
>     If so, perhaps there's a stray blank at the end of a line.

I don't control the processes that generate file1. I apologize for not
being more specific before. I receive 2 files from external sources in a
common format, and I maintain a third file locally (in the same format).
The script currently combines them as follows:

cat file1 file2 file3 | sort | uniq > file4

As Steve Baker questioned earlier, my test this evening was with over 3
meg of data. I cut it down to about 50 lines and 'cat file1 | sort |
uniq > file2'  worked properly.

Apparently one of my input files contains some garbage. These should be
straight text files and they are being sorted by the full line (no
options used with sort or uniq). How can I validate the format of the
input files prior to processing? (I need to check to see if there is a
grep option to select only text lines...)?

> Are you sorting entire lines or just one field of the line?

Entire lines.

> Is there some reason your script doesn't just do?:
>       sort -u file1 > file2

Learning curve, and multiple input files. Shoot, I was tickled pink when
I found that I could use the cat|sort|uniq method! I've since learned of
the unique option in sort (merge, too), in fact, I also tested with
the -u option earlier. Using the 3.5 meg file the following two command
lines produced identical results, with the output file still containing
duplicates:

cat file1 | sort | uniq > file2
sort -u file1 > file2

Thanks for your help!

Rick Matthews