Removing duplicate lines in dictionary file
I tried using
sort filename | uniq -u

but of course if a line was duplicated it was removing all the same lines and not keeping a unique line.
sort -u filename
Wow. That was too easy. Although it seems to be fluky on a large file.
Not sure what you mean by fluky, but if you have /tmp on tmpfs or a separate smaller partition, you might need to use -T to specify a directory on a larger partition.