+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Deprecated; Previous versions (https://hashcat.net/forum/forum-29.html)
+--- Forum: General Help (https://hashcat.net/forum/forum-8.html)
+--- Thread: Removing duplicate lines in dictionary file (/thread-3828.html)
Removing duplicate lines in dictionary file - slawson - 11-12-2014
I tried using
sort filename | uniq -u
but of course if a line was duplicated it was removing all the same lines and not keeping a unique line.
RE: Removing duplicate lines in dictionary file - epixoip - 11-12-2014
sort -u filename
RE: Removing duplicate lines in dictionary file - slawson - 11-12-2014
Wow. That was too easy. Although it seems to be fluky on a large file.
RE: Removing duplicate lines in dictionary file - epixoip - 11-13-2014
Not sure what you mean by fluky, but if you have /tmp on tmpfs or a separate smaller partition, you might need to use -T to specify a directory on a larger partition.