CPU, I would think.
Here's an alias that I stole from epixoip that works well on Linux:
Adjust 'parallel' and '-S' option sizes to taste, based on your number of cores, RAM, etc.
This assumes that you don't need to preserve order, and it's only one file. If you're managing a library of wordlists, for which order is significant (most likely words first), I've had good luck using rli/rli2 from hashcat-utils to remove duplicates among files.
Here's an alias that I stole from epixoip that works well on Linux:
Code:
bigsort() { LC_ALL=C sort --parallel=4 -S 4000M -T /path/to/fast/storage/ $*; }Adjust 'parallel' and '-S' option sizes to taste, based on your number of cores, RAM, etc.
This assumes that you don't need to preserve order, and it's only one file. If you're managing a library of wordlists, for which order is significant (most likely words first), I've had good luck using rli/rli2 from hashcat-utils to remove duplicates among files.
~