Posts: 803
Threads: 135
Joined: Feb 2011
Hi,
When using Combinator attack (with two wordlists), hashcat does not unique duplicate words.
I think it's useful, but it may exist a reason why developers did not implement that ?
Thanks.
Posts: 930
Threads: 4
Joined: Jan 2015
The assumption is that the user will deduplicate, if desired.
~
Posts: 803
Threads: 135
Joined: Feb 2011
From an algorithm perspective, would it be faster to sort & uniq on CPU or GPU, for huge files ?
Posts: 930
Threads: 4
Joined: Jan 2015
11-29-2017, 10:37 PM
(This post was last modified: 11-29-2017, 10:49 PM by royce.)
CPU, I would think.
Here's an alias that I stole from epixoip that works well on Linux:
Code:
bigsort() { LC_ALL=C sort --parallel=4 -S 4000M -T /path/to/fast/storage/ $*; }
Adjust 'parallel' and '-S' option sizes to taste, based on your number of cores, RAM, etc.
This assumes that you don't need to preserve order, and it's only one file. If you're managing a library of wordlists, for which order is significant (most likely words first), I've had good luck using rli/rli2 from hashcat-utils to remove duplicates among files.
~