hashcat Forum
Combinator Attack and unique - Printable Version

+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Support (https://hashcat.net/forum/forum-3.html)
+--- Forum: hashcat (https://hashcat.net/forum/forum-45.html)
+--- Thread: Combinator Attack and unique (/thread-7056.html)



Combinator Attack and unique - Mem5 - 11-28-2017

Hi,
When using Combinator attack (with two wordlists), hashcat does not unique duplicate words.
I think it's useful, but it may exist a reason why developers did not implement that ?
Thanks.


RE: Combinator Attack and unique - royce - 11-29-2017

The assumption is that the user will deduplicate, if desired.


RE: Combinator Attack and unique - Mem5 - 11-29-2017

From an algorithm perspective, would it be faster to sort & uniq on CPU or GPU, for huge files ?


RE: Combinator Attack and unique - royce - 11-29-2017

CPU, I would think.

Here's an alias that I stole from epixoip that works well on Linux:

Code:
   bigsort() { LC_ALL=C sort --parallel=4 -S 4000M -T /path/to/fast/storage/ $*; }

Adjust 'parallel' and '-S' option sizes to taste, based on your number of cores, RAM, etc.

This assumes that you don't need to preserve order, and it's only one file.  If you're managing a library of wordlists, for which order is significant (most likely words first), I've had good luck using rli/rli2 from hashcat-utils to remove duplicates among files.