(04-21-2021, 03:51 AM)Chick3nman Wrote: Right, and there's some things that simply can't be avoided, so be careful with those numbers.
For example this password and these two rules:
password
Z1
$d
Both will produce the plaintext "passwordd" however they are very clearly NOT duplicate rules of each other and can NOT be thrown away unless the plaintext is known BEFORE we even load/process(hence my comment about seeing the future). So while they may produce some instances of duplication, like with what you've measured there, it's not easily possible for us to throw them out prior to loading the wordlist. Some amount that duplication could come down simply to your wordlist or to neither the wordlist or rules on their own but simply the combination of the two. Some amount of that 37% is essentially unavoidable. The method for processing and avoiding the duplicates is clearly infeasible given a wordlist+rule combo of even moderate size as there's no way to store the words and sort/dedupe them. It simply costs less to run them through the attack than to try and remove them.
Definitely agree, however those rules can be 'avoided' in the sense they can be applied separately in a different attack. Not all of course but the fair majority.
But in the sense - If storage capabilities were cheaper, i'd have most of my lists presorted as dictionaries for straight attacks. Oh a robot can dream.