This is the write-up how I rebuild the new best64.rule out of all the submissions but for real-life use.
This new best64.rule will go into all hashcat distributions.
I took arex best 80's results posted here:
http://thepasswordproject.com/2012-03-28...ompetition
1. Check how useful it really is
To do this i made a MD5 hashlist of rockyou.txt which is the ultimate real-life password resource we have.
Code:
$ cat /root/dict/untouched/rockyou.txt | perl /root/dict2hash.pl > rockyou.md5
The resulting hashlist has 14 million hashes. This is to much.
Quote:14344391 rockyou.md5
I reduced them a bit by randomly fetching out 1 million hashes. This is the hashlist we will work with.
Code:
$ sort -R rockyou.md5 | head -1000000 > rockyou.md5.1000000
$ wc -l rockyou.md5.1000000
Quote:1000000 rockyou.md5.1000000
Now run the top 10k dictionary and the best 80 rules from the contest against it.
Code:
$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80.rule
Quote:...
Recovered.: 17097
Here is the original best64.rule. We will use it later for comparison with the final result:
Code:
$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r rules/best64.rule
Quote:...
Recovered.: 12345
Funny number.. Well, but thats it for real
2. Clean up arex's rules
There is a lot of useless stuff inside. Will do:
- Deobfuscate rules, stuff like ^7 { which is actually just $7
- Remove dupes that come out after deobfuscation.
- Remove as much as possible hashcat specific functions to make resulting hashlist jtr/pwp compatible.
- Remove useless functions to increase performance while rules are applied at runtime in rule-engine: Making it faster!
Code:
$2$4]] :
$4]]] ] ]
T0 T0
$1$2i42@2 $1
i42@2T0u u
}}D1{ ]
$si42@2 $s
]]]d ] ] ] d
$3$2]]]]] ] ] ]
krKz3x13 r
i42@2$2 $2
}}}}'4 } } } } '4
:so0 so0
$0$1i42@2 $0 $1
}]]{ } ] ] {
$1$2$3*22 $1 $2 $3
i42@2$9Z1 $9 $9
lx31 D3
$b]$7 $7
dx28Y4'4d d x28 Y4 '4 d
}}}}}'5 } } } } } '5
z3x13$3 $3
^5{ $5
$6 $9 $6 $9
f*A5'8x14 f *A5 '8 x14
si1 si1
z3x13$4 $4
$0$0$7}[ $0 $0
]]$a ] ] $a
]]]$m$a$n ] ] ] $m $a $n
+5]}}}}'4 +5 ] } } } } '4
*05x03d'3p1 *05 x03 d '3 p1
]]]$1$2$3 ] ] ] $1 $2 $3
i42@2$1$1 $1 $1
$9li42@2 $9
$1 $2 $1 $2
]]]$y ] ] ] $y
^b^b^p^h^p]]]] ^b ^b ^p ^h ^p ] ] ] ]
$2$2'5$1 '5 $1
Z4'8x42 Z4 '8 x42
$0 $0
$6 $6
i42@2$2$3 $2 $3
$2 $2 $2 $2
^&x03{{{{{{ x02 { { { { { {
$1]]]$e$r ] ] $e $r
se3 se3
] ] $i $e ] ] $i $e
]]l]$o ] ] ] $o
$8z3x13 $8
i01i42@2 ^1
}}}}}}Y4'4d } } } } } } Y4 '4 d
$2 $1 $2 $1
^t@tD2D2 D2 D2
{{{{{{{{x31'4 '5 D3
p2'7p1x58 p2 '7 p1 x58
]]D1] ] ] D1 ]
*44D4 D4
+0+0+0+0+0+0+0+0 +0 +0 +0 +0 +0 +0 +0 +0
*04+0'4 *04 +0 '4
$8$8 $8 $8
x14dp2'6 x14 d p2 '6
]]]l$p$h$p$b$b ] ] ] $p $h $p $b $b
Z5*75'5{x02 Z5 *75 '5 { x02
$7$7 $7 $7
x21i42@2 D2
^e^h^ti42@2 ^e ^h ^t
] ] ] $d $o $g ] ] ] $d $o $g
}}-0x12 } } -0 x12
o0to0b o0b
l[^d^t@t [ ^d
$e $e
l]$sl]$a ] $a
$.]]l]$s ] ] $s
+0+0+0x12 +0 +0 +0 x12
:}}} } } }
$1$3 $1 $3
o1ao0ml ^m ^a
Z5'6x31]p1 Z5 '6 x31 ] p1
$0$2 $0 $2
Some rules could not get deobfuscated/optimized.
Some of the changes might have an influence on the cracked results since I removed lots of stuff, too.
To make sure the changes are not so bad, here is a backup run to see its not so far away from the 17097 from the original set.
Code:
$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule
Quote:...
Recovered.: 17050
OK, thats fine
2. Remove the both rules that are only efficient cracking the phpbb hashlist
Code:
^b ^b ^p ^h ^p ] ] ] ]
] ] ] $p $h $p $b $b
Now we need to run again to see what we lost. In theory, should be zero:
Code:
$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule
Quote:...
Recovered.: 17050
Zero! Toda!
3. Regroup them
By doing this I mean I am doing no changes at all to the rules itself. But you will see, it gives a great new view on everything.
Code:
## nothing, reverse, case... base stuff
:
r
u
T0
## simple number append
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9
## special number append
$0 $0
$0 $1
$0 $2
$1 $1
$1 $2
$1 $3
$2 $1
$2 $2
$2 $3
$6 $9
$7 $7
$8 $8
$9 $9
$1 $2 $3
## high frequency append
$e
$s
## high frequency overwrite at end
] $a
] ] $s
] ] $a
] ] $e $r
] ] $i $e
] ] ] $o
] ] ] $y
] ] ] $1 $2 $3
] ] ] $m $a $n
] ] ] $d $o $g
## high frequency prepend
^1
^e ^h ^t
## high frequency overwrite at start
o0b
o0d
o0m o1a
## leetify
so0
si1
se3
## simple extracts
D2
D2 D2
D3
D4
## undouble word
'5 D3
'5 $1
## removes suffixes from 'strongified' passwords in dict
]
] ]
] ] ]
] ] ] d
] ] D1 ]
## rotates
+5 ] } } } } '4
x02 { { { { { {
} ] ] {
} } -0 x12
} } }
} } } } '4
} } } } } '5
} } } } } } Y4 '4 d
## unknown
*04 +0 '4
*05 x03 d '3 p1
+0 +0 +0 +0 +0 +0 +0 +0
+0 +0 +0 x12
Z4 '8 x42
Z5 '6 x31 ] p1
Z5 *75 '5 { x02
d x28 Y4 '4 d
f *A5 '8 x14
p2 '7 p1 x58
x14 d p2 '6
Verify results:
Code:
$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule
Quote:...
Recovered.: 17050
Perfect!
4. Reduce the 78 rules down to 64 by removing the rules that are less efficient
It is possible that two or more different rules can crack the same hash just because of the many words in our dictionary.
So what we focus in are all these hashes that have been cracked by only one single rule.
This will happend more or less often for each rule. We add a counter for each rule and with each hit we increase it.
When we are finished we simply sort out these rules that have the lowest number in their counter.
To do this, you can read arex way of doing it, see his writeup:
http://thepasswordproject.com/2012-03-28..._of_hashes
5.
Result
Here is our final new best64.rule:
Code:
## nothing, reverse, case... base stuff
:
r
u
T0
## simple number append
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9
## special number append
$0 $0
$0 $1
$1 $1
$1 $2
$2 $1
$2 $2
$2 $3
$6 $9
$7 $7
$8 $8
$9 $9
$1 $2 $3
## high frequency append
$s
## high frequency overwrite at end
] ] $a
] ] $e $r
] ] $i $e
] ] ] $o
] ] ] $y
] ] ] $1 $2 $3
] ] ] $m $a $n
## high frequency prepend
^1
^e ^h ^t
## leetify
so0
si1
se3
## simple extracts
D2 D2
D3
D4
## undouble word
'5 $1
## removes suffixes from 'strongified' passwords in dict
]
] ]
] ] ]
] ] ] d
] ] D1 ]
## rotates
+5 ] } } } } '4
x02 { { { { { {
} ] ] {
} } -0 x12
} } } } '4
} } } } } '5
} } } } } } Y4 '4 d
## unknown
*04 +0 '4
*05 x03 d '3 p1
+0 +0 +0 +0 +0 +0 +0 +0
Z4 '8 x42
Z5 *75 '5 { x02
d x28 Y4 '4 d
f *A5 '8 x14
p2 '7 p1 x58
x14 d p2 '6
And our final value:
Quote:...
Recovered.: 15357
The original best64.rule was:
Quote:...
Recovered.: 12345
The original best64.rule was already very efficient on the random hashlist.
The new one is
24.39% more efficient.
Again, thanks to all participating in the contest.
