Official Best64 Challenge Thread

[atom]

This is the write-up how I rebuild the new best64.rule out of all the submissions but for real-life use.

This new best64.rule will go into all hashcat distributions.

I took arex best 80's results posted here: http://thepasswordproject.com/2012-03-28...ompetition

1. Check how useful it really is

To do this i made a MD5 hashlist of rockyou.txt which is the ultimate real-life password resource we have.

Code:

$ cat /root/dict/untouched/rockyou.txt | perl /root/dict2hash.pl > rockyou.md5

The resulting hashlist has 14 million hashes. This is to much.

Code:

$ wc -l rockyou.md5

14344391 rockyou.md5

I reduced them a bit by randomly fetching out 1 million hashes. This is the hashlist we will work with.

Code:

$ sort -R rockyou.md5 | head -1000000 > rockyou.md5.1000000
$ wc -l rockyou.md5.1000000

1000000 rockyou.md5.1000000

Now run the top 10k dictionary and the best 80 rules from the contest against it.

Code:

$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80.rule

...
Recovered.: 17097

Here is the original best64.rule. We will use it later for comparison with the final result:

Code:

$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r rules/best64.rule

...
Recovered.: 12345

Funny number.. Well, but thats it for real

2. Clean up arex's rules

There is a lot of useless stuff inside. Will do:

• Deobfuscate rules, stuff like ^7 { which is actually just $7
  
• Remove dupes that come out after deobfuscation.
  
• Remove as much as possible hashcat specific functions to make resulting hashlist jtr/pwp compatible.
  
• Remove useless functions to increase performance while rules are applied at runtime in rule-engine: Making it faster!
  

Code:

$2$4]]                  :
$4]]]                   ] ]
T0                      T0
$1$2i42@2               $1
i42@2T0u                u
}}D1{                   ]
$si42@2                 $s
]]]d                    ] ] ] d
$3$2]]]]]               ] ] ]
krKz3x13                r
i42@2$2                 $2
}}}}'4                  } } } } '4
:so0                    so0
$0$1i42@2               $0 $1
}]]{                    } ] ] {
$1$2$3*22               $1 $2 $3
i42@2$9Z1               $9 $9
lx31                    D3
$b]$7                   $7
dx28Y4'4d               d x28 Y4 '4 d
}}}}}'5                 } } } } } '5
z3x13$3                 $3
^5{                     $5
$6 $9                   $6 $9
f*A5'8x14               f *A5 '8 x14
si1                     si1
z3x13$4                 $4
$0$0$7}[                $0 $0
]]$a                    ] ] $a
]]]$m$a$n               ] ] ] $m $a $n
+5]}}}}'4               +5 ] } } } } '4
*05x03d'3p1             *05 x03 d '3 p1
]]]$1$2$3               ] ] ] $1 $2 $3
i42@2$1$1               $1 $1
$9li42@2                $9
$1 $2                   $1 $2
]]]$y                   ] ] ] $y
^b^b^p^h^p]]]]          ^b ^b ^p ^h ^p ] ] ] ]
$2$2'5$1                '5 $1
Z4'8x42                 Z4 '8 x42
$0                      $0
$6                      $6
i42@2$2$3               $2 $3
$2 $2                   $2 $2
^&x03{{{{{{             x02 { { { { { {
$1]]]$e$r               ] ] $e $r
se3                     se3
] ] $i $e               ] ] $i $e
]]l]$o                  ] ] ] $o
$8z3x13                 $8
i01i42@2                ^1
}}}}}}Y4'4d             } } } } } } Y4 '4 d
$2 $1                   $2 $1
^t@tD2D2                D2 D2
{{{{{{{{x31'4           '5 D3
p2'7p1x58               p2 '7 p1 x58
]]D1]                   ] ] D1 ]
*44D4                   D4
+0+0+0+0+0+0+0+0        +0 +0 +0 +0 +0 +0 +0 +0
*04+0'4                 *04 +0 '4
$8$8                    $8 $8
x14dp2'6                x14 d p2 '6
]]]l$p$h$p$b$b          ] ] ] $p $h $p $b $b
Z5*75'5{x02             Z5 *75 '5 { x02
$7$7                    $7 $7
x21i42@2                D2
^e^h^ti42@2             ^e ^h ^t
] ] ] $d $o $g          ] ] ] $d $o $g
}}-0x12                 } } -0 x12
o0to0b                  o0b
l[^d^t@t                [ ^d
$e                      $e
l]$sl]$a                ] $a
$.]]l]$s                ] ] $s
+0+0+0x12               +0 +0 +0 x12
:}}}                    } } }
$1$3                    $1 $3
o1ao0ml                 ^m ^a
Z5'6x31]p1              Z5 '6 x31 ] p1
$0$2                    $0 $2

Some rules could not get deobfuscated/optimized.

Some of the changes might have an influence on the cracked results since I removed lots of stuff, too.

To make sure the changes are not so bad, here is a backup run to see its not so far away from the 17097 from the original set.

Code:

$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule

...
Recovered.: 17050

OK, thats fine

2. Remove the both rules that are only efficient cracking the phpbb hashlist

Code:

^b ^b ^p ^h ^p ] ] ] ]
] ] ] $p $h $p $b $b

Now we need to run again to see what we lost. In theory, should be zero:

Code:

$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule

...
Recovered.: 17050

Zero! Toda!

3. Regroup them

By doing this I mean I am doing no changes at all to the rules itself. But you will see, it gives a great new view on everything.

Code:

## nothing, reverse, case... base stuff
:
r
u
T0

## simple number append
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9

## special number append
$0 $0
$0 $1
$0 $2
$1 $1
$1 $2
$1 $3
$2 $1
$2 $2
$2 $3
$6 $9
$7 $7
$8 $8
$9 $9
$1 $2 $3

## high frequency append
$e
$s

## high frequency overwrite at end
] $a
] ] $s
] ] $a
] ] $e $r
] ] $i $e
] ] ] $o
] ] ] $y
] ] ] $1 $2 $3
] ] ] $m $a $n
] ] ] $d $o $g

## high frequency prepend
^1
^e ^h ^t

## high frequency overwrite at start
o0b
o0d
o0m o1a

## leetify
so0
si1
se3

## simple extracts
D2
D2 D2
D3
D4

## undouble word
'5 D3
'5 $1

## removes suffixes from 'strongified' passwords in dict
]
] ]
] ] ]
] ] ] d
] ] D1 ]

## rotates
+5 ] } } } } '4
x02 { { { { { {
} ] ] {
} } -0 x12
} } }
} } } } '4
} } } } } '5
} } } } } } Y4 '4 d

## unknown
*04 +0 '4
*05 x03 d '3 p1
+0 +0 +0 +0 +0 +0 +0 +0
+0 +0 +0 x12
Z4 '8 x42
Z5 '6 x31 ] p1
Z5 *75 '5 { x02
d x28 Y4 '4 d
f *A5 '8 x14
p2 '7 p1 x58
x14 d p2 '6

Verify results:

Code:

$ ./hashcat-cli64.bin rockyou.md5.1000000 passwords_top10k.dict.txt -o y -r best80-de.rule

...
Recovered.: 17050

Perfect!

4. Reduce the 78 rules down to 64 by removing the rules that are less efficient

It is possible that two or more different rules can crack the same hash just because of the many words in our dictionary.
So what we focus in are all these hashes that have been cracked by only one single rule.
This will happend more or less often for each rule. We add a counter for each rule and with each hit we increase it.
When we are finished we simply sort out these rules that have the lowest number in their counter.

To do this, you can read arex way of doing it, see his writeup: http://thepasswordproject.com/2012-03-28..._of_hashes

5. Result

Here is our final new best64.rule:

Code:

## nothing, reverse, case... base stuff
:
r
u
T0

## simple number append
$0
$1
$2
$3
$4
$5
$6
$7
$8
$9

## special number append
$0 $0
$0 $1
$1 $1
$1 $2
$2 $1
$2 $2
$2 $3
$6 $9
$7 $7
$8 $8
$9 $9
$1 $2 $3

## high frequency append
$s

## high frequency overwrite at end
] ] $a
] ] $e $r
] ] $i $e
] ] ] $o
] ] ] $y
] ] ] $1 $2 $3
] ] ] $m $a $n

## high frequency prepend
^1
^e ^h ^t

## leetify
so0
si1
se3

## simple extracts
D2 D2
D3
D4

## undouble word
'5 $1

## removes suffixes from 'strongified' passwords in dict
]
] ]
] ] ]
] ] ] d
] ] D1 ]

## rotates
+5 ] } } } } '4
x02 { { { { { {
} ] ] {
} } -0 x12
} } } } '4
} } } } } '5
} } } } } } Y4 '4 d

## unknown
*04 +0 '4
*05 x03 d '3 p1
+0 +0 +0 +0 +0 +0 +0 +0
Z4 '8 x42
Z5 *75 '5 { x02
d x28 Y4 '4 d
f *A5 '8 x14
p2 '7 p1 x58
x14 d p2 '6

And our final value:

...
Recovered.: 15357

The original best64.rule was:

...
Recovered.: 12345

The original best64.rule was already very efficient on the random hashlist.

The new one is 24.39% more efficient.

Again, thanks to all participating in the contest.