statsprocessor v0.07

[atom]

Hey Guys,

statsprocessor v0.07 is ready!

It can fully replace Brute-Force since it covers the full keyspace

NOTICE -t 0 for BF

Code:

root@sf:~# /root/maskprocessor-0.68/mp64.bin ?l?l?l?l --increment-min 4 --increment-max 4 | wc -l
456976
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l -t 0 --pw-min 4 --pw-max 4| wc -l
456976

Generated candidates are guaranteed to be unique

NOTICE the sort -u

Code:

root@sf:~# /root/maskprocessor-0.68/mp64.bin ?l?l?l?l --increment-min 4 --increment-max 4 | sort -u | wc -l
456976
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l -t 0 --pw-min 4 --pw-max  4 | sort -u | wc -l
456976

.. and here the chksum to prove its the same data

Code:

root@sf:~# /root/maskprocessor-0.68/mp64.bin ?l?l?l?l --increment-min 4 --increment-max 4 | sort -u | md5sum
84be7139d238465d22a4da2d3cb69d5a
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l -t 0 --pw-min 4 --pw-max 4 | sort -u | md5sum
84be7139d238465d22a4da2d3cb69d5a

Statistically based on specific input (dictionary) -> Creates .hcstat file

Code:

root@sf:~# /root/hashcat-utils-0.8/hcstatgen.bin rockyou.hcstat < /root/dict/untouched/rockyou.txt
...
root@sf:~# /root/hashcat-utils-0.8/hcstatgen.bin mil-dic.hcstat < /root/dict/untouched/mil-dic.txt
...
root@sf:~# /root/hashcat-utils-0.8/hcstatgen.bin facebook-firstnames.hcstat < /root/dict/untouched/facebook-firstnames.txt
...

This way you can optimize your attack by using your cracked list as input

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin rockyou.hcstat ?l?l?l?l?l --pw-min 5 | head -3
marin
sarin
anger
root@sf:~# /root/statsprocessor-0.06/sp64.bin mil-dic.hcstat ?l?l?l?l?l --pw-min 5 | head -3
saran
maran
alane
root@sf:~# /root/statsprocessor-0.06/sp64.bin facebook-firstnames.hcstat ?l?l?l?l?l --pw-min 5 | head -3
shana
marin
anana

Generator could be called AI since it applies statistics that are tricky and it actually prints out -real- words at the beginning

I think this is one of the strongest features. Havent seen anything similar before

root@sf:~# /root/statsprocessor-0.06/sp64.bin rockyou.hcstat ?l?l?l?l?l --pw-min 5 | head -30
marin
sarin
anger
chana
barin
jarin
lonan
tarin
darin
parin
karin
ronan
narin
garin
harin
elona
ilona
farin
warin
vinan
orina
yarin
zarin
unger
xxcke
qunan
minan
shana
alona
carin

Generator is still configurable due to the --threshold parameter

This way you can kick out guesses that are very unlikely to happen which reduces the keyspace, which makes the attack more fast

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l --pw-min 4 -t 26 | wc -l
456976
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l --pw-min 4 -t 22 | wc -l
234256

I have reduced to threshold to 22 (from 26) but saved nearly half of the full time to check it but just ignoring the 4 of them

Masks still can be used, but they act as filter-overlay - In case you dont want to miss the strength of Mask-Attack

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?u?u?u?u -t 0 --pw-min 4 | wc -l
456976
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?l?l -t 0 --pw-min 4 | wc -l
456976
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d?d?d -t 0 --pw-min 4 | wc -l
10000
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l     -t 0 --pw-min 2 | wc -l
676
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?d   -t 0 --pw-min 3 | wc -l
6760
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l?d?d -t 0 --pw-min 4 | wc -l
67600

Of course, adding a mask is not required. Its optional. It still works without. In case you omit the mask ?a?a?a?a?a... is used

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat -t 0 --pw-min 2 --pw-max 2 | wc -l
9025

Supports automatic increment with --pw-min and --pw-max

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l -t 0 --pw-min 1 | wc -l
26
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l -t 0 --pw-min 2 | wc -l
676
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?l?l -t 0 --pw-min 1 | wc -l
702

Restorable with -s

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d -t 0 --pw-min 2 | wc -l
100
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d -t 0 --pw-min 2 -s 10 | wc -l
90
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d -t 0 --pw-min 2 -s 20 | wc -l
80

Limitable with -l

... and therefore distributable (to multiple computers)

Code:

root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d -t 0 --pw-min 2 -l 10 | wc -l
10
root@sf:~# /root/statsprocessor-0.06/sp64.bin /root/atom.hcstat ?d?d -t 0 --pw-min 2 -s 20 -l 30 | wc -l
10

GPGPU compatible, will be merged into oclHashcat-lite soon

I will port this version to oclHashcat-lite v0.10. It will replace Mask-Attack as its default attack-mode.

There will be the same -t parameter to have similar syntax.

--

Enough said, download it here: http://hashcat.net/files/statsprocessor-0.08.7z

--
atom

[M@LIK]

W0W!

[Hash-IT]

atom,

I don't pretend for a minute to fully understand what you have done. However I know it is awesome, very awesome and I am extremely grateful we have you, to not only come up with this stuff but to share it freely.

Thank you very much for your time, we all owe you a huge thanks.

[thorsheim]

atom;
Did my very first run of hcstat & statsprocessor, and I have my first question / feature request for you:
I'd like some way to actually see the per-position stats in readable text. I wonder if this could be visualized in some easy way?

My presentation from Passwords^11 (http://www.slideshare.net/perthorsheim/b...statistics), slide number 13, show per-position entropy - or simply number of unique characters per position. The pattern seem to repeat across multiple datasets, although they are all LM/NTLM sets from domains with enforced complexity rules.

My colleague @KluZz (Jan Fredrik Leversund) also made me some code to simply output a table showing number of occurences per character per position, as I've written about before.

--
A next evolution to this would of course be to either develop or grab existing code that gives us stats on 'which word usually comes after word x', where we analyze either passphrases or plain language from books etc, to enable cracking of long pass phrases. With/out using rules for adjusting to complexity and/or mangling rules, Re:http://securitynirvana.blogspot.no/2012/05/challenge-received.html (which remains uncracked. Unsalted MD5, length 16-24....)

[undeath]

are you looking for something like passpal? http://thepasswordproject.com/passpal

[atom]

A next evolution to this would of course be to either develop or grab existing code that gives us stats on 'which word usually comes after word x', where we analyze either passphrases or plain language from books etc, to enable cracking of long pass phrases. With/out using rules for adjusting to complexity and/or mangling rules, Re:http://securitynirvana.blogspot.no/2012/05/challenge-received.html (which remains uncracked. Unsalted MD5, length 16-24....)

Hmm a markov-model for whole words? Sounds interessting but would require a bit a different concept. I am not sure yet how this could be done but I think with enough effort its possible, even on GPU. Maybe in a later version and thanks for the input!

[thorsheim]

Hmm a markov-model for whole words? Sounds interessting but would require a bit a different concept. I am not sure yet how this could be done but I think with enough effort its possible, even on GPU. Maybe in a later version and thanks for the input!

Absolutely something different, yes. Then again; my Swiftkey software on my Android phone analyses my sms inbox and more, and then does both word prediction and 'next word' prediction based on the statistics it generates.

Unless you do correct horse battery staples passphrases, but something more logical - even as simple as standard sentences with correct grammar, this would eventually be very effective against natural language passphrases.

I'm not going to claim any original idea here, just going back to the xkcd 936 discussions, including this interesting blog post from @davienthemoose: http://secmoose.blogspot.no/2011/08/why-...-suck.html

[korsa741]

hello, how can generate rokyou.hcstat for only numbers min 15 max 15

[The Mechanic]

hello, how can generate rokyou.hcstat for only numbers min 15 max 15

Thats done through the command line options, --pw-min=15 --pw-max=15 Your not generating a file for a specific length, just statistic file for most likely occurrence, the file works for any length program is capable of generating.

[korsa741]

thanks the mechanic