statsprocessor v0.03
#1
THIS THREAD IS OUTDATED, PLS CONTINUE HERE: https://hashcat.net/forum/thread-1285.html

I'm very proud to present you a new toy: statsprocessor

Download here: http://hashcat.net/files/statsprocessor-0.04.7z

This is a standalone word-generator like the maskprocessor, but based on markov-attack!

The markov-attack is a statistically based brute-force like attack, but instead of specifying a charset or a mask, we specify a file.

This file is once generated by ourself in a previous step. It contains statistical informations which are made out an automated analysis of a given dictionary.

To do the analysis we use another new tool called "hcstatgen" which is part of the new hashcat-utils package, starting with version 0.6.

Download 32 bit here: http://hashcat.net/files/hashcat-utils-0.7-32.7z
Download 64 bit here: http://hashcat.net/files/hashcat-utils-0.7-64.7z

The second program, the new tool "statprocessor", generates the words based on the statistical order of the .hcstat file.



NOTE: In Brute-Force Attack or in Mask Attack we can limit the keyspace by setting a smaller charset in order to reduce the attack-time.

In Markov Attack we have something similar, the "threshold". All you do is to specify a number. The higher the number, the higher the threshold to add a new link between two characters on the two-level table on which the markov-attack bases on.

This background is not so important, just remember that the higher the value, the smaller the keyspace and thus the faster the attack.

If you set the threshold to 0 its like a real Brute-Force attack but with markov ordering.



Here is a example usage:

Generate the .hcstat out of the rockyou.txt dictionary

Code:
root@sf:~/statsprocessor-0.01# /root/hashcat-utils-0.6/hcstatgen.bin rockyou.hcstat < /root/dict/untouched/rockyou.txt
Reading input...
Sorting tables...
Writing stats...

NOTE: I will add rockyou.hcstat as an example to the distribution so that new users have something to instantly play with.

Now lets run the statprocessor just to see what comes out. I will set an totally oversized threshold which is not for concrete usage but this way the output is so small that you will get an idea of how it looks like

Code:
root@sf:~/statsprocessor-0.01# ./sp64.bin --pw-min 5 --pw-max 5 --threshold 400000 rockyou.hcstat | head -20
anana
anane
anani
anara
anare
anari
anala
anali
anale
analo
aname
anama
anami
anamo
anaml
anasa
anat
anena
anene
anera

I will explain a bit in detail next. This if for those who want to know how markov-attack works internally. Otherwise you can skip to the next command.

As you can see, it starts with the char "a". That is because "a" is the most used char in rockyou.txt.

The first character is what we call the root-character. All the following char are looked up from the markov-table.

The table is an 1-to-n table. The key can be any char. For this char, the table holds an ORDERED array structure which char follows next.

From what we see here is that the char "n" is the most followed char after the char "a". Because of some statistical accident the char "a" is followed most by the char "n", so it ends up in a loop.

In the next iteration, the 2nd most followed char after "n" is the "e", and so on.

As you can see after the word "anasa" there is a four char word "anat" following, even though we specified --pw-min 5.

This happend because of the high threshold. There is simply no char following the "t" char more than 400000 times in rockyou.txt.

Since we always focus on performance, see how fast this beast is

Code:
root@sf:~/statsprocessor-0.01# ./sp64.bin --pw-min 5 --pw-max 5 rockyou.hcstat | wc -l    
1634805578
root@sf:~/statsprocessor-0.01# time ./sp64.bin -o /dev/null --pw-min 5 --pw-max 5 rockyou.hcstat        

real    0m18.234s
user    0m17.981s
sys     0m0.244s

This makes it 89.6 Mplains/s fast - on a single cpu core!

This 89.6 Mplains/s is fast enough to feed oclHashcat-plus for slow algorithms like DCC2 or WPA/WPA2. Here is an example run

Code:
root@sf:~/oclHashcat-plus-0.09# /root/statsprocessor-0.01/sp64.bin --pw-min 8 /root/statsprocessor-0.01/rockyou.hcstat | ./oclHashcat-plus64.bin -m 2500 test.hccap -n 80

...

Starting attack in wordlist stdin mode...    

Status.......: Running
Input.Mode...: Pipe
Hash.Target..: hashcat (00:aa:aa:aa:aa:aa <-> aa:aa:aa:aa:aa:aa)
Hash.Type....: WPA/WPA2
Time.Running.: 10 secs
Time.Util....: 10001.1ms/241.1ms Real/CPU, 2.5% idle
Speed........:   172.0k c/s Real,   181.3k c/s GPU
Recovered....: 0/1 Digests, 0/1 Salts
Progress.....: 1474560
Rejected.....: 0
HWMon.GPU.#1.: 98% Util, 60c Temp, 84% Fan
HWMon.GPU.#2.: 98% Util, 60c Temp, 30% Fan

As you can see, this is full speed for my hd6990.

But what if we want to use this with fast algorithms like MD5/SHA1? Well that would require a special new attack-mode for oclHashcat-lite or oclHashcat-plus. I will defenitly add this to a later version. Till that you can workaround this just by adding some workload to each generated word

Code:
$ /root/statsprocessor-0.01/sp64.bin --pw-min 6 /root/statsprocessor-0.01/rockyou.hcstat | ./oclHashcat-plus64.bin -n 80  example0.hash -r rules/hybrid/prepend_ld.rule -r rules/hybrid/prepend_ld.rule

...

Starting attack in wordlist stdin mode...    

007f8558448eba6c4f1d3fe33a0e52f7:jaanammi    
d77a07d8643df8b91975c5f3f8a582ee:brant124    
f8686f2dde3442b22166eac8bd0fa08b:c2anthea    
ec0962c15a328520b65d6172b8746774:alan1043    
63af54f143bd08246a78d371fc78eb25:deansail    
fc3d0a9d96d9a64722e05694033a8e05:vianchen    
3819f84ff69eab96e390816dba69481c:beans830

Well you got the idea...

NOTE
  • This is a very strong attack-mode. I think you will have a lot of fun with it.
  • Increment mode is already added. Like always, you can specify --pw-min and --pw-max.
  • The default threshold is set to 48.
#2
I didn't understand everything xD
But this looks very interesting!
Good job atom!
#3
Wow..

M@LIK might not have understood everything but I didn't understand anything ! Big Grin

However it looks great !! Thank you very much atom !
#4
Great! more tools to play with thanks atom
#5
Very nice Atom! :-)

I'm playing with it already... gonna be fun.
#6
All thing flies by over my head : lol
Very Nice Atom,A very handy tool indeed
☫
#7
In the meanwhile I've finished v0.02. This one accepts an optional typical hashcat mask.

Download it here: http://hashcat.net/files/statsprocessor-0.02.7z

This can be used as a filter. It is best explained with two examples:

Without filter-mask (as in v0.01)

Code:
root@sf:~/statsprocessor-0.02# ./sp64.bin --pw-min 3 rockyou.hcstat | head -10
ana
ane
ani
ang
and
ann
ano
ant
an1
ank

With new filter-mask

Code:
root@sf:~/statsprocessor-0.02# ./sp64.bin --pw-min 3 rockyou.hcstat ?l?l?d | head -10
an1
an2
an0
an3
an4
an9
an8
an7
an6
an5
#8
Steady on atom !!!

I've not learned how to use the first one properly yet !!! Big Grin
#9
awsome. thanks
#10
Changes:
  • Added -t shortcut for --threshold
  • Added --reverse parameter which lets the markov generator generate the words from right to left, not from left to right.
    Code:
    root@sf:~/statsprocessor-0.03# ./sp64.bin --pw-min 4 -t 4 rockyou.hcstat | head -10          
    anan
    anar
    anal
    anam
    anen
    aner
    anel
    anem
    anir
    anil
    root@sf:~/statsprocessor-0.03# ./sp64.bin --pw-min 4 -t 4 rockyou.hcstat --reverse | head -10
    nana
    rana
    lana
    mana
    nena
    rena
    lena
    mena
    rina
    lina
  • Changed how that most important parameter "--threshold" is interpreted by statsprocessor to a bit more easy-to-understand usage. You just define a maximum. This is the maximum number of chars which are added to the markov-table. One exception: In case of "0" it will allow all chars and you will end up in an ordered brute-force.
    Code:
    root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 1 rockyou.hcstat | head -10
    a
    an
    ana
    anan
    anana
    ananan
    ananana
    anananan
    anananana
    ananananan
    root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 2 rockyou.hcstat | head -10
    a
    e
    an
    ar
    en
    er
    ana
    ane
    ara
    are
    root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 0 rockyou.hcstat | head -10
    a
    e
    1
    0
    i
    2
    o
    n
    r
    l
    [*]
  • Added parameter -s and -l like in maskprocessor. Well you can also call it support for restore and distribution
    Code:
    root@sf:~/xy/statsprocessor-0.03# ./sp64.bin rockyou.hcstat ?a?a?a -s ctu -l ctk
    ctu
    cts
    ct1
    ctc
    ctl
    ctm
    ct2
    ctz
    ctb
    ct0
    ctw
    ctn
    ctk
    [*]