statsprocessor v0.03
#1
THIS THREAD IS OUTDATED, PLS CONTINUE HERE: https://hashcat.net/forum/thread-1285.html

I'm very proud to present you a new toy: statsprocessor

Download here: http://hashcat.net/files/statsprocessor-0.04.7z

This is a standalone word-generator like the maskprocessor, but based on markov-attack!

The markov-attack is a statistically based brute-force like attack, but instead of specifying a charset or a mask, we specify a file.

This file is once generated by ourself in a previous step. It contains statistical informations which are made out an automated analysis of a given dictionary.

To do the analysis we use another new tool called "hcstatgen" which is part of the new hashcat-utils package, starting with version 0.6.

Download 32 bit here: http://hashcat.net/files/hashcat-utils-0.7-32.7z
Download 64 bit here: http://hashcat.net/files/hashcat-utils-0.7-64.7z

The second program, the new tool "statprocessor", generates the words based on the statistical order of the .hcstat file.



NOTE: In Brute-Force Attack or in Mask Attack we can limit the keyspace by setting a smaller charset in order to reduce the attack-time.

In Markov Attack we have something similar, the "threshold". All you do is to specify a number. The higher the number, the higher the threshold to add a new link between two characters on the two-level table on which the markov-attack bases on.

This background is not so important, just remember that the higher the value, the smaller the keyspace and thus the faster the attack.

If you set the threshold to 0 its like a real Brute-Force attack but with markov ordering.



Here is a example usage:

Generate the .hcstat out of the rockyou.txt dictionary

Code:
root@sf:~/statsprocessor-0.01# /root/hashcat-utils-0.6/hcstatgen.bin rockyou.hcstat < /root/dict/untouched/rockyou.txt Reading input... Sorting tables... Writing stats...

NOTE: I will add rockyou.hcstat as an example to the distribution so that new users have something to instantly play with.

Now lets run the statprocessor just to see what comes out. I will set an totally oversized threshold which is not for concrete usage but this way the output is so small that you will get an idea of how it looks like

Code:
root@sf:~/statsprocessor-0.01# ./sp64.bin --pw-min 5 --pw-max 5 --threshold 400000 rockyou.hcstat | head -20 anana anane anani anara anare anari anala anali anale analo aname anama anami anamo anaml anasa anat anena anene anera

I will explain a bit in detail next. This if for those who want to know how markov-attack works internally. Otherwise you can skip to the next command.

As you can see, it starts with the char "a". That is because "a" is the most used char in rockyou.txt.

The first character is what we call the root-character. All the following char are looked up from the markov-table.

The table is an 1-to-n table. The key can be any char. For this char, the table holds an ORDERED array structure which char follows next.

From what we see here is that the char "n" is the most followed char after the char "a". Because of some statistical accident the char "a" is followed most by the char "n", so it ends up in a loop.

In the next iteration, the 2nd most followed char after "n" is the "e", and so on.

As you can see after the word "anasa" there is a four char word "anat" following, even though we specified --pw-min 5.

This happend because of the high threshold. There is simply no char following the "t" char more than 400000 times in rockyou.txt.

Since we always focus on performance, see how fast this beast is

Code:
root@sf:~/statsprocessor-0.01# ./sp64.bin --pw-min 5 --pw-max 5 rockyou.hcstat | wc -l 1634805578 root@sf:~/statsprocessor-0.01# time ./sp64.bin -o /dev/null --pw-min 5 --pw-max 5 rockyou.hcstat real 0m18.234s user 0m17.981s sys 0m0.244s

This makes it 89.6 Mplains/s fast - on a single cpu core!

This 89.6 Mplains/s is fast enough to feed oclHashcat-plus for slow algorithms like DCC2 or WPA/WPA2. Here is an example run

Code:
root@sf:~/oclHashcat-plus-0.09# /root/statsprocessor-0.01/sp64.bin --pw-min 8 /root/statsprocessor-0.01/rockyou.hcstat | ./oclHashcat-plus64.bin -m 2500 test.hccap -n 80 ... Starting attack in wordlist stdin mode... Status.......: Running Input.Mode...: Pipe Hash.Target..: hashcat (00:aa:aa:aa:aa:aa <-> aa:aa:aa:aa:aa:aa) Hash.Type....: WPA/WPA2 Time.Running.: 10 secs Time.Util....: 10001.1ms/241.1ms Real/CPU, 2.5% idle Speed........: 172.0k c/s Real, 181.3k c/s GPU Recovered....: 0/1 Digests, 0/1 Salts Progress.....: 1474560 Rejected.....: 0 HWMon.GPU.#1.: 98% Util, 60c Temp, 84% Fan HWMon.GPU.#2.: 98% Util, 60c Temp, 30% Fan

As you can see, this is full speed for my hd6990.

But what if we want to use this with fast algorithms like MD5/SHA1? Well that would require a special new attack-mode for oclHashcat-lite or oclHashcat-plus. I will defenitly add this to a later version. Till that you can workaround this just by adding some workload to each generated word

Code:
$ /root/statsprocessor-0.01/sp64.bin --pw-min 6 /root/statsprocessor-0.01/rockyou.hcstat | ./oclHashcat-plus64.bin -n 80 example0.hash -r rules/hybrid/prepend_ld.rule -r rules/hybrid/prepend_ld.rule ... Starting attack in wordlist stdin mode... 007f8558448eba6c4f1d3fe33a0e52f7:jaanammi d77a07d8643df8b91975c5f3f8a582ee:brant124 f8686f2dde3442b22166eac8bd0fa08b:c2anthea ec0962c15a328520b65d6172b8746774:alan1043 63af54f143bd08246a78d371fc78eb25:deansail fc3d0a9d96d9a64722e05694033a8e05:vianchen 3819f84ff69eab96e390816dba69481c:beans830

Well you got the idea...

NOTE
  • This is a very strong attack-mode. I think you will have a lot of fun with it.
  • Increment mode is already added. Like always, you can specify --pw-min and --pw-max.
  • The default threshold is set to 48.
#2
I didn't understand everything xD
But this looks very interesting!
Good job atom!
#3
Wow..

M@LIK might not have understood everything but I didn't understand anything ! Big Grin

However it looks great !! Thank you very much atom !
#4
Great! more tools to play with thanks atom
#5
Very nice Atom! :-)

I'm playing with it already... gonna be fun.
#6
All thing flies by over my head : lol
Very Nice Atom,A very handy tool indeed
☫
#7
In the meanwhile I've finished v0.02. This one accepts an optional typical hashcat mask.

Download it here: http://hashcat.net/files/statsprocessor-0.02.7z

This can be used as a filter. It is best explained with two examples:

Without filter-mask (as in v0.01)

Code:
root@sf:~/statsprocessor-0.02# ./sp64.bin --pw-min 3 rockyou.hcstat | head -10 ana ane ani ang and ann ano ant an1 ank

With new filter-mask

Code:
root@sf:~/statsprocessor-0.02# ./sp64.bin --pw-min 3 rockyou.hcstat ?l?l?d | head -10 an1 an2 an0 an3 an4 an9 an8 an7 an6 an5
#8
Steady on atom !!!

I've not learned how to use the first one properly yet !!! Big Grin
#9
awsome. thanks
#10
Changes:
  • Added -t shortcut for --threshold
  • Added --reverse parameter which lets the markov generator generate the words from right to left, not from left to right.
    Code:
    root@sf:~/statsprocessor-0.03# ./sp64.bin --pw-min 4 -t 4 rockyou.hcstat | head -10 anan anar anal anam anen aner anel anem anir anil root@sf:~/statsprocessor-0.03# ./sp64.bin --pw-min 4 -t 4 rockyou.hcstat --reverse | head -10 nana rana lana mana nena rena lena mena rina lina
  • Changed how that most important parameter "--threshold" is interpreted by statsprocessor to a bit more easy-to-understand usage. You just define a maximum. This is the maximum number of chars which are added to the markov-table. One exception: In case of "0" it will allow all chars and you will end up in an ordered brute-force.
    Code:
    root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 1 rockyou.hcstat | head -10 a an ana anan anana ananan ananana anananan anananana ananananan root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 2 rockyou.hcstat | head -10 a e an ar en er ana ane ara are root@sf:~/statsprocessor-0.03# ./sp64.bin --threshold 0 rockyou.hcstat | head -10 a e 1 0 i 2 o n r l [*]
  • Added parameter -s and -l like in maskprocessor. Well you can also call it support for restore and distribution
    Code:
    root@sf:~/xy/statsprocessor-0.03# ./sp64.bin rockyou.hcstat ?a?a?a -s ctu -l ctk ctu cts ct1 ctc ctl ctm ct2 ctz ctb ct0 ctw ctn ctk [*]