increasing threads for stdout?

[Kgx Pnqvhm]

I haven't run hashcat to generate mangles for quite a while, but trying the new 0.45 (hashcat-cli64.exe) in --stdout mode piped to a file seemed to run slower than I remembered, and I noticed it was only running 3 threads.

Isn't the default 8 threads?

So I tried --threads=32 and still only got 3 threads.

Is it ignoring the setting due to being in --stdout mode?
And would --segment-size also be ignored?

[philsmd]

The main problem w/ more threads in stdout mode is that the bottleneck for this mode is indeed *the* stdout. What I mean is, the process of writing to stdout slows down the whole process group that much that it makes absolutely *no* sense to use additional threads.

So -n option w/ --stdout shoudn't be used since the number of thread will be automatically set/reduced to 3 anyway (because of the output bottleneck).

[Kgx Pnqvhm]

I remember this from the original thread about that problem from way back.

So my question would be what settings can we change to make the --stdout usage for generating the list of mangles faster?

[philsmd]

I think it is just impossible to significantly increase the speed (and for sure increasing the # of threads doesn't help at all).

I'm not sure if you understood what I meant by saying the stdout is the problem.
Think like this: there is a worker that is *very*,very fast, but the output that he hands over to the input/output (I/O) - to the operating system let's say - will need much time to get printed. The threads need to "wait" until I/O reacts/did it's job... this is what we call a bottleneck and there is kind nothing we can do about it... there is too much to print and too much I/O involved ... also if we had used additional threads we need to wait again for I/O... so more threads do not help.

Faster I/O may help ... so a *really* slow output could depend on what you do with it (e.g. your pipe, your disk - if you use redirect - , it could also depend on operating system, how good it is to handle I/O etc etc).

[Kgx Pnqvhm]

Maybe a formal specified mangle output file mode is needed, as opposed to stdout.

[epixoip]

the speed of --stdout has not changed. it helps to actually measure, rather than relying on perception.

0.44 without rules

Code:

epixoip@db:~/hashcat-0.44$ ./hashcat-cli64.bin --stdout ~/wordlists/rockyou.txt | pv -l >/dev/null
14.3M 0:02:57 [80.7k/s]

0.45 without rules

Code:

epixoip@db:~/hashcat-0.45$ ./hashcat-cli64.bin --stdout ~/wordlists/rockyou.txt | pv -l >/dev/null
14.3M 0:02:57 [80.7k/s]

0.44 with rules

Code:

epixoip@db:~/hashcat-0.44$ ./hashcat-cli64.bin --stdout ~/wordlists/top20k.txt -r rules/best64.rule | pv -l >/dev/null
1.66M 0:00:20 [81.1k/s]

0.45 with rules

Code:

epixoip@db:~/hashcat-0.45$ ./hashcat-cli64.bin --stdout ~/wordlists/top20k.txt -r rules/best64.rule | pv -l >/dev/null
1.66M 0:00:20 [81.0k/s]

since you are writing to a file, it would seem that your hard drive or filesystem is the bottleneck here, not hashcat.

also, it could be that the rules that you are testing are un-optimized and slow.

[Kgx Pnqvhm]

What was more fresh in my mind that that the same dictionary and ruleset took a few seconds to run as an attack, but several hours to output the mangles to a file.

As pointed out, stdout isn't really made for high speed I/O. And atom hasn't optimized hashcat for outputting mangles, but for doing actual attacks.

The hashcat utilities mp and sp have a formal specified output file, and are very fast.

What might be needed is, rather than fiddle with hashcat itself, have just the rule engine in a separate mp or sp style program, for the express purpose of generating the mangles.

(The big picture of this question, and my JtR memory commands question in another thread, is rather than running my rules against normalized dictionaries, the running of those cracked word lists against someone else's rules. I think there are a lot of duplicates, so a lot of wasted time, but really can't "see" it, unless I can get the actual mangles to examine.)

[epixoip]

What was more fresh in my mind that that the same dictionary and ruleset took a few seconds to run as an attack, but several hours to output the mangles to a file.

Well yes, that's absolutely to be expected.

As pointed out, stdout isn't really made for high speed I/O.

You misinterpreted what philsmd said. There's nothing special about stdout, it's just a file. It's not that writing to stdout is slow, it's that writing in general is slow. What slows down stdout specifically depends on what is reading from stdout, or where you are redirecting stdout. But this, as philsmd said, is unavoidable, since this is the task at hand.

And atom hasn't optimized hashcat for outputting mangles, but for doing actual attacks.

This is not true, atom actually has greatly optimized --stdout. It's as optimized as it's ever going to be.

The hashcat utilities mp and sp have a formal specified output file, and are very fast.

This is not why mp and sp are faster. Again, stdout is just a file. All three programs write to files in the same way, regardless of what the file is.

The reason mp and sp are faster is because they generate candidates internally, using a static, compiled algorithm. The rule engine generates candidates using a dynamic, interpreted scripting language. It also has to do simultaneous reads and writes. So this will of course be much slower.

What might be needed is, rather than fiddle with hashcat itself, have just the rule engine in a separate mp or sp style program, for the express purpose of generating the mangles.

The speed would be the same regardless.

[Kgx Pnqvhm]

Watching my taskbar HD LED while hashcat is doing the mangles shows very infrequent writes, so quite a bit of processing is happening between the writes, so I still don't think the I/O involving my HD is the bottleneck.
If it was showing full time writing, I could believe that the HD wasn't keeping up.

UPDATE: Re-trying the same task that took hours the night I first made this post took just minutes, so something must'be been wrong with my system then.
First, I ran mp64.exe to watch how it used system resources. It used less CPU than hashcat, around 10%, but wrote to disk continuously.
Then, I re-ran the same rule & dict combo that I did the other might, that took hours, with very little disk writing.
This time it was writing briefly about once a second, and was using the same 25% CPU like the other night.
And it "felt" as fast as I expected, meaning a brief wait rather than watching a 2 hour movie and going to bed.

I have almost any scheduled task that Windows want to do disabled, so don't know what could have been preventing hashcat from running properly.

[Kgx Pnqvhm]

Upon attempting to boot my computer today, I got beeps indicating a RAM problem.
Pulling out and re-seating all the RAM modules let me boot.
Maybe something was flaky with my RAM the last couple of days.
There had to be something funny going on for a half megabyte file to take eight hours to write that night.