+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Deprecated; Previous versions (https://hashcat.net/forum/forum-29.html)
+--- Forum: Old hashcat Support (https://hashcat.net/forum/forum-20.html)
+--- Thread: increasing threads for stdout? (/thread-2335.html)
increasing threads for stdout? - Kgx Pnqvhm - 06-02-2013
I haven't run hashcat to generate mangles for quite a while, but trying the new 0.45 (hashcat-cli64.exe) in --stdout mode piped to a file seemed to run slower than I remembered, and I noticed it was only running 3 threads.
Isn't the default 8 threads?
So I tried --threads=32 and still only got 3 threads.
Is it ignoring the setting due to being in --stdout mode?
And would --segment-size also be ignored?
RE: increasing threads for stdout? - philsmd - 06-02-2013
The main problem w/ more threads in stdout mode is that the bottleneck for this mode is indeed *the* stdout. What I mean is, the process of writing to stdout slows down the whole process group that much that it makes absolutely *no* sense to use additional threads.
So -n option w/ --stdout shoudn't be used since the number of thread will be automatically set/reduced to 3 anyway (because of the output bottleneck).
RE: increasing threads for stdout? - Kgx Pnqvhm - 06-02-2013
I remember this from the original thread about that problem from way back.
So my question would be what settings can we change to make the --stdout usage for generating the list of mangles faster?
RE: increasing threads for stdout? - philsmd - 06-02-2013
I think it is just impossible to significantly increase the speed (and for sure increasing the # of threads doesn't help at all).
I'm not sure if you understood what I meant by saying the stdout is the problem.
Think like this: there is a worker that is *very*,very fast, but the output that he hands over to the input/output (I/O) - to the operating system let's say - will need much time to get printed. The threads need to "wait" until I/O reacts/did it's job... this is what we call a bottleneck and there is kind nothing we can do about it... there is too much to print and too much I/O involved ... also if we had used additional threads we need to wait again for I/O... so more threads do not help.
Faster I/O may help ... so a *really* slow output could depend on what you do with it (e.g. your pipe, your disk - if you use redirect - , it could also depend on operating system, how good it is to handle I/O etc etc).
RE: increasing threads for stdout? - Kgx Pnqvhm - 06-02-2013
Maybe a formal specified mangle output file mode is needed, as opposed to stdout.
RE: increasing threads for stdout? - epixoip - 06-03-2013
the speed of --stdout has not changed. it helps to actually measure, rather than relying on perception.
0.44 without rules
Code:
epixoip@db:~/hashcat-0.44$ ./hashcat-cli64.bin --stdout ~/wordlists/rockyou.txt | pv -l >/dev/null
14.3M 0:02:57 [80.7k/s]0.45 without rules
Code:
epixoip@db:~/hashcat-0.45$ ./hashcat-cli64.bin --stdout ~/wordlists/rockyou.txt | pv -l >/dev/null
14.3M 0:02:57 [80.7k/s]0.44 with rules
Code:
epixoip@db:~/hashcat-0.44$ ./hashcat-cli64.bin --stdout ~/wordlists/top20k.txt -r rules/best64.rule | pv -l >/dev/null
1.66M 0:00:20 [81.1k/s]0.45 with rules
Code:
epixoip@db:~/hashcat-0.45$ ./hashcat-cli64.bin --stdout ~/wordlists/top20k.txt -r rules/best64.rule | pv -l >/dev/null
1.66M 0:00:20 [81.0k/s]since you are writing to a file, it would seem that your hard drive or filesystem is the bottleneck here, not hashcat.
also, it could be that the rules that you are testing are un-optimized and slow.
RE: increasing threads for stdout? - Kgx Pnqvhm - 06-03-2013
What was more fresh in my mind that that the same dictionary and ruleset took a few seconds to run as an attack, but several hours to output the mangles to a file.
As pointed out, stdout isn't really made for high speed I/O. And atom hasn't optimized hashcat for outputting mangles, but for doing actual attacks.
The hashcat utilities mp and sp have a formal specified output file, and are very fast.
What might be needed is, rather than fiddle with hashcat itself, have just the rule engine in a separate mp or sp style program, for the express purpose of generating the mangles.
(The big picture of this question, and my JtR memory commands question in another thread, is rather than running my rules against normalized dictionaries, the running of those cracked word lists against someone else's rules. I think there are a lot of duplicates, so a lot of wasted time, but really can't "see" it, unless I can get the actual mangles to examine.)
RE: increasing threads for stdout? - epixoip - 06-03-2013
(06-03-2013, 02:42 AM)Kgx Pnqvhm Wrote: What was more fresh in my mind that that the same dictionary and ruleset took a few seconds to run as an attack, but several hours to output the mangles to a file.
Well yes, that's absolutely to be expected.
(06-03-2013, 02:42 AM)Kgx Pnqvhm Wrote: As pointed out, stdout isn't really made for high speed I/O.
You misinterpreted what philsmd said. There's nothing special about stdout, it's just a file. It's not that writing to stdout is slow, it's that writing in general is slow. What slows down stdout specifically depends on what is reading from stdout, or where you are redirecting stdout. But this, as philsmd said, is unavoidable, since this is the task at hand.
(06-03-2013, 02:42 AM)Kgx Pnqvhm Wrote: And atom hasn't optimized hashcat for outputting mangles, but for doing actual attacks.
This is not true, atom actually has greatly optimized --stdout. It's as optimized as it's ever going to be.
(06-03-2013, 02:42 AM)Kgx Pnqvhm Wrote: The hashcat utilities mp and sp have a formal specified output file, and are very fast.
This is not why mp and sp are faster. Again, stdout is just a file. All three programs write to files in the same way, regardless of what the file is.
The reason mp and sp are faster is because they generate candidates internally, using a static, compiled algorithm. The rule engine generates candidates using a dynamic, interpreted scripting language. It also has to do simultaneous reads and writes. So this will of course be much slower.
(06-03-2013, 02:42 AM)Kgx Pnqvhm Wrote: What might be needed is, rather than fiddle with hashcat itself, have just the rule engine in a separate mp or sp style program, for the express purpose of generating the mangles.
The speed would be the same regardless.
RE: increasing threads for stdout? - Kgx Pnqvhm - 06-04-2013
Watching my taskbar HD LED while hashcat is doing the mangles shows very infrequent writes, so quite a bit of processing is happening between the writes, so I still don't think the I/O involving my HD is the bottleneck.
If it was showing full time writing, I could believe that the HD wasn't keeping up.
UPDATE: Re-trying the same task that took hours the night I first made this post took just minutes, so something must'be been wrong with my system then.
First, I ran mp64.exe to watch how it used system resources. It used less CPU than hashcat, around 10%, but wrote to disk continuously.
Then, I re-ran the same rule & dict combo that I did the other might, that took hours, with very little disk writing.
This time it was writing briefly about once a second, and was using the same 25% CPU like the other night.
And it "felt" as fast as I expected, meaning a brief wait rather than watching a 2 hour movie and going to bed.
I have almost any scheduled task that Windows want to do disabled, so don't know what could have been preventing hashcat from running properly.
RE: increasing threads for stdout? - Kgx Pnqvhm - 06-04-2013
Upon attempting to boot my computer today, I got beeps indicating a RAM problem.
Pulling out and re-seating all the RAM modules let me boot.
Maybe something was flaky with my RAM the last couple of days.
There had to be something funny going on for a half megabyte file to take eight hours to write that night.