First GPU Rig - A few questions

[globus]

Awesome thanks for the detailed info, I think that was the case I was looking at older forum posts,
the only thing that mislead me was this website: https://blog.netspi.com/gpu-cracking-rebuilding-box/
their build was running 6x R9 290 with an Intel Celeron 2.7Ghz and only 8GB ram.

Karl from NetSPI obviously doesn't know what he's talking about. Ask him for a real world hashcat job instead of posting simple benchmarks. There are many ways to do hash cracking and a 2.7GHz Dual-Core feeding six GPUs is limiting you from most of them.

On top he's dealing with a lot of heat from the six 290x as he writes in his blog, well why not using reference cards as it is claimed here since years? And put at least some fans in the case. Then your benchmark might stop showing lousy 10.4 GH/s in MD5 - almost 10% under the expected value.

So again, that blog is at no point a good advice for building a rig.

Karl from NetSPI checking in here...

As far as I can tell, the RAM and CPU have not really been a bottleneck for us. We started with lower specs on those to keep the budget down and our boxes have been working well enough for us to be happy.

As for the GPU choice, we bought our 290s right before the Nvidia cards got better. We have a Titan X at the moment, and I'd love to have a few more of those to use.

The 290s do run a little hot, but we have three case fans up front (see the picture from the blog) that are pushing solid amounts of air. So far it's been keeping things pretty cool.

Our boxes have been built piecemeal and have come together over time, using stuff that we might have already had on hand. That being said, we haven't always made the best long term choices for our boxes. Aside from that, the blog post is there to show what works for us. We may not be maxing out our cards, but our boxes have been working solidly and quite reliably for multiple years now. I'd love to have more time to research how to optimize our set up, but I'm pretty busy using the boxes for cracking client hashes.

Real World Hashcat Job for Scale:
Here's our "Basic" cracking dictionary (1.7 GB) running with d3ad0neV3.rule against a recent domain dump of 6,273 hashes.
I wouldn't be surprised that the CPU/Memory is choking on the number of hashes, but the job finishes in less than a minute and cracked 57% of the hashes. We're usually able to hit 85% cracked (for NTLMs) in the first hour of effort.
The hash rates are not insane, but it's more than adequate for our current needs.

Session.Name...: oclHashcat
Status.........: Running
Rules.Type.....: File (/rules/d3ad0neV3.rule)
Input.Mode.....: File (/wordlists/NetSPIMaster.txt)
Hash.Target....: File (/ntlm-hashes.txt)
Hash.Type......: NTLM
Time.Started...: Thu Feb 11 12:31:24 2016 (16 secs)
Time.Estimated.: Thu Feb 11 12:32:16 2016 (33 secs)
Speed.GPU.#1...:  2140.6 MH/s
Speed.GPU.#2...:  2306.7 MH/s
Speed.GPU.#3...:  1605.4 MH/s
Speed.GPU.#4...:  1530.9 MH/s
Speed.GPU.#5...:  1199.5 MH/s
Speed.GPU.#6...:  1965.4 MH/s
Speed.GPU.#*...: 10748.5 MH/s
Recovered......: 3308/6273 (52.73%) Digests, 0/1 (0.00%) Salts
Recovered/Time.: CUR:N/A,N/A,N/A AVG:11786.62,707196.94,16972726.00 (Min,Hour,Day)
Progress.......: 169143173120/502126675560 (33.69%)
Rejected.......: 0/169143173120 (0.00%)
Restore.Point..: 45219840/149043240 (30.34%)
HWMon.GPU.#1...: 100% Util, 50c Temp, 100% Fan
HWMon.GPU.#2...: 100% Util, 46c Temp, 100% Fan
HWMon.GPU.#3...: 100% Util, 43c Temp, 100% Fan
HWMon.GPU.#4...: 100% Util, 48c Temp, 100% Fan
HWMon.GPU.#5...: 100% Util, 45c Temp, 100% Fan
HWMon.GPU.#6...: 100% Util, 48c Temp, 100% Fan

[s]tatus [p]ause [r]esume [b]ypass [q]uit =>