First GPU Rig - A few questions

[zozeri]

Hi all I am new to the forum but have referenced posts here for all my curiosities up until I decided to get serious and build a small rig.

I would first of like to thank epixoip, every question I have had led to a solid answer by you so thank you for your help I am sure many other unregistered users are grateful for your help.

So back to my rig... specs are (currently) as follows

Motherboard: Asrock H81 Pro BTC Intel H81
PSU: Corsair AX1200 Professional Series AX 1200W
CPU: Intel Celeron G1840 Processor 2.8 GHz
RAM: Kingston 4 GB 1600 MHz DDR3
GPU: None purchased - either 7970 or 290x
SSD: Will order soon - Sandisk SSD Plus 120GB

OS: Not sure but probably a linux build

Accessories;
Add2PSU 24 Pin ATX Cable Adapter Splitter
6x PCI-E 1X to 16X Powered Extension USB 3.0 Risers
Homebuilt wooden frame - built it today

Everything listed above has been ordered this week and should be arriving soon, I am wondering if apart from the GPU's am I missing any important or useful items?

GPU
Now I have looked at the GPU Hashcat benchmark and for WPA/2 and my budget (around £100 each) I believe the Radeon HD 7970 will be the best rated for my money, unless someone can advise me otherwise? I have also tried to get lucky and find a R9 290X within my budget but it seems far from likely - I have watched them go over the last few days and the cheapest was £170. (not including the OEM designs)

PSU
I have managed to get hold of a Corsair 1200w, my plan is to initially use this to power the mobo/cpu/ssd + 2 GPUs
Eventually I will be running a 6 (if possible to multiply the PCI-e slots then 8) GPUs. 
Scenario 1: Powering 6x 7970
Scenario 2: Powering 6x R9 290X
Scenario 3: Powering 8x 7970
Scenario 4: Powering 8x R9 290X
What would I need for my second PSU? would a 850w PSU be sufficient in the case of scenario 1?
850w powering 2xGPU + mobo/cpu/ssd - 1200w powering 4xGPU
Also want to know, how GPU's can I safely power using 1200w on this rig before requiring a second PSU?


OS
My aim is to have the rig working in 'headless' mode, ideally I could be at a different location using my laptop and via command line run oclHashcat on my home rig. RDP is an option but is not really what I am looking for.
To be honest I have not done research on this (don't flame), my rig was a spontaneous decision and I have only been reading up on this for around a week.


Summary for those who don't want to read everything;
1) What GPU do you recommend for WPA/2 and a £100 budget (will buy used on ebay)
2) What combination of PSU should I use assuming that I am gradually adding GPUs to it.
3) How many GPUs can I power with the whole rig using a 1200w PSU.
4) OS - I want to use the rig as a remote server that I can use via command line.

Any other recommendations or optional extras - anything at all you might deem useful I would be glad to hear and take it on board as a potential addition to my rig.

Thanks in advanced!!




p.s. for those who are interested my rig frame was based on this how-to video
https://www.youtube.com/watch?v=Cscm8blV40c

See photo below 

[epixoip]

Your chassis looks nice, well done. 

CPU: Intel Celeron G1840 Processor 2.8 GHz
RAM: Kingston 4 GB 1600 MHz DDR3

... So, are you building a password cracking rig, or a bitcoin miner?  If you plan to run six GPUs with 4GB VRAM each, then you should have at least 24GB of host RAM. There's certainly no reason to skimp on the CPU, either. You don't have to go with a i7-4790K, but you should at least go with an i5-4590 or something.

GPU: None purchased - either 7970 or 290x

Why those GPUs? There's usually never a good reason to use AMD GPUs in a new build. I guess it's because you're wanting to spend no more than 100 GBP per GPU? Keep in mind those GPUs are power hungry pigs, you should budget for them pulling 325-350W each. This means on your 1200W PSU you should only power three GPUs max. Any more than that, and your nice wooden chassis will turn into a pile of ashes.

4) OS - I want to use the rig as a remote server that I can use via command line.

Linux is the obvious choice.

[zozeri]

Thanks for the quick response!

... So, are you building a password cracking rig, or a bitcoin miner?  If you plan to run six GPUs with 4GB VRAM each, then you should have at least 24GB of host RAM. There's certainly no reason to skimp on the CPU, either. You don't have to go with a i7-4790K, but you should at least go with an i5-4590 or something.

To be honest I was not too sure, I was watching all the old bitcoin miner videos to get an idea of set-ups.

For the CPU ill look at getting an i5/i7 as you recommend.

Ok so ram - there seems to only be 2 ram slots on this motherboard and going by the manufactures guide at http://www.asrock.com/mb/Intel/H81%20Pro...cat=Memory
it can only support a maximum of 8GB in a slot - so max 16GB ram

maybe I need to look into an alternative motherboard? I would rather not if possible.

Why those GPUs? There's usually never a good reason to use AMD GPUs in a new build. I guess it's because you're wanting to spend no more than 100 GBP per GPU? Keep in mind those GPUs are power hungry pigs, you should budget for them pulling 325-350W each. This means on your 1200W PSU you should only power three GPUs max. Any more than that, and your nice wooden chassis will turn into a pile of ashes.

I got the impression from the forum that those GPUs are the most popular otherwise would of looked at others - bare in mind I have only researched this for a week or so on my off time.
What GPU/s do you recommend I look at - if there is a substantial savings in power usage and possibly a better hash rate I do not mind getting something slightly out of my budget.

Linux is the obvious choice.

Thought it would be, ill mess around with this once all is setup - only afraid of having trouble setting up drivers but will cross that bridge when I get there.

I will not be using the rig constantly and will be mainly a project to play with when I have some free time.


****EDIT****
Actually looking back at the threads I have bookmarked they seem to be outdated by a couple years so it makes sense that they are no longer the optimal choice GPUs

Also I have found the guide I was going by at https://blog.netspi.com/gpu-cracking-rebuilding-box/

They stated the following which lead me to believe CPU/RAM/HDD do not really affect the rig;

CPU/RAM/Hard Drive
All of these can be generic. It doesn’t hurt to max these out, but they don't really impact cracking performance. If you’re going to throw money around, throw it at your cards.

Their rig specs;
Component Model Est. Price
Case 6 GPU Rackmount Server Case $495
Motherboard: ASRock Motherboard H81 PRO BTC $64
Risers (6): PCI-E 1X To 16X USB 3.0 Riser Card $24
GPUs (6): XFX Black Edition R9 290 $1,884
Power Supply (2): CORSAIR AX1200i 1200W $618
Power-Splitter: Dual PSU Adapter Cable $9
RAM 8 GB: - Any Brand $50
CPU: Intel Celeron G1820 Dual-Core (2.7GHz) $45
HDD: 1 TB - Any Brand $50
Total $3,239

[epixoip]

Yeah, it looks like most of what you've been researching is bitcoin/litecoin mining stuff from 2011-2013. Which can be helpful, but mining is only analogous to one facet of password cracking: single-hash brute force. This is why miners can get away with ultra-low CPU and RAM requirements. Also there's an economic incentive for miners to do everything on a shoestring budget, as money spent is profit lost. But there's much more than just single hash brute force when cracking passwords, so we need things like CPU and RAM.

Currently the best GPUs for password cracking are from the Nvidia Maxwell family, i.e. GTX 970, 980, 980 Ti, and Titan X. You can go with an older used AMD card if you'd like, but they won't be as fast and they'll draw a lot more power than Nvidia Maxwell cards.

I wouldn't do more than 4 GPUs with only 16 GB of host memory. If that board only supports 16 GB of RAM, then I'd look into getting a different board.

[zozeri]

Yeah, it looks like most of what you've been researching is bitcoin/litecoin mining stuff from 2011-2013. Which can be helpful, but mining is only analogous to one facet of password cracking: single-hash brute force. This is why miners can get away with ultra-low CPU and RAM requirements. Also there's an economic incentive for miners to do everything on a shoestring budget, as money spent is profit lost. But there's much more than just single hash brute force when cracking passwords, so we need things like CPU and RAM.

Currently the best GPUs for password cracking are from the Nvidia Maxwell family, i.e. GTX 970, 980, 980 Ti, and Titan X. You can go with an older used AMD card if you'd like, but they won't be as fast and they'll draw a lot more power than Nvidia Maxwell cards.

I wouldn't do more than 4 GPUs with only 16 GB of host memory. If that board only supports 16 GB of RAM, then I'd look into getting a different board.

Awesome thanks for the detailed info, I think that was the case I was looking at older forum posts,
the only thing that mislead me was this website: https://blog.netspi.com/gpu-cracking-rebuilding-box/
their build was running 6x R9 290 with an Intel Celeron 2.7Ghz and only 8GB ram.


I think what I will do at the moment is carry on with this motherboard as I picked it up for only £20 I might as well use it as my starter rig and later upgrade when I get my third GPU - will look at also getting an i5 CPU later in the next month or when I come across a decent offer.

I am still a bit confused how their setup managed to run 6x GPU's with 8GB ram and that CPU.

And one last thing, The Nvidia cards you recommended is the reference design still recommended over the OEM?

***EDIT***
I have had a look at power consumptions
Radeon HD 7970 - 195w - 115000 hash/s
GTX 970 - 145w - 150000 hash/s

Although the 970 offers better 'everything' the price is more than double even at second hand they go from £200-£300 and have not come across any reference designs sold at the lower end.
Currently as this is only a project and a test rig nothing too serious I will probably go with the 7970 just to mess about with - once more money comes in ill look at the higher end NVIDIA cards or just build a second more efficient rig (maybe a new frame made out of aluminium)

Current CPU was £22 & RAM was £10.80 so really cheap crap but initially will see if it can get up and running and ill keep my eye open incase I come across any offers, probably upgrade them within the next month or so.

Anyway if anyone is interested I can post progress of the rig.

[Flomac]

Awesome thanks for the detailed info, I think that was the case I was looking at older forum posts,
the only thing that mislead me was this website: https://blog.netspi.com/gpu-cracking-rebuilding-box/
their build was running 6x R9 290 with an Intel Celeron 2.7Ghz and only 8GB ram.

Karl from NetSPI obviously doesn't know what he's talking about. Ask him for a real world hashcat job instead of posting simple benchmarks. There are many ways to do hash cracking and a 2.7GHz Dual-Core feeding six GPUs is limiting you from most of them.

On top he's dealing with a lot of heat from the six 290x as he writes in his blog, well why not using reference cards as it is claimed here since years? And put at least some fans in the case. Then your benchmark might stop showing lousy 10.4 GH/s in MD5 - almost 10% under the expected value.

So again, that blog is at no point a good advice for building a rig.

[globus]

Awesome thanks for the detailed info, I think that was the case I was looking at older forum posts,
the only thing that mislead me was this website: https://blog.netspi.com/gpu-cracking-rebuilding-box/
their build was running 6x R9 290 with an Intel Celeron 2.7Ghz and only 8GB ram.

Karl from NetSPI obviously doesn't know what he's talking about. Ask him for a real world hashcat job instead of posting simple benchmarks. There are many ways to do hash cracking and a 2.7GHz Dual-Core feeding six GPUs is limiting you from most of them.

On top he's dealing with a lot of heat from the six 290x as he writes in his blog, well why not using reference cards as it is claimed here since years? And put at least some fans in the case. Then your benchmark might stop showing lousy 10.4 GH/s in MD5 - almost 10% under the expected value.

So again, that blog is at no point a good advice for building a rig.

Karl from NetSPI checking in here...

As far as I can tell, the RAM and CPU have not really been a bottleneck for us. We started with lower specs on those to keep the budget down and our boxes have been working well enough for us to be happy.

As for the GPU choice, we bought our 290s right before the Nvidia cards got better. We have a Titan X at the moment, and I'd love to have a few more of those to use.

The 290s do run a little hot, but we have three case fans up front (see the picture from the blog) that are pushing solid amounts of air. So far it's been keeping things pretty cool.

Our boxes have been built piecemeal and have come together over time, using stuff that we might have already had on hand. That being said, we haven't always made the best long term choices for our boxes. Aside from that, the blog post is there to show what works for us. We may not be maxing out our cards, but our boxes have been working solidly and quite reliably for multiple years now. I'd love to have more time to research how to optimize our set up, but I'm pretty busy using the boxes for cracking client hashes.

Real World Hashcat Job for Scale:
Here's our "Basic" cracking dictionary (1.7 GB) running with d3ad0neV3.rule against a recent domain dump of 6,273 hashes.
I wouldn't be surprised that the CPU/Memory is choking on the number of hashes, but the job finishes in less than a minute and cracked 57% of the hashes. We're usually able to hit 85% cracked (for NTLMs) in the first hour of effort.
The hash rates are not insane, but it's more than adequate for our current needs.

Session.Name...: oclHashcat
Status.........: Running
Rules.Type.....: File (/rules/d3ad0neV3.rule)
Input.Mode.....: File (/wordlists/NetSPIMaster.txt)
Hash.Target....: File (/ntlm-hashes.txt)
Hash.Type......: NTLM
Time.Started...: Thu Feb 11 12:31:24 2016 (16 secs)
Time.Estimated.: Thu Feb 11 12:32:16 2016 (33 secs)
Speed.GPU.#1...:  2140.6 MH/s
Speed.GPU.#2...:  2306.7 MH/s
Speed.GPU.#3...:  1605.4 MH/s
Speed.GPU.#4...:  1530.9 MH/s
Speed.GPU.#5...:  1199.5 MH/s
Speed.GPU.#6...:  1965.4 MH/s
Speed.GPU.#*...: 10748.5 MH/s
Recovered......: 3308/6273 (52.73%) Digests, 0/1 (0.00%) Salts
Recovered/Time.: CUR:N/A,N/A,N/A AVG:11786.62,707196.94,16972726.00 (Min,Hour,Day)
Progress.......: 169143173120/502126675560 (33.69%)
Rejected.......: 0/169143173120 (0.00%)
Restore.Point..: 45219840/149043240 (30.34%)
HWMon.GPU.#1...: 100% Util, 50c Temp, 100% Fan
HWMon.GPU.#2...: 100% Util, 46c Temp, 100% Fan
HWMon.GPU.#3...: 100% Util, 43c Temp, 100% Fan
HWMon.GPU.#4...: 100% Util, 48c Temp, 100% Fan
HWMon.GPU.#5...: 100% Util, 45c Temp, 100% Fan
HWMon.GPU.#6...: 100% Util, 48c Temp, 100% Fan

[s]tatus [p]ause [r]esume [b]ypass [q]uit =>

[epixoip]

You're cracking a small list of unsalted hashes. Try cracking a larger set of salted hashes, and you'll see why CPU & RAM matter.

[globus]

You're cracking a small list of unsalted hashes. Try cracking a larger set of salted hashes, and you'll see why CPU & RAM matter.

That's fair. The bulk of what we're seeing is MD5, NTLM, and NetNTLMv2. I'd love to see the NetNTLMv2 hashes go up in speed, but most of these are coming from domain users, so it's not hard to crack Winter2016.

[mamexp]

For anyone with non ref cards, the best place to put the fans is on the top of the cards, not side front and back like traditional cases. If you already bought non ref cards, you got no choice but making custom case for them.

This is why ref design is alot better for air controlling, front to back for rackmount.

Also another tip for anyone custom their case for air flow. Reduce the air volume in the case, so you can increase air velocity and control air going through hotspot.

You're cracking a small list of unsalted hashes. Try cracking a larger set of salted hashes, and you'll see why CPU & RAM matter.

That's fair. The bulk of what we're seeing is MD5, NTLM, and NetNTLMv2. I'd love to see the NetNTLMv2 hashes go up in speed, but most of these are coming from domain users, so it's not hard to crack Winter2016.

Hi Karl i can't help but i notice you have a ref card facing the wrong direction of the case airflow.