Done my research, now I need a review. 20 GPU cracking cluster.

[concedonulli]

Hi all!

First off...  yes.  This is a serious post.

I won't bore you with the details, but essentially I've been tasked by my team at work to create a password cracking solution and have a budget of up to $30,000.  I've looked into using PCIE extenders, but those seem to be disliked by the community (price/return thing I guess).  Anyways, I did a bit of research and came up with a solution that I think would fit our needs.  It consists of five hosts with four GTX 980 TIs each.  The five hosts would be clustered to distribute the load.

I'd like to get some opinions from the crowd here since you are all pretty much the experts when it comes to hardware builds for this kind of thing.  Below is the config.  Feel free to rip it apart, and if you have any concerns, PLEASE post some details as to why.  

Each host will be identical, and contain the components below.  They will be located in a datacenter with the front facing a cold aisle:

Component     Quantity    Description
Case          1           Xigmatek Elysium CCC-HSA0DS-U01
PSU           2           Corsair HXi HX120i
Motherboard   1           MSI Gaming X99S Gaming 9 ACK
CPU           1           Intel Core i7-5930K
Memory        1           G.Skill Ripjaws 4 DDR4 2133 64GB (8x8GB)
GPU           4           Gigabyte GeForce GTX 980 Ti GV-N98TG1
SSD           3           Mushkin Enhanced Reactor 1TB SATA III
CPU Cooler    1           Zalman CNPS9900MAX-B

I'd really appreciate any feedback you all can offer.  If you are especially helpful, I'll be sure to put a badge on one of the boxes with your username on it.

[intertan]

Why not wait till Nvidia pascal comes out? Suppose to be 10x faster.
Does one also need an expensive motherboard and cpu?
I'm no expert but couldn't a basic motherboard that supports 4 gpu and a basic i5 work?

[concedonulli]

Why not wait till Nvidia pascal comes out? Suppose to be 10x faster.
Does one also need an expensive motherboard and cpu?
I'm no expert but couldn't a basic motherboard that supports 4 gpu and a basic i5 work?

This is the first I'm hearing about the Pascal.  That thing sounds insane!

As far as the motherboard and processor goes, we needed a processor with enough PCIE lanes to support four GTX 980 TIs.  The motherboard also had to be able to support the four cards running at 8x each since I hear that the GTX 980 TI can't run at anything lower.  The motherboard also needed the PCIE slots to be spaced out enough to fit four double-wide cards.

Thanks for giving everything a look!  Please keep it coming.  I want to make sure we've considered everything before we drop all this money on it.

[undeath]

you can get a preconfigured system at https://sagitta.pw/

[intertan]

Why not wait till Nvidia pascal comes out? Suppose to be 10x faster.
Does one also need an expensive motherboard and cpu?
I'm no expert but couldn't a basic motherboard that supports 4 gpu and a basic i5 work?

This is the first I'm hearing about the Pascal.  That thing sounds insane!

As far as the motherboard and processor goes, we needed a processor with enough PCIE lanes to support four GTX 980 TIs.  The motherboard also had to be able to support the four cards running at 8x each since I hear that the GTX 980 TI can't run at anything lower.  The motherboard also needed the PCIE slots to be spaced out enough to fit four double-wide cards.

Thanks for giving everything a look!  Please keep it coming.  I want to make sure we've considered everything before we drop all this money on it.

You don't need the full x16. Don't quote me but even x1 speeds work

[stepMode]

I'm not entirely sure, but there shouldn't be any difference in x16 and x1 regarding the speed of hashcat, other than maybe large wordlists.

[jodler303]

Another DIY alternative is to use e.g. ASrock BTC series mainboard and connect 6 GPUs via cheap PCIe riser cables. Bitcoin miners often do this. This mobo can drive 6 GPUs for a very low price (~65 $). Riser cables are around 5 - 15$ per GPU depending on the type (the plug is PCIe x1 and the female part is x16 to hold one GPU). Obviously it's not as good and professional as what the Sagitta guys are offering. That particular chassis from the video is sold out, but you get the idea https://www.youtube.com/watch?v=ItVVNGrypz8

[Flomac]

Besides that I don't get why someone needs three 1GB-SSDs in a cracking-rig: if you you want to cluster 5 hosts and share the load, which software do you want to use with hashcat? hashtopus is a nice piece of software, but spending 30K and then rely on an free soft project you have to be some sort of a chancer.

The only reliable working alternative is hashstack, which is binded to Sagitta systems. Maybe you should get two of those instead of building rigs on your own, which does not seem to be your profession (otherwise you would not ask for advice here).

EDIT: You also need to tell us if these machines are supposed to run 24/7 or only on certain occasions. In case of the first latter, most of your hardware is not ready for that. You're building a gaming rig, nothing professional.

And the GPU you've chosen is also for gaming only. For your use, no matter what, it has to look like this:

[intertan]

you can get a preconfigured system at https://sagitta.pw/

I have been curious as to what motherboard they use.
While buying a pre build system might be a good choice for some actually building it yourself is more satisfying.

With the component discount I can get through work I could save a few grand probably.

[Flomac]

.

you can get a preconfigured system at https://sagitta.pw/

I have been curious as to what motherboard they use.
While buying a pre build system might be a good choice for some actually building it yourself is more satisfying.

With the component discount I can get through work I could save a few grand probably.

Building a rig on your own for private purposes is totally fine and makes absolutly sense. In fact you can build a nice rig with 4x 980Ti for four grand easily. But if you go professional you know what you're doing or you leave it to the guys that do. 30K is serious money.

It's like planning to build your own house and your only expirience is the wooden shed in the garden, then maybe you better hire someone.