Done my research, now I need a review. 20 GPU cracking cluster.

[Betawave]

Hi all!

First off...  yes.  This is a serious post.

I won't bore you with the details, but essentially I've been tasked by my team at work to create a password cracking solution and have a budget of up to $30,000.  I've looked into using PCIE extenders, but those seem to be disliked by the community (price/return thing I guess).  Anyways, I did a bit of research and came up with a solution that I think would fit our needs.  It consists of five hosts with four GTX 980 TIs each.  The five hosts would be clustered to distribute the load.

I'd like to get some opinions from the crowd here since you are all pretty much the experts when it comes to hardware builds for this kind of thing.  Below is the config.  Feel free to rip it apart, and if you have any concerns, PLEASE post some details as to why.  

Each host will be identical, and contain the components below.  They will be located in a datacenter with the front facing a cold aisle:

Component     Quantity    Description
Case          1           Xigmatek Elysium CCC-HSA0DS-U01
PSU           2           Corsair HXi HX120i
Motherboard   1           MSI Gaming X99S Gaming 9 ACK
CPU           1           Intel Core i7-5930K
Memory        1           G.Skill Ripjaws 4 DDR4 2133 64GB (8x8GB)
GPU           4           Gigabyte GeForce GTX 980 Ti GV-N98TG1
SSD           3           Mushkin Enhanced Reactor 1TB SATA III
CPU Cooler    1           Zalman CNPS9900MAX-B

I'd really appreciate any feedback you all can offer.  If you are especially helpful, I'll be sure to put a badge on one of the boxes with your username on it.

Paging epixoip.

Just going to chime in with what everyone else has already said.. buy a Sagitta system if you have the budget...the markup is reasonable.  You can build a system for less, but you'll waste a ton of time researching and then getting it right.

I didn't have the luxury of a big budget and just recently built my own rig that is very similar in spec to what you're proposing (though I have a lot more disk storage).  Now that I've lived with it a while, I'm already convinced it's vastly underpowered for the type of research I'm doing, but your mileage may vary depending on what you do with the box(es).  I'm running 4x GTX970 (best bang for the buck) and that's just not enough.  I'm now debating how I can get more horsepower without breaking the bank.

[jodler303]

I've built a DIY server for max 7 GPUs within the last weeks, it's based on a cheap 15-slot 19" chassis and i'm using "cheap" alternative hardware as opposed to the professional rack chassis that Brutalis is using. Mine is currently loaded with 4xGTX 980 and 1x TITAN X, so 2 more to go. All i can tell you it's hell of a job and requires a LOT more time, fiddling and research than i would have thought to get that box working, the power supply and cooling issues are bigger than you would ever think of (even for the lower power nvidia cards). Soon i've realized that Sagitta box is not as expensive as it might seem at first sight, the prices in the high-end segment are raising exponentially, alone the chassis costs something around $5000. - Also, you get the hashstack software with that box. If you have the budget, go for Sagitta. You won't be disappointed, from the best knowledge i have gained within the last months, the setup they are offering is what you are looking for. If not, plan to use half of your budget for research, at least. Unfortunately I didn't have the budget for a Brutalis, so i had no choice.

[intertan]

Does anybody know what hardware the Brutalis uses?
I enjoy building things so I may take a swing at build a 8th monster.

[Flomac]

Does anybody know what hardware the Brutalis uses?

Sure.

[epixoip]

Certainly doesn't seem like you've done your research, none of the components in your proposed build make any sense.

[concedonulli]

Thanks for all the replies!  Good stuff!

I reached out to Sagitta for a quote on a box with 980 TIs.  There's definitely something to be said about getting a pre-built box even if it's a tab more expensive (less overhead and hassle).

Certainly doesn't seem like you've done your research, none of the components in your proposed build make any sense.

That's not what I want to hear from someone with a "hardware guru" tag under their name!!  Making me nervous man...

So, do you mind if we step through it all?  I know I didn't give any details about why we selected the various components, but I think it might be time to do that.

So first of all, the GPU was selected based on benchmarks found here in these forums.  Price/performance was definitely considered, but we're not pinching pennies too hard here.  The goal was to make something that would last (lol, loaded statement for anything IT related, I know).  The processor was chosen because of the amount of PCIE lanes it has.  From what I am reading, the GTX 980 TI will not run in a PCIE slot running under 8x so a compatible proc and mobo is required.  Please correct me if I'm wrong here.  Yes, I get it.  I know the PCIE speed doesn't matter in this application, but it does if the card won't even post.  Again, please let me know if I'm way off here.

Power supply was selected for the +12v rails and efficiency rating.  Each system will have two since it's never really a good idea to run a PS at full load all the time.  We'll figure out how we're going to split up the power consumption once we get the build going.

Hard drives were selected because we wanted to load up a few rainbow tables while we were at it to knock out the low-hanging-fruit.  I don't think each system needs them though, so maybe we just just load up one with a few larger hard drives.

if you you want to cluster 5 hosts and share the load, which software do you want to use with hashcat? hashtopus is a nice piece of software, but spending 30K and then rely on an free soft project you have to be some sort of a chancer.

The only reliable working alternative is hashstack, which is binded to Sagitta systems.

That's actually a good point that I didn't consider.  In your experience, has hashtopus been well-maintained?  This might actually be the thing that sways us into going with the sagitta box instead.  Again, like I said (and like you implied), there's a lot to be said about getting a pre-built and SUPPORTED product instead of putting something together on our own when considering the price tag.

[epixoip]

I reached out to Sagitta for a quote on a box with 980 TIs.

Can you PM me with the email address you contacted us with so I can ensure you receive your Hashcat Forums discount?

So, do you mind if we step through it all?

Sure.

The specific GPU you picked is extremely inappropriate for hash cracking. You picked an OEM design GPU with 3x axial coolers on a vertical-fin heatsink, which was clearly designed for gaming workloads and will be unable to cope with compute workloads. You will have massive cooling and reliability issues if you attempt to pack four of these into a single system. This has been covered extensively on these forums, so this alone tells me you didn't really do much research, as anyone who has spent even an hour on these forums knows you only buy reference design GPUs for hashcat.

You mentioned this was going into a datacenter, so desktop ATX case makes no sense. For a 4-GPU system you'd be better off with something like Chenbro RM41300-FS81 or Supermicro 7048GR-TR (both of these chassis are rackmountable and specifically designed for GPGPU.)

The PSU you picked isn't really that great. It looks good on paper, but it's not the best option. Also 1200W isn't giving you very much headroom. You'd be pushing it with 4x 980 Ti, when cracking e.g. NTLM you'd be at ~87% load. For maximum efficiency you want to be around 50% load, so I'd go with something like an EVGA 1600 T2 instead.

64GB of RAM doesn't make much sense. 980 Ti only has 6GB of VRAM so your host memory requirements would be an absolute maximum of 24GB . So 32GB of RAM would be more far more appropriate, but as you can't really use the full 6GB of VRAM for hashcat, 16GB of RAM would likely be more than sufficient. 

2TB of storage also doesn't make much sense (assuming you're doing RAID-5.) You'll surely never use more than 100GB on the compute nodes. 1TB would be more appropriate for the cluster controller, which you've not specified (maybe you didn't realize you'd need one?) I'm also not crazy about Mushkin SSDs, I tend to lean toward Intel or Crucial.

Lastly the CPU cooler you picked would be terrible for chassis airflow. If you were to actually use your CPU for cracking (which I presume you will since you picked such a high-end CPU), this cooler likely wouldn't be able to keep up, and not only would the CPU temps be very high, but the temperature inside the chassis would rise quickly as well. A Dynatron R-series cooler with some Delta chassis fans would be far more appropriate.

[concedonulli]

PM sent. I really appreciate all the feedback.

That level of detail definitely tells me that you know what you're talking about. Well, so does a Google search for your user name! Let's chat.

[Flomac]

I reached out to Sagitta for a quote on a box with 980 TIs.  There's definitely something to be said about getting a pre-built box even if it's a tab more expensive (less overhead and hassle).

Surely a very good idea.

The processor was chosen because of the amount of PCIE lanes it has. From what I am reading, the GTX 980 TI will not run in a PCIE slot running under 8x so a compatible proc and mobo is required. Please correct me if I'm wrong here.

Never heard of it, so I guess you misunderstood something. Post a source if you have one.

Power supply was selected for the +12v rails and efficiency rating.  Each system will have two since it's never really a good idea to run a PS at full load all the time.  We'll figure out how we're going to split up the power consumption once we get the build going.

Two power supplies are a hassle and no good idea. Double chance one of them fails.

Hard drives were selected because we wanted to load up a few rainbow tables while we were at it to knock out the low-hanging-fruit.  I don't think each system needs them though, so maybe we just just load up one with a few larger hard drives.

No one uses rainbow tables anymore. Way too slow.

That's actually a good point that I didn't consider.  In your experience, has hashtopus been well-maintained?

From what I've read in the forum its a non-commercial software project so it might work well or not at all. Who knows.


Again, before your company spends 30K the questions to solve would be:
- for what purpose do you buy that hardware?
- which kind of passwords are you trying to crack?
- how many h per week shall it run?
- security monitoring, regular audits or determing PKIs?

Answering these questions is essential for the successful outcome of your project. Only then you can determine the amount of hardware needed and adress a necessary price tag. It might be much lower or way higher. Let epixoip advise you well.

[mamexp]

OP,

Assume you go with the build, do you have a dedicated room for the cluster? If you dont, have you considered watercooling? If you do, have you check the air ventilation of that room to be able to cool the cluster?

Please share your findings.

Thanks