'Budget' rackmount/server rig questions

[GrepItAll]

Hi,

I'm new to the forums, so go easy on me! I'm a big fan of Hashcat and I am looking at taking things to the next level.


I am currently looking to build a password cracking rig that can be used as a demo/training/R&D machine, which means that my requirements are enterprise/business level: 24/7 uptime; multiple users queuing jobs on the server; headless; rackmount available as an option (but can be freestanding for this first demo unit); must play nicely in a server rack with other machines.

Budget is approx £5000 (~$6500), I will be installing all components myself so labour is not a problem.

Currently basing my shopping off of this: http://www.netmux.com/blog/how-to-build-...acking-rig

Looking at 4 x GTX 1070 FE for budget purposes. As this is a demo unit, subsequent builds could use more powerful hardware (1080 etc.) so ideally case/mobo should support this. I can currently source the 1070s at £400 each.

May also be able to fit additional GPUs down the line to increase performance, so would appreciate suggestions for 6-8 GPU cases too.

If I have 4 x GTX 1070 8GB, so 32GB VRAM total, how much RAM do I need? I saw a post saying double your VRAM is a good rule of thumb.

Also, how frugal can I be with the CPU? My use of manipulating wordlists and using rules is fairly limited, but is something I need to start looking at due to the slow speeds of some of the hashes I am cracking. Do I need 2 x Intel Xeon E5-2620 2.4 GHz or can I make do with less? 2 x 1.7GHz Xeon instead?

Apologies if this is all a bit vague, it's early days for this project for me but thought I'd get advice from the experts first! I'm happy to provide some more info about my requirements if necessary!

Thanks

EDIT: Just to note: I have been looking at Sagitta and other system manufacturers, but wanted to get a well-rounded view of the market!

[Flomac]

You want to build a cracking rig, right? So cracking performance should be No. 1 on your list. And that configuration by netmux is rubbish and totally wrong for that budget. There is a big trade-off from cracking performance to reliability (and wrong components). Most of the money is spent on an expensive server rig that deoes not serve you any value in performance.

1. Always buy the GPU with the highest performance/value and calculate your rig in. Therefor in almost every case you have to go with the GTX 1080Ti. Two of them makes ~4x GTX1070 performancewise. Take a decent case, a nice PSU, a usual CPU/MB combo, some SSD and some RAM and you land at around $2k. Same performance as that 5k-rig for less than half the price.

2. If you want to improve reliability, get a premium PSU and a very good mainboard with high quality transistors and stuff. Put lots of fans in your case, front, bottom, back. Front/bottom blowing in, back out. Keep the area around the system clean and free of dust. Keep the room as cool as possible. Maybe even buy a UPS (but those bastards sometimes turn out to be a SPoF themselves).

3. Aleways buy FE-cards with radial fans. Nothing else. Brand technically doesn't matter, but there are some that offer 5yrs warranty for free if you register the cards. That's always a nice goody to offer if you sell them one or two years later.

4. Oversize your PSU. For three GTX1080 Ti get >1500W. Get 2000W for 4 GPUs. Buy at least platinum level. Electricity is a crucial cost factor when it comes to 24/7-cracking. A percentage difference in efficiency usually pays of in the long run.

5. The best compromise and less trouble with heat is to put three GPUs on one MB, with one slot space between the cards. If you put in four, get better cooling and make sure you know what're doing.

6. CPU performance is negligible if you don't plan to use it for cracking. 32GB RAM should be fine. SSD doesn't matter, neither speed nor size. 128GB is usually more than enough, 256GB completely fine, even if you start collecting wordlists.


Here's what I would do:
- Mainboard $300
- CPU $300
- RAM $200
- Case $100
- PSU $300
- SSD $100
- Cooler $100
- 3x GTX 1080Ti $2100

That's roughly $3500. You could even through in another Ti and stay well below your 6.5k budget. If something breaks down, replace it. Nothing is bulletproof, not even that 5k-rig. More expensive does not mean more reliable. If you don't earn money with it, than down-time costs you $0.

[GrepItAll]

Thank you for the advice Flomac.

Your advice on going with 1080 Ti instead of 1070 sounds reasonable. We could always start with 2 and add more if necessary if we had a case/psu/cpu to support it!

Your points on the CPU/SSD/RAM are also good.

I think the reason I was looking at the Supermicro server in the netmux blog post is that this same case appears to be used a number of suppliers that can sell pre-configured servers.

Whilst I'm happy to explore cheaper cases and provide the mobo and PSU independently of that to save money, I have not seen many systems like that with positive feedback. Do you have any suggestions for a manufacturer of these items? Particularly the case.

You budgeted $800 total for case/mobo/psu/fans. The Supermicro case can be purchased for about $1950 but I've seen positive reviews of that and I know that it will 'work'. The supermicro case also has a total of 8 fans, plus active heatsinks supplied.

On the 3 GPU on one mainboard, we would not be able to leave 1 space between each card if each PCI-e slot is 2 'slots' apart

I'm happy to get my hands dirty, but if all goes well then the system we build will be recommended to our customers who may not be so keen to 'cheap out' on parts and would rather spend more for peace of mind.

[Flomac]

Ah, ok, so you want to sell that system to other costumers? That's of course a different story. Thought that's your personal play-around-rig.

Then don't make the same mistake as Nemux:
"I know it's shocking! Who would have thought GPU's run hot?!? But it's very interesting to see the clear separation of cooling."

...and take off the backplates

Personally I don't like that little fan on the chipset since it can end up as a nasty SPoF. The newer version (7049) looks better to me.

[GrepItAll]

Not sell per se, but make a recommendation. My employer is a training provider and one of our courses will involve the use of the rig that we will be building. We always provide students with a full kit list with recommendations of suppliers and parts, but we're vendor-neutral so will provide various options, including (but not limited to) suppliers like Sagitta as well as building it themselves (many of the customers have internal teams for managing/designing their HPC/GPGPU setups)

Therefore whilst of course cracking performance is the number one priority, getting premium components is easily justifiable for our customers.

Ah the backplate removal is a good suggestion! Just the back half right?

7049... this? https://www.supermicro.nl/products/syste...GP-TRT.cfm

Using the 7049 would require Xeon Phi processors as it's LGA 3647 on the board, which are veeeeery pricey!

[Flomac]

Using the 7049 would require Xeon Phi processors as it's LGA 3647 on the board, which are veeeeery pricey!

Jep, doesn't make much sense then