Extensible Build Review

[NoReply]

Hey guys,


going to build a GPU Cluster for a small Forensics company. It should be extensible to 7 GPUs with the current specifcations, however the chassis is able to hold 8. Already did some research, here is what I have so far:


Fixed (i. e. already purchased):

• Chassis: Rexgear Hydra II 6U Rack Mount (https://www.nanoxiausa.com/collections/s...erver-case)
  
• GPUs: 4x RTX 2080Ti FE (Not a good choice, already had to learn that)   
  
• Drive: 1 TB WB Blue M.2 SATA + some spare HDD
  

Unfixed:

• Motherboard: Asus WS X299 SAGE 10G (has 7 PCIe 3.0 x16 slots)
  
• CPU: i7-7800X (6C/12T @ 3.5 GHz)
  
• RAM: 64GB Corsair Vengeance LPX DDR4-2666, upgrade to 128 GB as soon as needed
  
• PSU: (2x) Enermax Platimax 1700W Modular 80+
  
• Riser: Thermaltake AC-045-CN1OTN-C1 PCI Express Extender
  
• Fans: 6x Noctua NF-F12 iPPC-3000 PWM
  

Issues:

• The PCIe extender seems fairly expensive, open to suggestions (no USB based risers, please)
  
• An i7-7800X has 28 lanes, but the mainboard manual says it will run all seven PCIe ports at x8, which would need 56 lanes. Does someone happen to know if this is downgradable to x4? If not I will ask the Asus support.
  
• Not sure about the fan choice, was also looking at some delta fans, e. g. AFB1212GHE-CF00. I am worried that the ones from Noctua wont be enough due to the axial fans of the FE cards. The noise is irrelevant, we have a separate server room.
  

Happy about any kind of feedback

[MrMeeseeks]

RTX 2080 Ti is definitely not a good choice - the cards do not stack and they draw way too much power (enough to damage the motherboard.) And why do you want to use a 6U chassis with PCIe risers for only 4 GPUs? You may want to check out Sagitta HPC since you don't really seem to know what you are doing.

[NoReply]

Thank you for the feedback.

RTX 2080 Ti is definitely not a good choice

Agreed, but it is too late to RMA them. So we are looking for a way to make some use of them.

they draw way too much power (enough to damage the motherboard.)

How does one determine the maximum amount of cards / power a motherboard can handle? Besides, we are moving away from the thought of further extending this build, i. e. we replace the board and the CPU with something that only handles 4 GPUs at max. (E. g. ASRock Fatal1ty X399 Professional Gaming + 1900X)

And why do you want to use a 6U chassis with PCIe risers for only 4 GPUs?

Got plenty of space in our cabinets and found the 6U chassis to be fairly suitable. The chassis requires the use of risers. Really looking for a setting where those 4 cards work properly cooled in our cabinet.

You may want to check out Sagitta HPC since you don't really seem to know what you are doing.

Already did, but it does not help in this case. The goal for now is to build something working around the RTX cards. Mistakes have been made purchasing them before anything else, now looking to find a solution.

Best,
NoReply

[undeath]

and they draw way too much power (enough to damage the motherboard.)

Has Nvidia pulled an AMD on the new cards? It's the first time I hear Nvidia cards exhibiting that behaviour.

[slyexe]

Has Nvidia pulled an AMD on the new cards? It's the first time I hear Nvidia cards exhibiting that behaviour.

A peak of 4.3A on the motherboard slot's 12V rail is well within the PCI-SIG's 5.5A limit. Nvidia's balancing scheme works perfectly, leaving ample room for enthusiasts to overclock.

Seems to me he's making up information.

[soxrok2212]

I haven’t tested myself, but I don’t think we’re nearly at the realm of AMD. My dual R9 390X each sucked up almost 400W. The 2080 Ti, while certainly a HORRIBLE card for the money, can’t even use any of the fancy ray tracing cores so I doubt power consumption would be so high that it can cause damage.

Perhaps someone who dropped an arm and a leg to get one can post numbers.

[NoReply]

Sorry for the late response, here is an update for the planning of the build.

Concerning @soxrok2212 's question, according to hwinfo, one of our single 2080 Ti's peaked at 280W during a real hashcat job, not just the benchmark.

For the build, we moved away from the thought of being extensible, here are the things that changed:

• Motherboard: ASRock X399 Gaming Professional
  
• CPU: AMD Ryzen Threadripper 1900X
  
• RAM: 48 GB Crucial Ballistix Sport DDR4-2666
  

Still open for suggestions concerning PCIe extenders (Ribbons) as I did not find some concrete examples here on the forum.

Best,
NoReply