Am I doing this right?

[mercystarf]

Hi all,

Ordered a Syrenis from Terahash in April but since that is not moving, I need to look for alternatives. I had this quote from a local supplier. Any remarks/feedback?

1 Supermicro SYS-4028GR-TRT2
2 Intel Scalable CLX-SP 4210R 10C/20T 2.4G 13.75M 9.6GT 2UPI 100W 3647 P0
2 Samsung 32GB DDR4-2933 ECC REG DIMM 2Rx4LP M393A4K40***-CVF
2 Intel S4510 480GB. SATA 6Gb/s. 2.5'' SSD. 3D. TLC. up to 2DWPD SSDSC2KB480G801
1 Supermicro MCP-230-41803-0N Top cover for SC418 with GTX card support
1 Supermicro 3U Railkit L Edition For 842TQ MCP-290-00057-0N
4 RTX 3090 
4 Supermicro 8-pin to two 6+2 Pin 5cm Power GPU Cable CBL-PWEX-1040

Furthermore, what kind of OS should I be running? I assume some kind of Linux flavor is best but which distro/kernel has best driver performance/stability? 

Stupid question maybe, but what bout SLI? I guess SLI doesn't matter when using hashcat?

[Sondero]

Hey,
i would change:
the 32GB RAM to 64GB
the 480SSD to 2x NVME4.0 (Raid0)
the 4x 3090 to 4,5 or 6x 3080 and add a watercooler.

For me, the best solution ist Win-10 for hashcat, and wirh Linux-Subsystem you can use most of the important linux commands like dd, cat, set, egrep without stating a VM.

[Longtail]

I do not think that you are doing right...

The Barebone GPGPU Server you chose will not work with the CPUs that you chose. Supermicro SYS-4028GR-TRT2 has LGA 2011 sockets which are compatible with Xeon E5-2600v4 series of processors.

Either change your CPUs to Xeon E5-2600v4 Series OR change your barebone to Supermicro SYS-4029GP-TRT2 which has LGA 3647 sockets and which will work with Xeon Scalable Series.

By the way, I am very surprised that your vendor didn't warn you about this incompatibility.

I would go for Supermicro SYS-4029GP-TRT2 because it is a newer model, and stick to the Xeon Scalable Series processors. (I mentioned Supermicro models only, thinking that it is the brand of your choice. Otherwise, I would suggest that you check models from TYAN or other manufacturers, as well.)

Secondly, your RAM choice is incorrect. Xeon E5-2600v4 series support 4 channels of RAM, and Xeon Scalable Series support 6 channels of RAM. In order to get maximum RAM speed, you should be using 4 or 6 pieces of RAM modules per CPU, respectively. You are only getting two pieces, 1 per CPU! So if you go for Xeon E5-2600v4 you need at least 8 pieces of RAM and if you go for the Xeon Scalable Series you need at least 12 pieces of RAM. 64 GB of RAM is not enough, either. Get at least 128GB for Xeon E5-2600v4 Series (that is 8X 16GB) or get at least 192GB for Xeon Scalable Series (12X 16GB).

E5-2600v4 Series support 2400MHz ECC RDIMM memory and Xeon Scalable Series support 2666MHz ECC RDIMM memory. However the Silver models of Scalable Series, the CPU of your choice, support max 2400MHz. (You can buy 2933MHz RAMs, but they will work at a lower frequency. Nowadays, price wise they are same as 2666MHz RAMs, and 2400MHz RAMs are slightly cheaper.) Just make sure that the RAMs that you will order will be compatible with the barebone you will choose.

Thirdly, your SSD... If you go for the SYS-4029GP-TRT2, it supports NVME SSDs which will definitely be a better choice because they are much faster than SATA SSDs. If you choose SYS-4028GR-TRT2, it won't support NVME so you can stick to the SATA SSDs.

Your GPU choice is also wrong in my opinion. Both of these barebones support 10 GPUs. Currently, the 2 Slot wide Turbo models are not in the market yet, so you can use up to 5 GPUs with either of the barebones with a 4 slot spacing. In the future, when the turbo models are released,  you can sell the initial GPUs that you bought, and upgrade to 10 GPUs.

I see that you are ordering 4X 3090s. I would instead get 5X 3080s. 4X 3090s and 5X 3080s will give you similar hashrates, but you will pay much much less for the 3080s... (Approximately USD3750 for 5X 3080s vs approximately USD7000 for 4X 3090s) I believe 8 of the PCIE power cables already come with the barebones, you should check that with your vendor, or even better from Supermicro web site. I think you should order 2 more to make it 10. In all cases, make sure you have enough PCIE power cables for 10 GPUs for your future upgrades.

Also, do not forget to limit your GPU Watts to 300 Watts max, because all of the current GPGPU Servers allow max 300 watts per GPU, and both the RTX 3080 and RTX 3090 draw more than 300 Watts. This 300 Watts limitation will not change your hashrates, because the RTX cards have Tensor Cores and RT Cores which are not being used by Hashcat. 300 Watts will be enough for the cores that will be used. How to change the max power draw of Nvidia GPUs was already covered in Hashcat forums.

SLI is useless with Hashcat and is not required.

I would suggest that you use Ubuntu 20.04 OS.

Having said all of these, if I were you, I would reconsider switching back to Terahash (or any other vendor with a better understanding of password cracking systems, or GPGPU systems), as well. Obviously you lack the basic hardware knowledge. In the long run, you can of course, study and learn all of these, but if you are not willing to do that, it may be a better choice for you to be able to get professional support on these matters. Do not forget that these companies will add a premium (uplift) to the prices, which is understandable and acceptable. If you are only buying a single system, I suggest you just to pay the premium. If you are intending to buy multiple systems (a cluster), then calculate the total premium that you will be paying over the market prices, and compare that amount with the fees of professional IT Systems consultancy that you might get, and choose whichever is more beneficial for you.

Before you order anything, if you post the quoted prices here, people can warn you if you are paying more than you should. Prices may change from region to region so it would be best to mention which country you are from.

Lastly, keep in mind that that these systems are very very noisy. You cannot use them in your regular workspace. You will need a separate room for these systems, ideally a data center environment, or just a separate room with adequate cooling, and an adequate power infrastructure.

Please do not hesitate to ask if you have further questions...

[NoReply]

Right, don't use SLI with Hashcat.

The recommended OS for Hashcat is Ubuntu (Hashcat is developed on Ubuntu), but Windows is also fine.

I agree with Sondero, that you should have at least 64GB RAM and also upgrade this to 128GB once you add more GPUs.

I would discourage the 3090, since the Hashes/$ ratio is much better for a 3080. Also it's absolutely necessary to have blower style models for your GPUs.

The rest looks fine, IMO.

[mercystarf]

Having said all of these, if I were you, I would reconsider switching back to Terahash (or any other vendor with a better understanding of password cracking systems, or GPGPU systems), as well.

Thanks for the extensive feedback. I will surely go through all of the details. Of course I wanted to go with a professional party but if I order a system in April, I'd expect to have it by October. Have not heard back from Terahash since May. Not an appropriate way of working in a corporate context. So since I have the budget allocated, and I need the capability, I need to spend the money ASAP. If you would know of any other vendor who could help me out, please let me know (preferably in Europe).

And yes, we'll lock this one up in a datacenter rack.

[Longtail]

I must start by telling you that I am by no means affiliated with Terahash or the owners of the enterprise. However I must also add that, according to my best knowledge, they are one of the leading companies in the field of password cracking in North America. Again to my best knowledge, the owners/shareholders of the mentioned company are among the well known developers and contributors of the Hashcat software itself.

On the other hand, I truly understand you, not having heard from a vendor for such a long period of time is not the best practice for a client. I cannot comment furthermore on the subject without knowing the exact nature of professional relations between the parties, nor I think it would be appropriate to interfere with the commercial relations of a vendor and a client in a public forum. There may be a number of reasons causing the delay, and I cannot not comment by hearing only one of the parties involved.

I do have a few contacts from Europe. Mainly, they are companies that I met in international trade shows and conventions. I am not really sure if it is allowed to recommend third party private companies here in this forum publicly. Please PM me with your contact details and I will try to hook you up with some of these companies. I cannot guarantee though, if they will get in touch with you at all, nor if they will be able to come up with better and more tempting deals compared to what you already have in hand.

[mercystarf]

I must start by telling you that I am by no means affiliated with Terahash or the owners of the enterprise. However I must also add that, according to my best knowledge, they are one of the leading companies in the field of password cracking in North America. Again to my best knowledge, the owners/shareholders of the mentioned company are among the well known developers and contributors of the Hashcat software itself.

That's exactly the reason I chose to work with them. I'm convinced Jeremy is an intelligent man. No longer convinced of his customer relationship abilities though.

I do have a few contacts from Europe. Mainly, they are companies that I met in international trade shows and conventions. I am not really sure if it is allowed to recommend third party private companies here in this forum publicly. Please PM me with your contact details and I will try to hook you up with some of these companies. I cannot guarantee though, if they will get in touch with you at all, nor if they will be able to come up with better and more tempting deals compared to what you already have in hand.

Thanks for the help but I'll keep it local for now. Will also be much easier in case of hardware issues.

Godspeed!

[mercystarf]

Get at least 128GB for Xeon E5-2600v4 Series (that is 8X 16GB) or get at least 192GB for Xeon Scalable Series (12X 16GB).

Can you elaborate on how hashcat will be able to put 192GB of RAM to good use?

[undeath]

Due to the way the Nvidia drivers work the same amount of vRAM hashcat allocates is also allocated in RAM. Hence the recommendation to always have RAM >= vRAM.

[Longtail]

If you go for the Xeon Scalable series, you have two alternatives which will make sense. Either 96GB (12X 8GB) or 192GB (12X 16GB)... When using 10X RTX 3080s, 96GB will be slightly less than your total VRAM, which will be 100GB (10X 10GB) so 192 will be the minimum RAM capacity that you will need.