Building a password rig with 0 success... help needed!

[kcccs12]

I have been building a rig for four weeks with zero success.. 

So this is word vomit, but I'm dumping all of my info here, hoping someone can pinpoint some red flags or issues to avoid.. I need any help I can get. 

My first two weeks were screwing with AMD GPUs til I learned about PCIe Atomics - so I've since thrown those out of the equation. I'm now running with GTX 1070's and I have it limping along with the current setup: 
Windows 10
AMD Driver 436.48
Hashcat 5.1.0

This will 
- boot half the time
- perform at slow speeds (benchmarked WPA/2500 at about ~110kh/s... ~290kh/s is expected and ~360kh/s is achieved when only one card is running)
- randomly crash

On another note I completely hate Windows and would like this to work with some flavor of linux. 

I have tried: 
Ubuntu 14.04.6 server
Ubuntu 16.04 desktop
Ubuntu 18.04 desktop
Ubuntu 18.04 server
Parrot OS (forgot vers)

Each of the above, I've tried Nvidia drivers from apt packages, as well as installing directly from Nvidia. Specifically, I've tried to manually install 430.50 and 390.129 using the *.run installer and I've tried a flavor of drivers from apt packages. 

I have two motherboards/build that I've swapped out throughout my trials:
Setup 1. Used Mining Rig:
- BioStar TB250-BTC Motherboard
- Intel Celeron Processor (unknown)
- 8GB RAM (unknown)

Setup 2. New Build:
- MSI MPG Z390 Gaming Plus Motherboard
- Intel Core i5-9400 2.9GHz
- 2x16GB Corsair DDR4 RAM

Either setup would use PCIe risers, which I have about 20x of and I've swapped out to ensure functionality. 

Powered by:
1x 750W EVGA Gold (3x risers, 3x GPUs)
1x 1000W EVGA Gold (Mobo, SSD, 3x risers, 3x GPUs)

With the Ubuntu installs, I've had the following outcome:
- Ubuntu never boots once GPU is installed
- or, Hashcat benchmark's fine with 1 GPU, then OS won't book with 2+ GPUs
- Once I finally got all six working, then Hashcat would die halfway through benchmark, and state "GPU cannot be found"

I've also tried variants of Hashcat binaries zip'd, compiled my own, and installed from apt. So far the most successful has been compiling from source. 

================================================================

I'm unsure where to go from here. I feel like I'm going back in circles and ending up in the same spot. I must've reinstalled various OS's 20+ times over the past few weeks.

Are my 1070s too old for the current drivers?
Should I be using a specific version of Ubuntu?
Do you see any red flags in anything I'm doing? 
Is there something I'm missing?

Every walk through I have read seems so very simple. Install OS, install drivers, install hashcat, go. Mine experience has never worked that easily. 

I'm about to wipe everything and start again, I'll reply to this thread with steps/details to see if anyone can pinpoint what I'm doing wrong. 

Seriously any info/tips/pointers would be appreciated.

[kcccs12]

Update to my post above with some numbers and info...

Installed Ubuntu 18.04.3 Server:

1. All defaults + openssh
   
2. Did a full update, installed gcc/make
   
3. Downloaded linux driver from Nvidia: link
   (Installer screamed about Nouveu, run the disable command, reboot, didn't work. Disabled Nouveu according to blackhills blog --> here. Then errors about X and 32bit libraries, but otherwise seemed to install correctly.)

First hashcat benchmark runs successful, one GPU installed:
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
Speed.#1.........:  340.2 kH/s (89.52ms) @ Accel:128 Loops:64 Thr:1024 Vec:1


Then two GPUs kill performance:
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
Speed.#1.........:  134.1 kH/s (30.18ms) @ Accel:128 Loops:16 Thr:1024 Vec:1
Speed.#2.........:  190.4 kH/s (23.62ms) @ Accel:128 Loops:16 Thr:1024 Vec:1
Speed.#*.........:  324.5 kH/s



Tried to go for three GPUs and the system won't boot up, simply hangs on a black screen. 

Ideas?

[philsmd]

did you try to setup the system for instance with only 2 GPUs and without using risers at all ? risers could introduce a lot of headaches for sure

[undeath]

The problems you are seeing sound like defective risers.

[kcccs12]

did you try to setup the system for instance with only 2 GPUs and without using risers at all ? risers could introduce a lot of headaches for sure

The problems you are seeing sound like defective risers.

Thanks! You are correct about the risers! I swapped all risers with brand new ones, and it seems to change the 1->2 GPU performance decrease. For testing purposes, new risers vs direct connect perform the same. Im seeing roughly ~340kH/s for WPA2 whether on risers or not. 

Now the current problem - whenever I attempt 5 or 6 GPUs, the machine won't boot. It is simply a black screen with a rapid flashing cursor.. 

Using a max of four at a time, I have tested all GPUs, all risers, and all PCIE Slots. No matter the config, four cards work fine but once the 5th card is plugged in, the machine won't boot. 

It sounds like it would be a power overload problem but I have 1000W powering 3GPUs + Mobo + sata. An additional 750W is only powering the fourth GPU and the potential 5th GPU I'm trying. 

Any ideas?

[Mrblank]

Hi, my first post here, I wouldn't have anywhere near the experience as some users here but I'm currently assembling parts for a 4 gpus chenbro fs81 build, I have looked over these forums for a while and they say you need more system RAM then the combined vram of the gpus, those 1070s afaik are 8gb vram each, do you have sufficient RAM inserted?

[kcccs12]

Hi, my first post here, I wouldn't have anywhere near the experience as some users here but I'm currently assembling parts for a 4 gpus chenbro fs81 build, I have looked over these forums for a while and they say you need more system RAM then the combined vram of the gpus, those 1070s afaik are 8gb vram each, do you have sufficient RAM inserted?

you're correct, the 1070s run 8gb each. I'm currently sitting with 32gb ddr4 on the board, but that shouldn't be a hard stop to prevent more than four? I've always heard RAM>VRAM as a strong suggestion, but not a strict requirement..
But for sake of trying, I'll throw two more sticks in there and see if that solves anything.

[kcccs12]

So I was running multiple tests with 4 GPUs. I got some good benchmarks and decided to give it room to stretch. A couple of test hashes and wordlists popped out without issues. But when I fed it a true dictionary with OneRuletoRuleThemAll, it spun for about five minutes then lost one of the GPUs. Another couple minutes later the fourth GPU was reporting 0% utilization. I killed all processes and halted the machine.

From posts online, I figured it might be USB3.0 cabling for the risers, so I swapped all cabling, and even swapped some GPUs and PCI slots.

At this point now, only two GPUs come up hashcat's identification & when trying to run anything I'm getting the "CL_PLATFORM_NOT_FOUND_KHR" error, which I believe is usually driver related.
Why is this happening now, when the rig was running (limping) along earlier?

& Still looking to see if anyone has a suggestion for my hard 4 GPU limit above...

[Mrblank]

Could 4 gpus be the limit for that board with single CPU? Is it equipped with PLX chips?

[undeath]

you need more system RAM then the combined vram of the gpus, those 1070s afaik are 8gb vram each, do you have sufficient RAM inserted?

This is true for hashcat, but it's got nothing to do with the system not booting.

Could 4 gpus be the limit for that board with single CPU? Is it equipped with PLX chips?

I'm not aware of any mainboard/cpu-imposed limits on the amount of graphics cards, other than available PCIe slots/lanes. Isn't PLX only to help with insufficient PCIe lanes?