Low utilization with multi GPU

[cpu_pirate]

As stated, I recently upgraded my cracking rig from 1-1080Ti to 2x1080TI + 3x1070Ti taking advantage of the cheap GPU's with the mining exodus. The issue is, I have the 2x1080Ti's on water, when I only had1 gpu performance was great in hashcat, gpu was 100% utilized and rates were 630Mh/s+ (wpa2) (As this was also my display gpu). After adding the rest of them, and in no matter what order I run them in ( using -d 1,2 for example) each GPU will only get to about 36% utilization and the performance therefore is terrible. Is there an easy fix for this I am missing?

I will add that I have tweaked the cards , power and clock rate etc via afterburner (currently in windows, will test results in linux today as well) and using the benchmark feature (-b) they all put out great numbers, but when using in practice its down to ~36% utilization .. see benchmark below

OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: GeForce GTX 1080 Ti, 2816/11264 MB allocatable, 28MCU
* Device #2: GeForce GTX 1080 Ti, 2816/11264 MB allocatable, 28MCU
* Device #3: GeForce GTX 1070 Ti, 2048/8192 MB allocatable, 19MCU
* Device #4: GeForce GTX 1070 Ti, 2048/8192 MB allocatable, 19MCU

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)

Speed.Dev.#1.....:   663.8 kH/s (84.77ms) @ Accel:128 Loops:64 Thr:1024 Vec:1
Speed.Dev.#2.....:   626.5 kH/s (44.77ms) @ Accel:128 Loops:32 Thr:1024 Vec:1
Speed.Dev.#3.....:   439.0 kH/s (87.31ms) @ Accel:128 Loops:64 Thr:1024 Vec:1
Speed.Dev.#4.....:   459.6 kH/s (83.42ms) @ Accel:128 Loops:64 Thr:1024 Vec:1
Speed.Dev.#*.....:  2188.9 kH/s

[atom]

Benchmark numbers look fine, you probably do not feed your GPU with enough workload on the base. Try running in -a 3 mode.

[cpu_pirate]

Hmm .. same result:

Session..........: GL-brute-8-10
Status...........: Running
Hash.Type........: WPA-EAPOL-PBKDF2
Hash.Target......: d:\Training\wireless\captures\Greenlight.hccapx
Time.Started.....: Tue Aug 14 13:06:25 2018 (28 secs)
Time.Estimated...: Next Big Bang (> 10 years)
Guess.Mask.......: ?a?a?l?l?l?l?l?a?a?a [10]
Guess.Queue......: 1/1 (100.00%)
Speed.Dev.#1.....: 230.7 kH/s (5.72ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#2.....: 230.3 kH/s (5.73ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#3.....: 156.5 kH/s (5.78ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#4.....: 156.9 kH/s (5.72ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#*.....: 774.4 kH/s
Recovered........: 0/17 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 21561344/91935822600700000 (0.00%)
Rejected.........: 0/21561344 (0.00%)
Restore.Point....: 0/967745501060000 (0.00%)
Candidates.#1....: parieriner -> p:rdfanere
Candidates.#2....: p`jmnanere -> p]eginaner
Candidates.#3....: p?wmelller -> pNqeponere
Candidates.#4....: pMqeponere -> plaxvinere
HWMon.Dev.#1.....: Temp: 33c Fan: 0% Util: 37% Core:2113MHz Mem:4704MHz Bus:8
HWMon.Dev.#2.....: Temp: 35c Fan: 23% Util: 38% Core:2088MHz Mem:4908MHz Bus:8
HWMon.Dev.#3.....: Temp: 55c Fan: 7% Util: 35% Core:2025MHz Mem:3499MHz Bus:1
HWMon.Dev.#4.....: Temp: 52c Fan: 0% Util: 38% Core:2050MHz Mem:3499MHz Bus:1

[soxrok2212]

You have 17 handshakes in your .hccapx file. Remove 16 of them.

[tacohashcat]

I know this is not related to your problem but you said you have 2x1080Ti and 3x1070Ti and I only see 4 devices listed. You are missing a 1070Ti. Unless you made a typo.
soxrok2212 makes a good point.

[cpu_pirate]

I think I have seen in the past that number of handshakes doesn't have an effect on speed, I know I did not have a huge drop in hashrate on this same capture with one card, but I will try it with a single handshake just to be sure.

And yes I have 3 1070Ti's but during this run on only had 2 connected. I also tested with just the 2 1080Ti's and had the same result. Still pending testing on Linux. Hopefully be able to try that tomorrow.

[cpu_pirate]

Have not tweaked any settings in Linux so the performance is down but just quick testing and the issue does not appear to follow, hitting 97% utilization on each card.

Session..........: hashcat
Status...........: Running
Hash.Type........: WPA-EAPOL-PBKDF2
Hash.Target......: /media/cpu_pirate/shared/Training/wireless/captures/Greenlight.hccapx
Time.Started.....: Thu Aug 16 09:52:20 2018 (1 min, 59 secs)
Time.Estimated...: Fri Mar 24 08:34:47 2248 (229 years, 219 days)
Guess.Mask.......: ?a?a?a?a?a?a?a?a [8]
Guess.Queue......: 1/1 (100.00%)
Speed.Dev.#1.....: 467.7 kH/s (6.49ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#2.....: 447.9 kH/s (6.79ms) @ Accel:32 Loops:16 Thr:1024 Vec:1
Speed.Dev.#*.....: 915.6 kH/s
Recovered........: 0/17 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 108265472/6634204312890625 (0.00%)
Rejected.........: 0/108265472 (0.00%)
Restore.Point....: 0/69833729609375 (0.00%)
Candidates.#1....: Zarierin -> Z:[olerd
Candidates.#2....: Y`=monan -> Y]/merin
HWMon.Dev.#1.....: Temp: 46c Fan: 0% Util: 96% Core:1847MHz Mem:5005MHz Bus:8
HWMon.Dev.#2.....: Temp: 50c Fan: 29% Util: 97% Core:1771MHz Mem:5005MHz Bus:8